5:05 p.m.
Hi,
I have a two node + engine ovirt setup, and I was having problems
doing a live migration between nodes. I looked in the vdsm logs and
noticed selinux errors, so I checked the selinux config, and both the
ovirt-engine host and one of the nodes had selinux disabled. So I
thought I would enable it on these two hosts, as it is officially
supported anyway. I started with the node, and put it into maintenance
mode, which interestingly, migrated the VMs off to the other node
without issue. After modifying the selinux config, I then rebooted
that node, which came back up. I then tried to activate the node but
it fails and marks it as unresponsive.
----------8<----------
2016-04-28 16:34:31,326 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [29acb18b] Connecting to
kvm-ldn-02/172.16.23.12
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Command
'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
execution failed:
org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
failed
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
Connection failed
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
failed
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
Source) [:1.8.0_71]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497)
[rt.jar:1.8.0_71]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
Connection failed
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
[vdsbroker.jar:]
... 14 more
----------8<----------
Any ideas?
Thanks for any help,
Cam
3:38 p.m.
Cam,
Please provide engine and failing vdsm logs.
Thanks,
Piotr
On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu(a)gmail.com> wrote:
Hi,
I have a two node + engine ovirt setup, and I was having problems
doing a live migration between nodes. I looked in the vdsm logs and
noticed selinux errors, so I checked the selinux config, and both the
ovirt-engine host and one of the nodes had selinux disabled. So I
thought I would enable it on these two hosts, as it is officially
supported anyway. I started with the node, and put it into maintenance
mode, which interestingly, migrated the VMs off to the other node
without issue. After modifying the selinux config, I then rebooted
that node, which came back up. I then tried to activate the node but
it fails and marks it as unresponsive.
----------8<----------
2016-04-28 16:34:31,326 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [29acb18b] Connecting to
kvm-ldn-02/172.16.23.12
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Command
'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
execution failed:
org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
failed
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
Connection failed
2016-04-28 16:34:31,327 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
failed
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
Source) [:1.8.0_71]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497)
[rt.jar:1.8.0_71]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
Connection failed
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
[vdsbroker.jar:]
... 14 more
----------8<----------
Any ideas?
Thanks for any help,
Cam
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
5:35 p.m.
Hi Piotr,
Attached are the vdsm log, the engine log and the supervdsm log. I've
attached them as a .tgz.
I noticed it is complaining about configuring an interface in one of the
node logs. It shows as up in the engine web GUI though (and on the command
line).
Thanks for the help.
-Cam
On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski <piotr.kliczewski(a)gmail.com
wrote:
> Cam,
>
> Please provide engine and failing vdsm logs.
>
> Thanks,
> Piotr
>
> On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu(a)gmail.com wrote:
> > Hi,
> >
> > I have a two node + engine ovirt setup, and I was having problems
> > doing a live migration between nodes. I looked in the vdsm logs and
> > noticed selinux errors, so I checked the selinux config, and both the
> > ovirt-engine host and one of the nodes had selinux disabled. So I
> > thought I would enable it on these two hosts, as it is officially
> > supported anyway. I started with the node, and put it into maintenance
> > mode, which interestingly, migrated the VMs off to the other node
> > without issue. After modifying the selinux config, I then rebooted
> > that node, which came back up. I then tried to activate the node but
> > it fails and marks it as unresponsive.
> >
> > ----------8<----------
> >
> > 2016-04-28 16:34:31,326 INFO
> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
> > Reactor) [29acb18b] Connecting to
> > kvm-ldn-02/172.16.23.12
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
> > execution failed:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> > [dal.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
> > [vdsbroker.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
> > [vdsbroker.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
> > [vdsbroker.jar:]
> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
> > Source) [:1.8.0_71]
> > at
> >
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [rt.jar:1.8.0_71]
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > [rt.jar:1.8.0_71]
> > at
> >
> org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
> > [scheduler.jar:]
> > at
> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
> > [scheduler.jar:]
> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> > [quartz.jar:]
> > at
> >
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> > [quartz.jar:]
> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > at
> >
> org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
> > [vdsbroker.jar:]
> > ... 14 more
> >
> > ----------8<----------
> >
> > Any ideas?
> >
> > Thanks for any help,
> >
> > Cam
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
1:37 p.m.
Cam,
It is really hard to understand partial logs. Please send me directly
full logs and I will analyze.
Thanks,
Piotr
On Mon, May 2, 2016 at 4:35 PM, Cam Mac <iucounu(a)gmail.com> wrote:
Hi Piotr,
Attached are the vdsm log, the engine log and the supervdsm log. I've
attached them as a .tgz.
I noticed it is complaining about configuring an interface in one of the
node logs. It shows as up in the engine web GUI though (and on the command
line).
Thanks for the help.
-Cam
On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski
<piotr.kliczewski(a)gmail.com> wrote:
>
> Cam,
>
> Please provide engine and failing vdsm logs.
>
> Thanks,
> Piotr
>
> On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu(a)gmail.com> wrote:
> > Hi,
> >
> > I have a two node + engine ovirt setup, and I was having problems
> > doing a live migration between nodes. I looked in the vdsm logs and
> > noticed selinux errors, so I checked the selinux config, and both the
> > ovirt-engine host and one of the nodes had selinux disabled. So I
> > thought I would enable it on these two hosts, as it is officially
> > supported anyway. I started with the node, and put it into maintenance
> > mode, which interestingly, migrated the VMs off to the other node
> > without issue. After modifying the selinux config, I then rebooted
> > that node, which came back up. I then tried to activate the node but
> > it fails and marks it as unresponsive.
> >
> > ----------8<----------
> >
> > 2016-04-28 16:34:31,326 INFO
> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
> > Reactor) [29acb18b] Connecting to
> > kvm-ldn-02/172.16.23.12
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
> > execution failed:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> > [dal.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
> > [vdsbroker.jar:]
> > at
> >
> > org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
> > [vdsbroker.jar:]
> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
> > Source) [:1.8.0_71]
> > at
> >
> >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [rt.jar:1.8.0_71]
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > [rt.jar:1.8.0_71]
> > at
> >
> > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
> > [scheduler.jar:]
> > at
> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
> > [scheduler.jar:]
> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> > [quartz.jar:]
> > at
> >
> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> > [quartz.jar:]
> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > at
> >
> >
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
> > [vdsbroker.jar:]
> > at
> >
> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
> > [vdsbroker.jar:]
> > ... 14 more
> >
> > ----------8<----------
> >
> > Any ideas?
> >
> > Thanks for any help,
> >
> > Cam
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
2:40 p.m.
No problem Piotr, I will send you the full logs shortly.
Thanks,
Campbell
On 5 May 2016 at 11:37, Piotr Kliczewski <piotr.kliczewski(a)gmail.com> wrote:
Cam,
It is really hard to understand partial logs. Please send me directly
full logs and I will analyze.
Thanks,
Piotr
On Mon, May 2, 2016 at 4:35 PM, Cam Mac <iucounu(a)gmail.com> wrote:
> Hi Piotr,
>
> Attached are the vdsm log, the engine log and the supervdsm log. I've
> attached them as a .tgz.
>
> I noticed it is complaining about configuring an interface in one of the
> node logs. It shows as up in the engine web GUI though (and on the command
> line).
>
> Thanks for the help.
>
> -Cam
>
> On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski
> <piotr.kliczewski(a)gmail.com> wrote:
>>
>> Cam,
>>
>> Please provide engine and failing vdsm logs.
>>
>> Thanks,
>> Piotr
>>
>> On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu(a)gmail.com> wrote:
>> > Hi,
>> >
>> > I have a two node + engine ovirt setup, and I was having problems
>> > doing a live migration between nodes. I looked in the vdsm logs and
>> > noticed selinux errors, so I checked the selinux config, and both the
>> > ovirt-engine host and one of the nodes had selinux disabled. So I
>> > thought I would enable it on these two hosts, as it is officially
>> > supported anyway. I started with the node, and put it into maintenance
>> > mode, which interestingly, migrated the VMs off to the other node
>> > without issue. After modifying the selinux config, I then rebooted
>> > that node, which came back up. I then tried to activate the node but
>> > it fails and marks it as unresponsive.
>> >
>> > ----------8<----------
>> >
>> > 2016-04-28 16:34:31,326 INFO
>> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
>> > Reactor) [29acb18b] Connecting to
>> > kvm-ldn-02/172.16.23.12
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
>> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
>> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
>> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
>> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
>> > execution failed:
>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
>> > failed
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
>> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
>> > Connection failed
>> > 2016-04-28 16:34:31,327 ERROR
>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
>> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
>> > failed
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
>> > [vdsbroker.jar:]
>> > at
>> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
>> > [dal.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
>> > [vdsbroker.jar:]
>> > at
>> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
>> > [vdsbroker.jar:]
>> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
>> > Source) [:1.8.0_71]
>> > at
>> >
>> >
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > [rt.jar:1.8.0_71]
>> > at java.lang.reflect.Method.invoke(Method.java:497)
>> > [rt.jar:1.8.0_71]
>> > at
>> >
>> >
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
>> > [scheduler.jar:]
>> > at
>> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
>> > [scheduler.jar:]
>> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>> > [quartz.jar:]
>> > at
>> >
>> >
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
>> > [quartz.jar:]
>> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
>> > Connection failed
>> > at
>> >
>> >
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
>> > [vdsm-jsonrpc-java-client.jar:]
>> > at
>> >
>> >
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
>> > [vdsm-jsonrpc-java-client.jar:]
>> > at
>> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
>> > [vdsm-jsonrpc-java-client.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
>> > [vdsbroker.jar:]
>> > at
>> >
>> >
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
>> > [vdsbroker.jar:]
>> > ... 14 more
>> >
>> > ----------8<----------
>> >
>> > Any ideas?
>> >
>> > Thanks for any help,
>> >
>> > Cam
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users(a)ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Campbell McLeay · Senior Systems Administrator
London · New York · Los Angeles · Montréal
T +44 (0)20 7344 8000
19-23 Wells Street, London W1T 3PQ
Twitter · Facebook · framestore.com
5:40 p.m.
To update the thread here after some offline discussion with Piotr, the
problem was that vdsmd had stopped running and wasn't able to start. It
wasn't reported in the logs that I could see, but I noticed it after trying
to restart it at Piotr's suggestion, after running a 'systemctl'. A
'journalctl -xe' reported:
[root@kvm-ldn-02 vdsm]# journalctl -xe
-- Unit vdsmd.service has failed.
--
-- The result is failed.
May 05 15:15:15 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm:
Running check_is_configured
May 05 15:15:16 kvm-ldn-02.ldn.org sasldblistusers2[19047]: DIGEST-MD5
common mech free
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Error:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: One of the
modules is not configured to work with VDSM.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: To
configure the module use the following:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool
configure [--module module-name]'.
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: If all
modules are not configured try to use:
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool
configure --force'
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: (The force
flag will stop the module's service and start it
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: afterwards
automatically to load the new configuration.)
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Current
revision of multipath.conf detected, preserving
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: libvirt is
already configured for vdsm
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Modules
sebool are not configured
May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm:
stopped during execute check_is_configured task (task returned with error
code 1).
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: vdsmd.service: control
process exited, code=exited status=1
May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: Failed to start Virtual
Desktop Server Manager.
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
So I ran a 'vdsm-tool configure --module sebool', and then I was able to
start vdsmd and mom-vdsm. This fixed the connectivity issue and the node is
now back online.
Many thanks to Piotr for all his help.
Cheers,
Cam
On Mon, May 2, 2016 at 3:35 PM, Cam Mac <iucounu(a)gmail.com> wrote:
Hi Piotr,
Attached are the vdsm log, the engine log and the supervdsm log. I've
attached them as a .tgz.
I noticed it is complaining about configuring an interface in one of the
node logs. It shows as up in the engine web GUI though (and on the command
line).
Thanks for the help.
-Cam
On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski <
piotr.kliczewski(a)gmail.com> wrote:
> Cam,
>
> Please provide engine and failing vdsm logs.
>
> Thanks,
> Piotr
>
> On Sun, May 1, 2016 at 4:05 PM, Cam Mac <iucounu(a)gmail.com> wrote:
> > Hi,
> >
> > I have a two node + engine ovirt setup, and I was having problems
> > doing a live migration between nodes. I looked in the vdsm logs and
> > noticed selinux errors, so I checked the selinux config, and both the
> > ovirt-engine host and one of the nodes had selinux disabled. So I
> > thought I would enable it on these two hosts, as it is officially
> > supported anyway. I started with the node, and put it into maintenance
> > mode, which interestingly, migrated the VMs off to the other node
> > without issue. After modifying the selinux config, I then rebooted
> > that node, which came back up. I then tried to activate the node but
> > it fails and marks it as unresponsive.
> >
> > ----------8<----------
> >
> > 2016-04-28 16:34:31,326 INFO
> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
> > Reactor) [29acb18b] Connecting to
> > kvm-ldn-02/172.16.23.12
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command
> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02,
> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa',
> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})'
> > execution failed:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds
> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > 2016-04-28 16:34:31,327 ERROR
> > [org.ovirt.engine.core.vdsbroker.HostMonitoring]
> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception:
> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
> > failed
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
> > [dal.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
> > [vdsbroker.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
> > [vdsbroker.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
> > [vdsbroker.jar:]
> > at
> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227)
> > [vdsbroker.jar:]
> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown
> > Source) [:1.8.0_71]
> > at
> >
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [rt.jar:1.8.0_71]
> > at java.lang.reflect.Method.invoke(Method.java:497)
> > [rt.jar:1.8.0_71]
> > at
> >
> org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
> > [scheduler.jar:]
> > at
> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
> > [scheduler.jar:]
> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
> > [quartz.jar:]
> > at
> >
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
> > [quartz.jar:]
> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
> > Connection failed
> > at
> >
> org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
> > [vdsm-jsonrpc-java-client.jar:]
> > at
> >
> org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:68)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15)
> > [vdsbroker.jar:]
> > at
> >
>
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110)
> > [vdsbroker.jar:]
> > ... 14 more
> >
> > ----------8<----------
> >
> > Any ideas?
> >
> > Thanks for any help,
> >
> > Cam
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
3060
days inactive
3064
days old
5 comments
3 participants
participants (3)
-
Cam Mac -
Campbell McLeay -
Piotr Kliczewski