HI
After installing ovirt 3.6,unable to add ovirt node to the ovirt
application ,below are the logs.
The moment I add to ovirt application network is getting disconnected
prior to that ovirt node was able to reach the gateway without any issues.
any solution that wold resolve my issues ?
2015-10-21 17:44:44,179 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:44:46,179 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:44:46,179 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-40) [] Exception during connection
2015-10-21 17:44:46,179 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler_Worker-40) [] Command
'GetAllVmStatsVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:46,179 INFO
[org.ovirt.engine.core.vdsbroker.PollVmStatsRefresher]
(DefaultQuartzScheduler_Worker-40) [] Failed to fetch vms info for host
'infraesx2' - skipping VMs monitoring.
2015-10-21 17:44:46,179 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:44:48,179 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:44:48,179 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-41) [] Exception during connection
2015-10-21 17:44:48,179 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-41) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:48,179 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-41) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:48,179 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-41) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:44:51,182 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:44:53,183 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:44:53,183 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-14) [22712a82] Exception during connection
2015-10-21 17:44:53,183 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-14) [22712a82] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:53,183 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-14) [22712a82] Failure to refresh Vds
runtime info: java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:53,183 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-14) [22712a82] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:44:56,187 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:44:58,187 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:44:58,187 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-11) [] Exception during connection
2015-10-21 17:44:58,187 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-11) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:58,187 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-11) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:44:58,187 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-11) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:45:01,180 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:03,180 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:03,180 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-64) [] Exception during connection
2015-10-21 17:45:03,180 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler_Worker-64) [] Command
'GetAllVmStatsVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:03,180 INFO
[org.ovirt.engine.core.vdsbroker.PollVmStatsRefresher]
(DefaultQuartzScheduler_Worker-64) [] Failed to fetch vms info for host
'infraesx2' - skipping VMs monitoring.
2015-10-21 17:45:03,181 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:05,181 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:05,181 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-66) [] Exception during connection
2015-10-21 17:45:05,181 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-66) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:05,181 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-66) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:05,181 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-66) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:45:08,191 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:10,191 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:10,191 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-75) [] Exception during connection
2015-10-21 17:45:10,191 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-75) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:10,191 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-75) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:10,191 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-75) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:45:13,195 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:15,195 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:15,195 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-94) [] Exception during connection
2015-10-21 17:45:15,195 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-94) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:15,195 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-94) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:15,195 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-94) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:45:18,182 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:20,182 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:20,182 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-40) [] Exception during connection
2015-10-21 17:45:20,182 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(DefaultQuartzScheduler_Worker-40) [] Command
'GetAllVmStatsVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:20,182 INFO
[org.ovirt.engine.core.vdsbroker.PollVmStatsRefresher]
(DefaultQuartzScheduler_Worker-40) [] Failed to fetch vms info for host
'infraesx2' - skipping VMs monitoring.
2015-10-21 17:45:20,182 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-21 17:45:22,183 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor)
[] Retry failed
2015-10-21 17:45:22,183 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(DefaultQuartzScheduler_Worker-60) [] Exception during connection
2015-10-21 17:45:22,183 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-60) [] Command
'GetCapabilitiesVDSCommand(HostName = infraesx2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c',
vds='Host[infraesx2,2d74bd31-6894-4c9c-8260-97c305a6998c]'})' execution
failed: java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:22,183 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-60) [] Failure to refresh Vds runtime info:
java.rmi.ConnectException: Connection timeout
2015-10-21 17:45:22,183 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-60) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.rmi.ConnectException: Connection timeout
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:634)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:226)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: java.rmi.ConnectException: Connection timeout
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:116)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-21 17:45:24,914 INFO
[org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand]
(org.ovirt.thread.pool-8-thread-17) [6482a3e8] Running command:
MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID:
2d74bd31-6894-4c9c-8260-97c305a6998c Type: VDSAction group MANIPULATE_HOST
with role type ADMIN
2015-10-21 17:45:24,931 INFO
[org.ovirt.engine.core.bll.MaintenanceVdsCommand]
(org.ovirt.thread.pool-8-thread-17) [6482a3e8] Running command:
MaintenanceVdsCommand internal: true. Entities affected : ID:
2d74bd31-6894-4c9c-8260-97c305a6998c Type: VDS
2015-10-21 17:45:24,941 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-17) [6482a3e8] START,
SetVdsStatusVDSCommand(HostName = infraesx2,
SetVdsStatusVDSCommandParameters:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c', status='Maintenance',
nonOperationalReason='NONE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: c22525
2015-10-21 17:45:24,944 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-8-thread-17) [6482a3e8] FINISH,
SetVdsStatusVDSCommand, log id: c22525
2015-10-21 17:45:24,949 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-17) [6482a3e8] Correlation ID: 6482a3e8,
Job ID: 226758ca-59df-489f-af7f-33cc96f97d30, Call Stack: null, Custom
Event ID: -1, Message: Host infraesx2 was switched to Maintenance mode by
admin@internal (Reason: Not Specified).
2015-10-21 17:45:31,262 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand]
(default task-16) [2e9e5834] Lock Acquired to object
'EngineLock:{exclusiveLocks='[2d74bd31-6894-4c9c-8260-97c305a6998c=<VDS,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2015-10-21 17:45:31,268 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] Running command:
RemoveVdsCommand internal: false. Entities affected : ID:
2d74bd31-6894-4c9c-8260-97c305a6998c Type: VDSAction group DELETE_HOST with
role type ADMIN
2015-10-21 17:45:31,289 INFO
[org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] START, RemoveVdsVDSCommand(
RemoveVdsVDSCommandParameters:{runAsync='true',
hostId='2d74bd31-6894-4c9c-8260-97c305a6998c'}), log id: 4aa64f93
2015-10-21 17:45:31,289 INFO [org.ovirt.engine.core.vdsbroker.VdsManager]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] vdsManager::disposing
2015-10-21 17:45:31,290 WARN
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) []
Exception thrown during message processing
2015-10-21 17:45:31,290 INFO
[org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] FINISH, RemoveVdsVDSCommand,
log id: 4aa64f93
2015-10-21 17:45:31,292 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] Correlation ID: 2e9e5834,
Call Stack: null, Custom Event ID: -1, Message: Host infraesx2 was removed
by admin@internal.
2015-10-21 17:45:31,292 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand]
(org.ovirt.thread.pool-8-thread-20) [2e9e5834] Lock freed to object
'EngineLock:{exclusiveLocks='[2d74bd31-6894-4c9c-8260-97c305a6998c=<VDS,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2015-10-21 17:45:39,619 INFO
[org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand]
(org.ovirt.thread.pool-8-thread-19) [6b6695a5] Running command:
RemoveStoragePoolCommand internal: false. Entities affected : ID:
fef2019c-d087-44f5-852e-40df4f749363 Type: StoragePoolAction group
DELETE_STORAGE_POOL with role type ADMIN
2015-10-21 17:45:39,651 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-19) [6b6695a5] Correlation ID: 6b6695a5,
Job ID: 37859e0f-296d-4a37-b738-25c890d50d65, Call Stack: null, Custom
Event ID: -1, Message: Data Center Test was removed by admin@internal
2015-10-21 17:45:46,964 INFO
[org.ovirt.engine.core.bll.RemoveVdsGroupCommand]
(org.ovirt.thread.pool-8-thread-16) [23932ed9] Running command:
RemoveVdsGroupCommand internal: false. Entities affected : ID:
32e20320-b0b1-416a-a733-55971bec2798 Type: VdsGroupsAction group
DELETE_CLUSTER with role type ADMIN
2015-10-21 17:45:47,005 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-16) [23932ed9] Correlation ID: 23932ed9,
Call Stack: null, Custom Event ID: -1, Message: Host cluster
infraesx2-Local was removed by admin@internal
2015-10-21 17:45:53,005 INFO
[org.ovirt.engine.core.bll.RemoveVdsGroupCommand]
(org.ovirt.thread.pool-8-thread-18) [1af7dad2] Running command:
RemoveVdsGroupCommand internal: false. Entities affected : ID:
00000001-0001-0001-0001-000000000060 Type: VdsGroupsAction group
DELETE_CLUSTER with role type ADMIN
2015-10-21 17:45:53,050 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-18) [1af7dad2] Correlation ID: 1af7dad2,
Call Stack: null, Custom Event ID: -1, Message: Host cluster Default was
removed by admin@internal
2015-10-21 17:46:03,341 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-11) [] Correlation ID: null, Call Stack: null, Custom Event
ID: -1, Message: Refresh image list succeeded for domain(s):
ovirt-image-repository (All file type)Last login: Wed Oct 21 17:01:03 2015
from
nbudoor-t440p.bnglab.psecure.net
[root@cstlb1 ~]#
[root@cstlb1 ~]#
[root@cstlb1 ~]#
[root@cstlb1 ~]# tail -f /var/log/ovirt-engine/engine.log
2015-10-26 11:37:31,475 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Misc configuration.
2015-10-26 11:37:31,478 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Package
installation.
2015-10-26 11:37:31,815 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Misc configuration.
2015-10-26 11:37:31,904 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Setting time.
2015-10-26 11:37:32,687 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Enrolling certificate.
2015-10-26 11:37:34,255 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Enrolling serial console
certificate.
2015-10-26 11:37:36,343 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Transaction commit.
2015-10-26 11:37:36,473 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Closing up.
2015-10-26 11:37:43,867 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stopping libvirtd.
2015-10-26 11:37:44,014 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Starting vdsm.
2015-10-26 11:38:19,244 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Starting
ovirt-vmconsole-host-sshd.
2015-10-26 11:38:21,052 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage: Pre-termination.
2015-10-26 11:38:21,072 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Retrieving installation
logs to:
'/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-20151026113821-infraesx2.bnglab.psecure.net-7ed18269.log'.2015-10-26
11:38:21,381 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [7ed18269] Correlation ID: 7ed18269, Call Stack: null, Custom
Event ID: -1, Message: Installing Host infraesx2. Stage:
Termination.2015-10-26 11:38:21,439 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.972015-10-26
11:38:21,440 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable]
(SSL Stomp Reactor) [] Retry failed
2015-10-26 11:38:21,440 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(org.ovirt.thread.pool-8-thread-26) [] Exception during connection
2015-10-26 11:38:21,440 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [7ed18269] Error:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection
issues during send request
2015-10-26 11:38:21,440 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [7ed18269] Exception:
java.util.concurrent.ExecutionException:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection
issues during send request
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[rt.jar:1.7.0_85]
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
[rt.jar:1.7.0_85]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:72)
[vdsbroker.jar:]
at
org.ovirt.engine.core.bll.transport.ProtocolDetector.attemptConnection(ProtocolDetector.java:56)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.installHost(InstallVdsInternalCommand.java:204)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:93)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1211)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1355)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1979)
[bll.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
[utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
[utils.jar:]
at
org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:374)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:475)
[bll.jar:]
at
org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:457) [bll.jar:]
at
org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:667)
[bll.jar:]
at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
[wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
[wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
[wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:407)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:55)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
[wildfly-weld-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
[wildfly-ee-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:260)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:399)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:95)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
[wildfly-ejb3-8.2.0.Final.jar:8.2.0.Final]
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:448)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:326)
at
org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)
at
org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view5.runInternalAction(Unknown
Source) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
[:1.7.0_85]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_85]
at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:414)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:127)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:65)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source) [bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2342)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.access$800(AddVdsCommand.java:67)
[bll.jar:]
at
org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand$3.run(AddVdsCommand.java:196)
[bll.jar:]
at
org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:92)
[utils.jar:]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[rt.jar:1.7.0_85]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_85]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_85]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_85]
Caused by: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException:
Connection issues during send request
at
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:89)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:1012)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$3.call(JsonRpcVdsServer.java:1008)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:482)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:471)
[vdsbroker.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
... 5 more
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:
java.util.concurrent.ExecutionException: java.net.ConnectException:
Connection refused
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:142)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:115)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:87)
[vdsbroker.jar:]
... 10 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[rt.jar:1.7.0_85]
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:114)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:100)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
[vdsm-jsonrpc-java-client.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[rt.jar:1.7.0_85]
at
org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28)
[vdsm-jsonrpc-java-client.jar:]
at
org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61)
[vdsm-jsonrpc-java-client.jar:]
2015-10-26 11:38:21,446 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [7ed18269] Command
'TimeBoundPollVDSCommand(HostName = infraesx2,
TimeBoundPollVDSCommandParameters:{runAsync='true',
hostId='e2931140-c6c1-4d88-81ab-210bbfd23593'})' execution failed:
java.util.concurrent.ExecutionException:
org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection
issues during send request
2015-10-26 11:38:26,450 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-26 11:38:26,578 INFO
[org.ovirt.engine.core.bll.network.NetworkConfigurator]
(org.ovirt.thread.pool-8-thread-25) [7ed18269] Engine managed to
communicate with VDSM agent on host 'infraesx2'
('e2931140-c6c1-4d88-81ab-210bbfd23593')
2015-10-26 11:38:26,953 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(org.ovirt.thread.pool-8-thread-25) [5f48a291] Lock Acquired to object
'EngineLock:{exclusiveLocks='[e2931140-c6c1-4d88-81ab-210bbfd23593=<HOST_NETWORK,
ACTION_TYPE_FAILED_SETUP_NETWORKS_IN_PROGRESS>]', sharedLocks='null'}'
2015-10-26 11:38:26,974 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(org.ovirt.thread.pool-8-thread-25) [5f48a291] Running command:
HostSetupNetworksCommand internal: true. Entities affected : ID:
e2931140-c6c1-4d88-81ab-210bbfd23593 Type: VDSAction group
CONFIGURE_HOST_NETWORK with role type ADMIN
2015-10-26 11:38:26,976 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [5f48a291] START,
HostSetupNetworksVDSCommand(HostName = infraesx2,
HostSetupNetworksVdsCommandParameters:{runAsync='true',
hostId='e2931140-c6c1-4d88-81ab-210bbfd23593',
vds='Host[infraesx2,e2931140-c6c1-4d88-81ab-210bbfd23593]',
rollbackOnFailure='true', conectivityTimeout='120',
hostNetworkQosSupported='true',
networks='[HostNetwork:{defaultRoute='true', bonding='false',
networkName='ovirtmgmt', nicName='enp2s0f0', vlan='null',
mtu='0',
vmNetwork='true', stp='false', properties='null',
bootProtocol='DHCP',
address='10.206.68.97', netmask='255.255.254.0',
gateway='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]'}), log id:
507e585d
2015-10-26 11:38:26,976 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to
infraesx2.bnglab.psecure.net/10.206.68.97
2015-10-26 11:38:26,981 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [5f48a291] FINISH,
HostSetupNetworksVDSCommand, log id: 507e585d