[ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
Pavel Gashev
Pax at acronis.com
Fri Jan 6 20:50:30 UTC 2017
Rogério,
Related discussion: http://lists.ovirt.org/pipermail/devel/2016-October/014018.html
The question here is the same. What is the cause of the heartbeat issues?
Network issues? Firewall? DNS resolver issues?
From: Rogério Ceni Coelho <rogeriocenicoelho at gmail.com>
Date: Friday 6 January 2017 at 23:21
To: Pavel Gashev <Pax at acronis.com>, users <users at ovirt.org>
Subject: Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
My second oVirt enviroment with homolog (DEV) have the same problems ... :-(
[root at hlg-rbs-ovirt01-poa ~]# egrep -i "error" /var/log/ovirt-engine/engine.log | tail -50
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:01:15,606 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [323f3e59] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='7d9cab91-703d-432d-b5da-91e122950794', vds='Host[hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>,7d9cab91-703d-432d-b5da-91e122950794]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:01:15,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [122486ba] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm09-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm09-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cd5e5f1d-6c8b-49ad-a9ab-4f2fc3fedbb8', vds='Host[hlg-rbs-ovirt-kvm09-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm09-poa.rbs.com.br>,cd5e5f1d-6c8b-49ad-a9ab-4f2fc3fedbb8]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:01:15,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [73c03747] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='0ccedcb2-33ab-43be-9218-7743d711bc18', vds='Host[hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>,0ccedcb2-33ab-43be-9218-7743d711bc18]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:01:15,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [4faa7cb9] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f9c9d929-b460-4102-bb29-de1e6ad6ad72', vds='Host[hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>,f9c9d929-b460-4102-bb29-de1e6ad6ad72]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:01:15,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [48f78d41] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45b93272-7bfd-4d9a-b934-75c4a9b9514e', vds='Host[hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>,45b93272-7bfd-4d9a-b934-75c4a9b9514e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:01:15,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [52e555ea] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f527f077-dd27-46d2-bd39-3f157177277e', vds='Host[hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>,f527f077-dd27-46d2-bd39-3f157177277e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:02:21,810 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:02:21,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [a747abd] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm01-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm01-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5feddfba-d7b2-423e-a946-ac2bf36906fa', vds='Host[hlg-rbs-ovirt-kvm01-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm01-poa.rbs.com.br>,5feddfba-d7b2-423e-a946-ac2bf36906fa]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:03:27,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [52e555ea] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='7d9cab91-703d-432d-b5da-91e122950794', vds='Host[hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>,7d9cab91-703d-432d-b5da-91e122950794]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:04:33,936 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:04:33,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [48f78d41] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='0ccedcb2-33ab-43be-9218-7743d711bc18', vds='Host[hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>,0ccedcb2-33ab-43be-9218-7743d711bc18]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:05:40,059 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:05:40,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [5e3bcebd] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='02ead14e-0208-4a74-b1c2-4c19383820f9', vds='Host[hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>,02ead14e-0208-4a74-b1c2-4c19383820f9]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:06:46,193 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:06:46,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [48f78d41] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm09-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm09-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='cd5e5f1d-6c8b-49ad-a9ab-4f2fc3fedbb8', vds='Host[hlg-rbs-ovirt-kvm09-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm09-poa.rbs.com.br>,cd5e5f1d-6c8b-49ad-a9ab-4f2fc3fedbb8]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:07:52,336 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:07:52,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [127eef1a] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f9c9d929-b460-4102-bb29-de1e6ad6ad72', vds='Host[hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>,f9c9d929-b460-4102-bb29-de1e6ad6ad72]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:07:52,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [48f78d41] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='7d9cab91-703d-432d-b5da-91e122950794', vds='Host[hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>,7d9cab91-703d-432d-b5da-91e122950794]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:08:58,402 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [4faa7cb9] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45b93272-7bfd-4d9a-b934-75c4a9b9514e', vds='Host[hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>,45b93272-7bfd-4d9a-b934-75c4a9b9514e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:10:04,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:10:04,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [4faa7cb9] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f527f077-dd27-46d2-bd39-3f157177277e', vds='Host[hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>,f527f077-dd27-46d2-bd39-3f157177277e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:12:41,982 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-88) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt', will send a 404 error response.
2017-01-06 18:13:22,892 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,892 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:13:22,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [127eef1a] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='7d9cab91-703d-432d-b5da-91e122950794', vds='Host[hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>,7d9cab91-703d-432d-b5da-91e122950794]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:13:22,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [40706f05] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='02ead14e-0208-4a74-b1c2-4c19383820f9', vds='Host[hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>,02ead14e-0208-4a74-b1c2-4c19383820f9]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:13:22,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [4cb56911] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm01-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm01-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5feddfba-d7b2-423e-a946-ac2bf36906fa', vds='Host[hlg-rbs-ovirt-kvm01-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm01-poa.rbs.com.br>,5feddfba-d7b2-423e-a946-ac2bf36906fa]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:13:22,909 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [52e555ea] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f527f077-dd27-46d2-bd39-3f157177277e', vds='Host[hlg-rbs-ovirt-kvm05-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm05-poa.rbs.com.br>,f527f077-dd27-46d2-bd39-3f157177277e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:13:22,910 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [643a4714] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f9c9d929-b460-4102-bb29-de1e6ad6ad72', vds='Host[hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>,f9c9d929-b460-4102-bb29-de1e6ad6ad72]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:13:22,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7bd8bba3] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm04-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm04-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='ffde7cb6-30db-4d77-872d-ef4e1b3d1a8e', vds='Host[hlg-rbs-ovirt-kvm04-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm04-poa.rbs.com.br>,ffde7cb6-30db-4d77-872d-ef4e1b3d1a8e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:14:28,977 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:14:28,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [7bd8bba3] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45b93272-7bfd-4d9a-b934-75c4a9b9514e', vds='Host[hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>,45b93272-7bfd-4d9a-b934-75c4a9b9514e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:15:35,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [e367cca] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='7d9cab91-703d-432d-b5da-91e122950794', vds='Host[hlg-rbs-ovirt-kvm08-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm08-poa.rbs.com.br>,7d9cab91-703d-432d-b5da-91e122950794]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:16:41,200 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [48f78d41] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45b93272-7bfd-4d9a-b934-75c4a9b9514e', vds='Host[hlg-rbs-ovirt-kvm06-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm06-poa.rbs.com.br>,45b93272-7bfd-4d9a-b934-75c4a9b9514e]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:17:47,309 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:17:47,309 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:17:47,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [2cd75744] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f9c9d929-b460-4102-bb29-de1e6ad6ad72', vds='Host[hlg-rbs-ovirt-kvm02-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm02-poa.rbs.com.br>,f9c9d929-b460-4102-bb29-de1e6ad6ad72]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:17:47,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [4cb56911] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='0ccedcb2-33ab-43be-9218-7743d711bc18', vds='Host[hlg-rbs-ovirt-kvm07-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm07-poa.rbs.com.br>,0ccedcb2-33ab-43be-9218-7743d711bc18]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 18:18:53,497 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 18:18:53,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [36350da2] Command 'GetAllVmStatsVDSCommand(HostName = hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='02ead14e-0208-4a74-b1c2-4c19383820f9', vds='Host[hlg-rbs-ovirt-kvm03-poa.rbs.com.br<http://hlg-rbs-ovirt-kvm03-poa.rbs.com.br>,02ead14e-0208-4a74-b1c2-4c19383820f9]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
[root at hlg-rbs-ovirt01-poa ~]#
[asted3]
Em sex, 6 de jan de 2017 às 17:02, Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>> escreveu:
Hi Pavel,
I do not have a host with non-responsive status as you can see below. But the symptons are the same ... :-(
[root at prd-rbs-ovirt01-poa ~]# rpm -qa | grep -i vdsm-jsonrpc
vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch
I have some nodes running 4.0.4. This can be a problem ?
[asted2]
[asted1]
Em sex, 6 de jan de 2017 às 16:55, Pavel Gashev <Pax at acronis.com<mailto:Pax at acronis.com>> escreveu:
Rogério,
This bug is fixed in vdsm-jsonrpc-java-1.2.9.
Do you have a really non-responsive host?
From: Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>>
Date: Friday 6 January 2017 at 21:35
To: Pavel Gashev <Pax at acronis.com<mailto:Pax at acronis.com>>, users <users at ovirt.org<mailto:users at ovirt.org>>
Subject: Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
I found https://bugzilla.redhat.com/show_bug.cgi?id=1393714 which seem to be the same problem that occour with me ...
Em sex, 6 de jan de 2017 às 16:28, Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>> escreveu:
Pavel,
Take a look ...
[root at prd-rbs-ovirt01-poa ~]# grep -i error /var/log/ovirt-engine/engine.log | grep -v org.ovirt.engine.core.vdsbroker.HostDevListByCapsVDSCommand | tail -50
2017-01-06 16:20:06,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [a6fc972] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a', vds='Host[prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>,8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:20:06,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [19b85a77] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='795b917b-ea5b-499d-80c5-aa3aad4f2537', vds='Host[prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>,795b917b-ea5b-499d-80c5-aa3aad4f2537]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:20:06,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [1bccfc8a] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:23:26,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [7771902b] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', vds='Host[prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>,23b750ed-4ba6-4782-a9bd-c018c0f36e44]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:23:26,043 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "e5f7a478-841d-413c-baa1-d63632be7748"
2017-01-06 16:24:32,480 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "75bf7e9a-fb5f-4253-bdf6-ff0fcbf8c876"
2017-01-06 16:24:32,485 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [35cc958c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br> command failed: Heartbeat exceeded
2017-01-06 16:24:32,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [35cc958c] Command 'SpmStatusVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, SpmStatusVDSCommandParameters:{runAsync='true', hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', storagePoolId='00000001-0001-0001-0001-000000000198'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:24:32,524 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [152cde35] Correlation ID: 152cde35, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center RBS. Setting Data Center status to Non Responsive (On host prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, Error: Network error during communication with the Host.).
2017-01-06 16:25:37,493 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:25:37,495 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [44e1d6ed] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm03-poa.rbs.com.br<http://prd-rbs-ovirt-kvm03-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f7842244-646c-400a-9736-f8d4aa9b1cef', vds='Host[prd-rbs-ovirt-kvm03-poa.rbs.com.br<http://prd-rbs-ovirt-kvm03-poa.rbs.com.br>,f7842244-646c-400a-9736-f8d4aa9b1cef]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [4fb9c0c9] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='795b917b-ea5b-499d-80c5-aa3aad4f2537', vds='Host[prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>,795b917b-ea5b-499d-80c5-aa3aad4f2537]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [7bc94bc6] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm16-poa.rbs.com.br<http://prd-rbs-ovirt-kvm16-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e2e148b4-00b2-444a-b935-633882e840af', vds='Host[prd-rbs-ovirt-kvm16-poa.rbs.com.br<http://prd-rbs-ovirt-kvm16-poa.rbs.com.br>,e2e148b4-00b2-444a-b935-633882e840af]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,500 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [19b85a77] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm08-poa.rbs.com.br<http://prd-rbs-ovirt-kvm08-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='91dde882-0c86-4330-a206-499275557534', vds='Host[prd-rbs-ovirt-kvm08-poa.rbs.com.br<http://prd-rbs-ovirt-kvm08-poa.rbs.com.br>,91dde882-0c86-4330-a206-499275557534]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [1bccfc8a] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a', vds='Host[prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>,8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [152cde35] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='c4a21d0b-f003-4ee1-8b6b-2b26671d410f', vds='Host[prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>,c4a21d0b-f003-4ee1-8b6b-2b26671d410f]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [2c0caad] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm12-poa.rbs.com.br<http://prd-rbs-ovirt-kvm12-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='46a28187-ab69-4871-9182-24dd910d2784', vds='Host[prd-rbs-ovirt-kvm12-poa.rbs.com.br<http://prd-rbs-ovirt-kvm12-poa.rbs.com.br>,46a28187-ab69-4871-9182-24dd910d2784]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:25:37,558 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "c5556060-f18c-4021-a4b2-8300c9137133"
2017-01-06 16:26:43,588 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,588 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:26:43,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [585c6b32] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm09-poa.rbs.com.br<http://prd-rbs-ovirt-kvm09-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='da563ca2-eb39-451d-8ee8-20853b87d341', vds='Host[prd-rbs-ovirt-kvm09-poa.rbs.com.br<http://prd-rbs-ovirt-kvm09-poa.rbs.com.br>,da563ca2-eb39-451d-8ee8-20853b87d341]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:43,589 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [3acdc247] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm04-poa.rbs.com.br<http://prd-rbs-ovirt-kvm04-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='c0a42c7f-f8fc-4a7e-815a-aed9cf490180', vds='Host[prd-rbs-ovirt-kvm04-poa.rbs.com.br<http://prd-rbs-ovirt-kvm04-poa.rbs.com.br>,c0a42c7f-f8fc-4a7e-815a-aed9cf490180]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:43,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [71319dca] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm15-poa.rbs.com.br<http://prd-rbs-ovirt-kvm15-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='9e2642de-91b2-40ec-aec3-b03be2b00e93', vds='Host[prd-rbs-ovirt-kvm15-poa.rbs.com.br<http://prd-rbs-ovirt-kvm15-poa.rbs.com.br>,9e2642de-91b2-40ec-aec3-b03be2b00e93]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:43,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [2e9a9316] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm02-poa.rbs.com.br<http://prd-rbs-ovirt-kvm02-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='21116080-5e7d-40b4-ba59-056e55eadb5b', vds='Host[prd-rbs-ovirt-kvm02-poa.rbs.com.br<http://prd-rbs-ovirt-kvm02-poa.rbs.com.br>,21116080-5e7d-40b4-ba59-056e55eadb5b]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:43,592 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [f0fd53d] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='c4a21d0b-f003-4ee1-8b6b-2b26671d410f', vds='Host[prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>,c4a21d0b-f003-4ee1-8b6b-2b26671d410f]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:43,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [4fb9c0c9] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm05-poa.rbs.com.br<http://prd-rbs-ovirt-kvm05-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='445946b6-6c1a-4faf-a705-e9ba23261c7f', vds='Host[prd-rbs-ovirt-kvm05-poa.rbs.com.br<http://prd-rbs-ovirt-kvm05-poa.rbs.com.br>,445946b6-6c1a-4faf-a705-e9ba23261c7f]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:39,409 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32ab9c87] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br> command failed: Heartbeat exceeded
2017-01-06 16:26:39,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler10) [32ab9c87] Command 'GetStatsVDSCommand(HostName = prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='c4a21d0b-f003-4ee1-8b6b-2b26671d410f', vds='Host[prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>,c4a21d0b-f003-4ee1-8b6b-2b26671d410f]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:39,411 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [32ab9c87] Failed getting vds stats, host='prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>'(c4a21d0b-f003-4ee1-8b6b-2b26671d410f): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:39,411 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [32ab9c87] Failure to refresh host 'prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>' runtime info: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:26:39,411 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler10) [32ab9c87] Failed to refresh VDS, network error, continuing, vds='prd-rbs-ovirt-kvm01-poa.rbs.com.br<http://prd-rbs-ovirt-kvm01-poa.rbs.com.br>'(c4a21d0b-f003-4ee1-8b6b-2b26671d410f): VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:27:49,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:27:49,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:27:49,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null
2017-01-06 16:27:49,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [54a978b8] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', vds='Host[prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>,23b750ed-4ba6-4782-a9bd-c018c0f36e44]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [27c382ce] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm14-poa.rbs.com.br<http://prd-rbs-ovirt-kvm14-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='60997c2f-3ba0-4911-928a-b70f640f30b8', vds='Host[prd-rbs-ovirt-kvm14-poa.rbs.com.br<http://prd-rbs-ovirt-kvm14-poa.rbs.com.br>,60997c2f-3ba0-4911-928a-b70f640f30b8]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [54a978b8] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm03-poa.rbs.com.br<http://prd-rbs-ovirt-kvm03-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='f7842244-646c-400a-9736-f8d4aa9b1cef', vds='Host[prd-rbs-ovirt-kvm03-poa.rbs.com.br<http://prd-rbs-ovirt-kvm03-poa.rbs.com.br>,f7842244-646c-400a-9736-f8d4aa9b1cef]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [68fa51f0] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm11-poa.rbs.com.br<http://prd-rbs-ovirt-kvm11-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='d3c5cf1f-b203-4818-9767-5249387e3553', vds='Host[prd-rbs-ovirt-kvm11-poa.rbs.com.br<http://prd-rbs-ovirt-kvm11-poa.rbs.com.br>,d3c5cf1f-b203-4818-9767-5249387e3553]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler11) [2d5a3838] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded
2017-01-06 16:27:49,682 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "edd8b375-4598-4671-a2cf-0d62fc9bdd0a"
Em sex, 6 de jan de 2017 às 15:16, Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>> escreveu:
After apply the new vdsm-json package ? Yes, restart hole vm. :-(
[root at prd-rbs-ovirt01-poa ~]# rpm -q vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch
vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch
[root at prd-rbs-ovirt01-poa ~]# rpm -qi vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch
Name : vdsm-jsonrpc-java
Version : 1.2.10
Release : 1.20161211091707.gitaf70e3f.el7.centos
Architecture: noarch
Install Date: Fri Jan 6 09:57:25 2017
Group : Development/Libraries
Size : 144305
License : LGPLv2+
Signature : (none)
Source RPM : vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.src.rpm
Build Date : Sun Dec 11 07:17:40 2016
Build Host : vm0065.workers-phx.ovirt.org<http://vm0065.workers-phx.ovirt.org>
Relocations : (not relocatable)
URL : http://www.ovirt.org
Summary : JsonRpc java client (vdsm-jsonrpc-java) for oVirt
Description :
vdsm jsonrpc java
[root at prd-rbs-ovirt01-poa ~]# uptime
15:15:11 up 19 min, 1 user, load average: 0.50, 0.46, 0.37
[root at prd-rbs-ovirt01-poa ~]# last | head
rogerio_ pts/0 10.40.134.150 Fri Jan 6 15:06 still logged in
reboot system boot 3.10.0-514.2.2.e Fri Jan 6 14:56 - 15:15 (00:19)
rogerio_ pts/0 10.40.134.150 Fri Jan 6 14:46 - 14:55 (00:08)
reboot system boot 3.10.0-514.2.2.e Fri Jan 6 10:11 - 14:55 (04:44)
rogerio_ pts/0 10.40.134.150 Fri Jan 6 09:56 - 10:10 (00:14)
reboot system boot 3.10.0-514.2.2.e Fri Jan 6 09:55 - 10:11 (00:15)
rogerio_ pts/0 10.40.134.150 Fri Jan 6 09:53 - 09:53 (00:00)
reboot system boot 3.10.0-514.2.2.e Thu Jan 5 10:00 - 10:11 (1+00:10)
rogerio_ pts/0 10.40.134.150 Thu Jan 5 09:59 - 09:59 (00:00)
reboot system boot 3.10.0-514.2.2.e Wed Jan 4 14:53 - 10:11 (1+19:18)
[root at prd-rbs-ovirt01-poa ~]#
Em sex, 6 de jan de 2017 às 15:12, Pavel Gashev <Pax at acronis.com<mailto:Pax at acronis.com>> escreveu:
Rogério,
Did you restart the engine?
From: Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>>
Date: Friday 6 January 2017 at 19:56
To: Pavel Gashev <Pax at acronis.com<mailto:Pax at acronis.com>>, users <users at ovirt.org<mailto:users at ovirt.org>>
Subject: Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
Pavel and oVirt Admins,
Problem still the same ... Take a look ...
Error! Filename not specified.
[root at prd-rbs-ovirt01-poa ~]# tail -f /var/log/ovirt-engine/engine.log
2017-01-06 14:24:48,145 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-34) [] Host 'prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>' is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an attempt to fence the host will be issued.
2017-01-06 14:24:48,168 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br> is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an attempt to fence the host will be issued.
2017-01-06 14:26:00,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler17) [] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm07-poa.rbs.com.br<http://prd-rbs-ovirt-kvm07-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='1f128680-6152-4273-bb1d-8be545b43461', vds='Host[prd-rbs-ovirt-kvm07-poa.rbs.com.br<http://prd-rbs-ovirt-kvm07-poa.rbs.com.br>,1f128680-6152-4273-bb1d-8be545b43461]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
2017-01-06 14:26:00,521 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler17) [] Failed to fetch vms info for host 'prd-rbs-ovirt-kvm07-poa.rbs.com.br<http://prd-rbs-ovirt-kvm07-poa.rbs.com.br>' - skipping VMs monitoring.
2017-01-06 14:26:00,521 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-27) [] Host 'prd-rbs-ovirt-kvm07-poa.rbs.com.br<http://prd-rbs-ovirt-kvm07-poa.rbs.com.br>' is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an attempt to fence the host will be issued.
2017-01-06 14:26:00,545 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host prd-rbs-ovirt-kvm07-poa.rbs.com.br<http://prd-rbs-ovirt-kvm07-poa.rbs.com.br> is not responding. It will stay in Connecting state for a grace period of 62 seconds and after that an attempt to fence the host will be issued.
2017-01-06 14:44:37,136 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-97) [] User adm_coelho at rbs.net<mailto:adm_coelho at rbs.net> successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access
2017-01-06 14:44:37,245 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-98) [316f2664] Running command: CreateUserSessionCommand internal: false.
2017-01-06 14:44:39,137 INFO [org.ovirt.engine.docs.utils.servlet.ContextSensitiveHelpMappingServlet] (default task-100) [] Context-sensitive help is not installed. Manual directory doesn't exist: /usr/share/ovirt-engine/manual
2017-01-06 14:44:39,138 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-102) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt', will send a 404 error response.
Em qui, 5 de jan de 2017 às 17:58, Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>> escreveu:
Hi Pavel and oVirt Elves,
Today, I install vdsm-jsonrpc-java. Take a look. Let´s see if problem are solve. Thanks again.
[root at hlg-rbs-ovirt01-poa ~]# rpm -qa | grep -i vdsm-jsonrpc
vdsm-jsonrpc-java-1.2.7-1.el7.centos.noarch
[root at hlg-rbs-ovirt01-poa ~]# wget http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
--2017-01-05 17:52:51-- http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
Resolving resources.ovirt.org<http://resources.ovirt.org> (resources.ovirt.org<http://resources.ovirt.org>)... 66.187.230.28
Connecting to resources.ovirt.org<http://resources.ovirt.org> (resources.ovirt.org<http://resources.ovirt.org>)|66.187.230.28|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 123420 (121K) [application/x-rpm]
Saving to: 'vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm'
100%[=====================================================================================================================================================>] 123,420 202KB/s in 0.6s
2017-01-05 17:52:52 (202 KB/s) - 'vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm' saved [123420/123420]
[root at hlg-rbs-ovirt01-poa ~]# ll vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
-rw-r--r--. 1 root root 123420 Jan 4 22:39 vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
[root at hlg-rbs-ovirt01-poa ~]# rpm -Uvh vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
Preparing... ################################# [100%]
Updating / installing...
1:vdsm-jsonrpc-java-1.2.10-1.201612################################# [ 50%]
Cleaning up / removing...
2:vdsm-jsonrpc-java-1.2.7-1.el7.cen################################# [100%]
[root at hlg-rbs-ovirt01-poa ~]#
Em qua, 4 de jan de 2017 às 17:17, Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>> escreveu:
Wonderfull ... Seems to be the original problem with me on 4.0.5 ... I will try this and do some feedback about soon ...
Thanks !!!!
Em qua, 4 de jan de 2017 às 17:08, Pavel Gashev <Pax at acronis.com<mailto:Pax at acronis.com>> escreveu:
Rogério,
It looks like https://bugzilla.redhat.com/show_bug.cgi?id=1401585
Please try to install the following package, and restart the engine.
http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm
From: <users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> on behalf of Rogério Ceni Coelho <rogeriocenicoelho at gmail.com<mailto:rogeriocenicoelho at gmail.com>>
Date: Monday 2 January 2017 at 21:51
To: users <users at ovirt.org<mailto:users at ovirt.org>>
Subject: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
Hi oVirt Gurus,
Happy new year to everyone !!!
I update oVirt Engine to 4.0.5 from 4.0.4 and Centos to 7.3 from 7.2 last week and after that I have instability four times. Every time ovirt engine seems to loose communication with one or more node servers like this image below. Every time I rebooted oVirt engine server and everything came back to normal.
Anyone with this kind of problem ???
Error! Filename not specified.
After reboot :
Error! Filename not specified.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170106/4f802155/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 246037 bytes
Desc: image001.png
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170106/4f802155/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 230510 bytes
Desc: image002.png
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170106/4f802155/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 301841 bytes
Desc: image003.png
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170106/4f802155/attachment-0005.png>
More information about the Users
mailing list