[ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability

Tue Jan 3 12:05:19 UTC 2017

Here VDSM logs ...

Em ter, 3 de jan de 2017 às 09:58, Rogério Ceni Coelho <
rogeriocenicoelho at gmail.com> escreveu:

> Hi Everyone,
>
> I found a lot of Heartbeat exceeded like below ... I attach some logs ...
>
> I am thinking to do rollback to 4.0.4 and CentOS 7.2 ...
>
> 2016-12-28 03:49:14,962 ERROR
> [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) []
> Internal server error: null
> 2016-12-28 03:49:14,986 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm15-poa.rbs.com.br/10.151.252.235
> 2016-12-28 03:49:14,993 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm06-poa.rbs.com.br/10.151.252.226
> 2016-12-28 03:49:14,997 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm08-poa.rbs.com.br/10.151.252.228
> 2016-12-28 03:49:15,001 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm03-poa.rbs.com.br/10.151.252.30
> 2016-12-28 03:49:15,006 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm17-poa.rbs.com.br/10.151.252.237
> 2016-12-28 03:49:15,012 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm02-poa.rbs.com.br/10.151.252.223
> 2016-12-28 03:49:15,018 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (DefaultQuartzScheduler11) [36d1c163] Command
> 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm07-poa.rbs.com.br,
> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> hostId='1f128680-6152-4273-bb1d-8be545b43461', vds='Host[
> prd-rbs-ovirt-kvm07-poa.rbs.com.br,1f128680-6152-4273-bb1d-8be545b43461]'})'
> execution failed: VDSGenericException: VDSNetworkException: Heartbeat
> exceeded
> 2016-12-28 03:49:15,018 INFO
>  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]
> (DefaultQuartzScheduler11) [36d1c163] Failed to fetch vms info for host '
> prd-rbs-ovirt-kvm07-poa.rbs.com.br' - skipping VMs monitoring.
> 2016-12-28 03:49:15,018 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager]
> (org.ovirt.thread.pool-6-thread-8) [36d1c163] Host '
> prd-rbs-ovirt-kvm07-poa.rbs.com.br' is not responding. It will stay in
> Connecting state for a grace period of 62 seconds and after that an attempt
> to fence the host will be issued.
> 2016-12-28 03:49:15,025 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (DefaultQuartzScheduler9) [13bbd638] Command
> 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm10-poa.rbs.com.br,
> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> hostId='8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a', vds='Host[
> prd-rbs-ovirt-kvm10-poa.rbs.com.br,8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a]'})'
> execution failed: VDSGenericException: VDSNetworkException: Heartbeat
> exceeded
> 2016-12-28 03:49:15,025 INFO
>  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]
> (DefaultQuartzScheduler9) [13bbd638] Failed to fetch vms info for host '
> prd-rbs-ovirt-kvm10-poa.rbs.com.br' - skipping VMs monitoring.
> 2016-12-28 03:49:15,026 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm05-poa.rbs.com.br/10.151.252.225
> 2016-12-28 03:49:15,027 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler32) [5b86105] Correlation ID: null, Call Stack:
> null, Custom Event ID: -1, Message: VDSM
> prd-rbs-ovirt-kvm03-poa.rbs.com.br command failed: Heartbeat exceeded
> 2016-12-28 03:49:15,027 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
> (DefaultQuartzScheduler32) [5b86105] Command 'SpmStatusVDSCommand(HostName =
>  prd-rbs-ovirt-kvm03-poa.rbs.com.br,
> SpmStatusVDSCommandParameters:{runAsync='true',
> hostId='f7842244-646c-400a-9736-f8d4aa9b1cef',
> storagePoolId='98867d75-9c43-46b4-891a-ff3a5eb0f06e'})' execution failed:
> VDSGenericException: VDSNetworkException: Heartbeat exceeded
> 2016-12-28 03:49:15,027 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager]
> (org.ovirt.thread.pool-6-thread-10) [5b86105] Host '
> prd-rbs-ovirt-kvm03-poa.rbs.com.br' is not responding. It will stay in
> Connecting state for a grace period of 81 seconds and after that an attempt
> to fence the host will be issued.
> 2016-12-28 03:49:15,029 INFO
>  [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand]
> (DefaultQuartzScheduler32) [613ae8a1] Running command:
> SetStoragePoolStatusCommand internal: true. Entities affected :  ID:
> 98867d75-9c43-46b4-891a-ff3a5eb0f06e Type: StoragePool
> 2016-12-28 03:49:15,030 INFO
>  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper]
> (DefaultQuartzScheduler32) [613ae8a1] Storage Pool
> '98867d75-9c43-46b4-891a-ff3a5eb0f06e' - Updating Storage Domain
> '7b8c9293-f103-401a-93ac-550981837224' status from 'Active' to 'Unknown',
> reason: null
> 2016-12-28 03:49:15,030 INFO
>  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper]
> (DefaultQuartzScheduler32) [613ae8a1] Storage Pool
> '98867d75-9c43-46b4-891a-ff3a5eb0f06e' - Updating Storage Domain
> 'cfdbbda4-bd72-4c58-af73-8aa89d62ff01' status from 'Active' to 'Unknown',
> reason: null
> 2016-12-28 03:49:15,030 INFO
>  [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper]
> (DefaultQuartzScheduler32) [613ae8a1] Storage Pool
> '98867d75-9c43-46b4-891a-ff3a5eb0f06e' - Updating Storage Domain
> '0b5015d2-8f05-44c5-9e5a-d732b0b0e419' status from 'Active' to 'Unknown',
> reason: null
> 2016-12-28 03:49:15,031 INFO
>  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to prd-rbs-ovirt-kvm13-poa.rbs.com.br/10.151.252.233
> 2016-12-28 03:49:15,033 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (DefaultQuartzScheduler21) [b33cc28] Command
> 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br,
> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
> hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[
> prd-rbs-ovirt-kvm19-poa.rbs.com.br,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})'
> execution failed: VDSGenericException: VDSNetworkException: Heartbeat
> exceeded
>
>
>
> Em seg, 2 de jan de 2017 às 17:18, Yaniv Kaul <ykaul at redhat.com> escreveu:
>
> On Mon, Jan 2, 2017 at 8:51 PM, Rogério Ceni Coelho <
> rogeriocenicoelho at gmail.com> wrote:
>
> Hi oVirt Gurus,
>
>
> Happy new year to everyone !!!
>
>
>
> I update oVirt Engine to 4.0.5 from 4.0.4 and Centos to 7.3 from 7.2 last
> week and after that I have instability four times. Every time ovirt engine
> seems to loose communication with one or more node servers like this image
> below. Every time I rebooted oVirt engine server and everything came back
> to normal.
>
>
> It'd be great if you could share logs - engine.log from the Engine and
> vdsm.log from the host(s).
> Y.
>
>
>
>
> Anyone with this kind of problem ???
>
>
>
> [image: pasted1]
>
>
> After reboot :
>
>
>
> [image: pasted2]
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170103/595be2ab/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pasted2
Type: image/png
Size: 166275 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170103/595be2ab/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pasted1
Type: image/png
Size: 174071 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170103/595be2ab/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.log.72.xz
Type: application/octet-stream
Size: 961364 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170103/595be2ab/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.log.71.xz
Type: application/octet-stream
Size: 931212 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170103/595be2ab/attachment-0003.obj>