Re: [Users] PROBLEM - host is restarted aromatically by ovirt and becomes unresponsible

----- Original Message ----- | From: "Ricardo Esteves" <maverick.pt(a)gmail.com&gt; | To: Users(a)ovirt.org | Sent: Wednesday, August 7, 2013 5:43:26 PM | Subject: [Users] PROBLEM - host is restarted aromatically by ovirt and becomes unresponsible | | Good afternoon, | | I've having a problem with my hosts, at least one time per week the host that | has all the VMs running restarts and becomes unresponsible. | | After the restart sent to the ilo by the ovirt engine the host becomes | unresponsible, the fans on the enclosure go up like crazy. | | Then the only way to get the blade up is to stop it using ilo or onboard | administrator, | and then remove it from the enclosure and put it back in and then issue the | start using ovirt gui, because using stop/start on the ilo or onboard | administrator the blade powers up but becomes unresponsible, doesn't show | any image or any boot post messages. | | | Anyone else seen this problem before? | | BLADE ENCLOSURE: HP BladeSystem c3000 | BLADES: HP BL460c G6 | OS: CentOS 6.4 (64 bits) | OVIRT: 3.2 | | | engine.log: | | 2013-08-07 14:38:47,256 INFO | [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] Running command: | SetStoragePoolStatusCommand interna | l: true. Entities affected : ID: 06951dba-556b-4323-9356-819c9160fe8e Type: | StoragePool | 2013-08-07 14:38:47,257 ERROR | [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] | (DefaultQuartzScheduler_Worker-8) vds::refreshVdsStats Failed getVdsStats, | vds = 44d77dcb-b775-4aef-ae59- | 1dea8d5c691a : blade5, error = VDSNetworkException: | java.net.NoRouteToHostException: No route to host | 2013-08-07 14:38:47,263 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] | (DefaultQuartzScheduler_Worker-8) | ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = | 44d77dcb-b77 | 5-4aef-ae59-1dea8d5c691a : blade5, VDS Network Error, continuing. | java.net.NoRouteToHostException: No route to host | 2013-08-07 14:38:50,252 ERROR | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] | IrsBroker::Failed::GetStoragePoolInfoVDS due to: NoRout | eToHostException: No route to host | 2013-08-07 14:38:50,253 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] | (DefaultQuartzScheduler_Worker-10) [2b4cb7c5] | ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = | 44d77dcb-b775-4aef-ae59-1dea8d5c691a : blade5, VDS Network Error, continuing. | java.net.NoRouteToHostException: No route to host | 2013-08-07 14:38:53,252 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] Irs placed on server | 44d77dcb-b775-4aef-ae59-1dea8d5c69 | 1a failed. Proceed Failover | 2013-08-07 14:38:53,254 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] | (DefaultQuartzScheduler_Worker-4) VDS::handleNetworkException Server failed | to respond, vds_id = 44d77dcb-b775-4aef | -ae59-1dea8d5c691a, vds_name = blade5, error = | java.net.NoRouteToHostException: No route to host | 2013-08-07 14:38:53,296 INFO [org.ovirt.engine.core.bll.VdsEventListener] | (pool-3-thread-47) ResourceManager::vdsNotResponding entered for Host | 44d77dcb-b775-4aef-ae59-1dea8d5c691a, 192.168. | 10.25 | 2013-08-07 14:38:53,299 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] hostFromVds::selectedVds - | blade6, spmStatus Free, stor | age pool VI-DataCenter | 2013-08-07 14:38:53,308 ERROR | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] SPM Init: could not find | reported vds or not up - pool: | VI-DataCenter vds_spm_id: 1 | 2013-08-07 14:38:53,346 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] SPM selection - vds seems as | spm blade5 | 2013-08-07 14:38:53,355 WARN | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-7) [3b761f63] spm vds is non responsive, | stopping spm selection. | 2013-08-07 14:38:53,438 INFO [org.ovirt.engine.core.bll.FenceExecutor] | (pool-3-thread-47) Using Host blade6 from CLUSTER as proxy to execute | Restart command on Host blade5 | 2013-08-07 14:38:53,438 INFO [org.ovirt.engine.core.bll.FenceExecutor] | (pool-3-thread-47) Executing <Status> Power Management command, Proxy | Host:blade6, Agent:ilo, Target Host:blade5, Manag | ement IP:ilo5.vi.pt, User:Administrator, Options:secure=true | 2013-08-07 14:38:53,457 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] | (pool-3-thread-47) START, FenceVdsVDSCommand(HostName = blade6, HostId = | 2530f498-6029-496a-ab42-9 | 24ca2e3eb7f, targetVdsId = 44d77dcb-b775-4aef-ae59-1dea8d5c691a, action = | Status, ip = ilo5.vi.pt, port = , type = ilo, user = Administrator, password | = ******, options = 'secure=true'), log | id: 41a729f3 | 2013-08-07 14:39:02,533 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] | (pool-3-thread-47) FINISH, FenceVdsVDSCommand, return: Test Succeeded, Host | Status is: on, log id: | 41a729f3 | 2013-08-07 14:39:02,541 INFO | [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] | (pool-3-thread-47) Running command: VdsNotRespondingTreatmentCommand | internal: true. Entities affect | ed : ID: 44d77dcb-b775-4aef-ae59-1dea8d5c691a Type: VDS | 2013-08-07 14:39:02,598 INFO [org.ovirt.engine.core.bll.StopVdsCommand] | (pool-3-thread-47) [56fa00a1] Running command: StopVdsCommand internal: | true. Entities affected : ID: 44d77dcb-b775-4 | aef-ae59-1dea8d5c691a Type: VDS | 2013-08-07 14:39:02,619 INFO | [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-47) | [56fa00a1] START, SetVdsStatusVDSCommand(HostName = blade5, HostId = | 44d77dcb-b775-4a | ef-ae59-1dea8d5c691a, status=Reboot, nonOperationalReason=NONE), log id: | 20a49440 | 2013-08-07 14:39:02,622 INFO | [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-47) | [56fa00a1] VDS blade5 is spm and moved from up calling ResetIrs. | 2013-08-07 14:39:02,622 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] | (pool-3-thread-47) [56fa00a1] START, ResetIrsVDSCommand( storagePoolId = | 06951dba-556b-4323-9356-8 | 19c9160fe8e, ignoreFailoverLimit = false, compatabilityVersion = null, vdsId | = 44d77dcb-b775-4aef-ae59-1dea8d5c691a, ignoreStopFailed = false), log id: | 3b546d0a | 2013-08-07 14:39:02,643 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] START, SpmStopVDSCommand(HostName = blade5, | HostId = 44d77dcb-b775-4a | ef-ae59-1dea8d5c691a, storagePoolId = 06951dba-556b-4323-9356-819c9160fe8e), | log id: 283e8812 | 2013-08-07 14:39:02,644 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] SpmStopVDSCommand:: vds blade5 is in Reboot | status - not performing s | pm stop, pool id 06951dba-556b-4323-9356-819c9160fe8e | 2013-08-07 14:39:02,644 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] FINISH, SpmStopVDSCommand, log id: 283e8812 | 2013-08-07 14:39:02,645 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] | (pool-3-thread-47) [56fa00a1] FINISH, ResetIrsVDSCommand, log id: 3b546d0a | 2013-08-07 14:39:02,645 INFO | [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-47) | [56fa00a1] FINISH, SetVdsStatusVDSCommand, log id: 20a49440 | 2013-08-07 14:39:02,699 INFO [org.ovirt.engine.core.bll.FenceExecutor] | (pool-3-thread-47) [56fa00a1] Using Host blade6 from CLUSTER as proxy to | execute Stop command on Host blade5 | 2013-08-07 14:39:02,735 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] START, SpmStopVDSCommand(HostName = blade5, | HostId = 44d77dcb-b775-4a | ef-ae59-1dea8d5c691a, storagePoolId = 06951dba-556b-4323-9356-819c9160fe8e), | log id: 6bbeb9ca | 2013-08-07 14:39:02,736 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] SpmStopVDSCommand:: vds blade5 is in Reboot | status - not performing s | pm stop, pool id 06951dba-556b-4323-9356-819c9160fe8e | 2013-08-07 14:39:02,736 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (pool-3-thread-47) [56fa00a1] FINISH, SpmStopVDSCommand, log id: 6bbeb9ca | 2013-08-07 14:39:02,737 INFO [org.ovirt.engine.core.bll.FenceExecutor] | (pool-3-thread-47) [56fa00a1] Executing <Stop> Power Management command, | Proxy Host:blade6, Agent:ilo, Target Host:blad | e5, Management IP:ilo5.vi.pt, User:Administrator, Options:secure=true | 2013-08-07 14:39:02,755 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] | (pool-3-thread-47) [56fa00a1] START, FenceVdsVDSCommand(HostName = blade6, | HostId = 2530f498-6029- | 496a-ab42-924ca2e3eb7f, targetVdsId = 44d77dcb-b775-4aef-ae59-1dea8d5c691a, | action = Stop, ip = ilo5.vi.pt, port = , type = ilo, user = Administrator, | password = ******, options = 'secure=tru | e'), log id: 6f3cc543 | 2013-08-07 14:39:03,388 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-1) [18d8826] hostFromVds::selectedVds - | blade6, spmStatus Free, stora | ge pool VI-DataCenter | 2013-08-07 14:39:03,392 ERROR | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-1) [18d8826] SPM Init: could not find | reported vds or not up - pool:V | I-DataCenter vds_spm_id: 1 | 2013-08-07 14:39:03,411 INFO | [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] | (DefaultQuartzScheduler_Worker-1) [18d8826] SPM selection - vds seems as spm | blade5 | 2013-08-07 14:39:03,430 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (DefaultQuartzScheduler_Worker-1) [18d8826] START, | SpmStopVDSCommand(HostName = blade5, HostId = 44 | d77dcb-b775-4aef-ae59-1dea8d5c691a, storagePoolId = | 06951dba-556b-4323-9356-819c9160fe8e), log id: 5ae3209c | 2013-08-07 14:39:03,431 INFO | [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] | (DefaultQuartzScheduler_Worker-1) [18d8826] SpmStopVDSCommand:: vds blade5 | is in Reboot status - no | t performing spm stop, pool id 06951dba-556b-4323-9356-819c9160fe8e | | Best regards, | Ricardo Esteves. | Ricardo, see: 2013-08-07 14:38:47,257 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) vds::refreshVdsStats Failed getVdsStats, vds = 44d77dcb-b775-4aef-ae59- 1dea8d5c691a : blade5, error = VDSNetworkException: java.net.NoRouteToHostException: No route to host One of your hosts is loosing connectivity. I f you have HA VMs running there, the host will be fenced to allow restarting the VMs in another host.