[ovirt-users] hosted-engine HA: Engine dying unexpectedly

Wed Oct 22 08:52:25 UTC 2014

On 22.10.2014 10:17, Martin Sivak wrote:
> Hi,
Hello Martin,
>
> I think there is something weird going on with your storage, this is the crash snippet from the host that had the engine at the beginning:
This is what I suspected also; though its hard to track down atm,
because when I cecked NFS was accessible just fine from both hosts. In
the end I have a snippet from /var/log/messages; I suppose it all starts
with the sanlock warnings?
Also I suppose:
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state

this actually the HE being killed?

>
> /var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21 20:22:33,919::task::866::Storage.TaskManager.Task::(_setError) Task=`2ad31974-e1fc-4785-9423-ff3bd087a5aa`::Unexpected error
> /var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21 20:22:33,934::dispatcher::79::Storage.Dispatcher::(wrapper) Connection timed out
> /var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21 20:23:00,733::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 68aad705-7c9b-427a-a84c-6f32f23675b3
> /var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21 20:23:00,734::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 68aad705-7c9b-427a-a84c-6f32f23675b3
> /var/log/vdsm/vdsm.log:VM Channels Listener::ERROR::2014-10-21 20:23:04,258::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 53
>
> The second host's VDSM lost the connection to storage domain at the same time..
>
> 20:23:09,950::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host 192.168.50.201 (id 1)
> 20:23:12,365::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING
>
> The engine VM was restarted right after the connection was restored:
>
> 20:25:54,336::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING
> 20:26:20,572::hosted_engine::571::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Acquired lock on host id 2
> 20:26:20,572::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400)
> 20:26:20,572::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 192.168.50.201 (id: 1, score: 2400)
> 20:26:30,606::states::459::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (2400), attempting to start engine VM
>
> ...
>
> 20:27:34,423::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineStarting'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUp'>
> 20:27:34,430::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1413916054.43 type=state_transition detail=EngineStarting-EngineUp hostname='nodehv02.lab.mbox.loc'
> 20:27:34,498::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineUp) sent? sent
> 20:27:38,481::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)
>
> All was then well till the end of the log.
>
> 20:29:53,393::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost
> 20:29:55,372::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)
> 20:29:55,372::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 192.168.50.201 (id: 1, score: 0)
>
>
> Hosted engine had nothing to do with the engine crash according to the log. On the contrary, it properly re-started the VM once the cluster recovered from the storage issue.
Good to know :)
>
> Can you give us more information about the setup? Storage type, topology, ...
Nothing fancy really. Hardware is old, though. Shared storage for engine
is NFS; some nfs, iscsi and fc domains are used for VMs. I have had a
crash of the NFS service; but all this might be a switch behaving badly
in high traffic situations in the end.
>
> --
> Martin Sivák
> msivak at redhat.com
> Red Hat Czech
> RHEV-M SLA / Brno, CZ
>
> ----- Original Message -----
>> Hello,
>>
>> since upgrading to the latest hosted-engine-ha I have the follwing problem:
>>
>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
>> Engine vm died unexpectedly
>>
>> I suppose HA is forcing the engine down because liveliness check is
>> failing. I attached a log compile from the latest incident, 2014-10-21
>> 16:26:31,836. The 'host' logs are from the hosts the engine was running
>> on, host2 the other HA host
>> Interestingly this only happens when I was connected via a VNC console
>> to one of my Winodws 2012 VMs.
>>
>>
>> How can I further debug this?
>> The engine log seems empty and also the HE does not seem to have any
>> trouble when this happens. As precaustion / test I set my cluster to
>> global maintenance.
>>
>> Thanks,
>>
>> vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch
>> vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch
>> vdsm-4.16.7-1.gitdb83943.el6.x86_64
>> vdsm-python-4.16.7-1.gitdb83943.el6.noarch
>> vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch
>> vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch
>> vdsm-cli-4.16.7-1.gitdb83943.el6.noarch
>>
>> ovirt-hosted-engine-ha-1.2.4-1.el6.noarch
>> ovirt-release35-001-1.noarch
>> ovirt-host-deploy-1.3.0-1.el6.noarch
>> ovirt-hosted-engine-setup-1.2.1-1.el6.noarch
>> ovirt-release34-1.0.3-1.noarch
>> ovirt-engine-sdk-python-3.5.0.7-1.el6.noarch
>>
>>
>> --
>> Daniel Helgenberger
>> m box bewegtbild GmbH
>>
>> P: +49/30/2408781-22
>> F: +49/30/2408781-10
>>
>> ACKERSTR. 19
>> D-10115 BERLIN
>>
>>
>> www.m-box.de  www.monkeymen.tv
>>
>> Geschäftsführer: Martin Retschitzegger / Michaela Göllner
>> Handeslregister: Amtsgericht Charlottenburg / HRB 112767
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074
[23864]: s19 renewal error -202 delta_length 10 last_success 160043
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:10 nodehv01 sanlock[2946]: 2014-10-21 20:22:10+0200 160094
[2946]: s20 check_our_lease warning 60 last_success 160034
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160095
[2946]: s20 check_our_lease warning 61 last_success 160034
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:12 nodehv01 sanlock[2946]: 2014-10-21 20:22:12+0200 160096
[2946]: s20 check_our_lease warning 62 last_success 160034
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[2946]: s20 check_our_lease warning 63 last_success 160034
Oct 21 20:22:14 nodehv01 sanlock[2946]: 2014-10-21 20:22:14+0200 160098
[2946]: s20 check_our_lease warning 64 last_success 160034
Oct 21 20:22:15 nodehv01 sanlock[2946]: 2014-10-21 20:22:15+0200 160099
[2946]: s20 check_our_lease warning 65 last_success 160034
Oct 21 20:22:16 nodehv01 sanlock[2946]: 2014-10-21 20:22:16+0200 160100
[2946]: s20 check_our_lease warning 66 last_success 160034
Oct 21 20:22:17 nodehv01 sanlock[2946]: 2014-10-21 20:22:17+0200 160101
[2946]: s20 check_our_lease warning 67 last_success 160034
Oct 21 20:22:18 nodehv01 sanlock[2946]: 2014-10-21 20:22:18+0200 160102
[2946]: s20 check_our_lease warning 68 last_success 160034
Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103
[2946]: s20 check_our_lease warning 69 last_success 160034
Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103
[2946]: s19 check_our_lease warning 60 last_success 160043
Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104
[2946]: s20 check_our_lease warning 70 last_success 160034
Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104
[2946]: s19 check_our_lease warning 61 last_success 160043
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s21 check_our_lease warning 60 last_success 160045
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s20 check_our_lease warning 71 last_success 160034
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s19 check_our_lease warning 62 last_success 160043
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s21 check_our_lease warning 61 last_success 160045
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s20 check_our_lease warning 72 last_success 160034
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s19 check_our_lease warning 63 last_success 160043
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s21 check_our_lease warning 62 last_success 160045
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s20 check_our_lease warning 73 last_success 160034
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s19 check_our_lease warning 64 last_success 160043
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s21 check_our_lease warning 63 last_success 160045
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s20 check_our_lease warning 74 last_success 160034
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s19 check_our_lease warning 65 last_success 160043
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s21 check_our_lease warning 64 last_success 160045
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s20 check_our_lease warning 75 last_success 160034
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s19 check_our_lease warning 66 last_success 160043
Oct 21 20:22:25 nodehv01 wdmd[2934]: test warning now 160110 ping 160100
close 145768 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:25 nodehv01 wdmd[2934]: /dev/watchdog closed unclean
Oct 21 20:22:25 nodehv01 kernel: iTCO_wdt: Unexpected close, not
stopping watchdog!
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s21 check_our_lease warning 65 last_success 160045
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s20 check_our_lease warning 76 last_success 160034
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s19 check_our_lease warning 67 last_success 160043
Oct 21 20:22:26 nodehv01 wdmd[2934]: test warning now 160111 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s21 check_our_lease warning 66 last_success 160045
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s20 check_our_lease warning 77 last_success 160034
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s19 check_our_lease warning 68 last_success 160043
Oct 21 20:22:27 nodehv01 wdmd[2934]: test warning now 160112 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s21 check_our_lease warning 67 last_success 160045
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s20 check_our_lease warning 78 last_success 160034
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s19 check_our_lease warning 69 last_success 160043
Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s21 check_our_lease warning 68 last_success 160045
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s20 check_our_lease warning 79 last_success 160034
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s19 check_our_lease warning 70 last_success 160043
Oct 21 20:22:29 nodehv01 wdmd[2934]: test warning now 160114 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:29 nodehv01 wdmd[2934]: test failed rem 56 now 160114 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s21 check_our_lease warning 69 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s20 check_our_lease failed 80
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s20 kill 24784 sig 15 count 1
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s19 check_our_lease warning 71 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s21 check_our_lease warning 69 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s19 check_our_lease warning 71 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s20 kill 24784 sig 15 count 2
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:30 nodehv01 wdmd[2934]: test failed rem 55 now 160115 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s20 kill 24784 sig 15 count 3
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:31 nodehv01 wdmd[2934]: test failed rem 54 now 160116 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s20 kill 24784 sig 15 count 4
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:33 nodehv01 wdmd[2934]: test failed rem 53 now 160117 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s20 kill 24784 sig 15 count 5
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[23864]: s19 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[23864]: s19 renewal error -2 delta_length 11 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:34 nodehv01 wdmd[2934]: test failed rem 52 now 160118 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s20 kill 24784 sig 15 count 6
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[24471]: s21 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[24471]: s21 renewal error -2 delta_length 10 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:35 nodehv01 wdmd[2934]: test failed rem 51 now 160119 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s20 kill 24784 sig 15 count 7
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:36 nodehv01 wdmd[2934]: test failed rem 50 now 160120 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s20 kill 24784 sig 15 count 8
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:37 nodehv01 wdmd[2934]: test failed rem 49 now 160121 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s20 kill 24784 sig 15 count 9
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 48 now 160122 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s20 kill 24784 sig 15 count 10
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s19 check_our_lease failed 80
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s19 all pids clear
Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 47 now 160123 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160123 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s20 kill 24784 sig 15 count 11
Oct 21 20:22:39 nodehv01 wdmd[2934]: test failed rem 46 now 160124 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:39 nodehv01 wdmd[2934]: test warning now 160124 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s21 check_our_lease failed 80
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s20 kill 24784 sig 15 count 12
Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping
160100 close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s21 all pids clear
Oct 21 20:22:41 nodehv01 wdmd[2934]: test failed rem 44 now 160126 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:41 nodehv01 sanlock[2946]: 2014-10-21 20:22:41+0200 160126
[2946]: s20 kill 24784 sig 15 count 13
Oct 21 20:22:42 nodehv01 wdmd[2934]: test failed rem 43 now 160127 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:42 nodehv01 sanlock[2946]: 2014-10-21 20:22:42+0200 160127
[2946]: s20 kill 24784 sig 15 count 14
Oct 21 20:22:43 nodehv01 wdmd[2934]: test failed rem 42 now 160128 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:43 nodehv01 sanlock[2946]: 2014-10-21 20:22:43+0200 160128
[2946]: s20 kill 24784 sig 15 count 15
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128
[23864]: s19 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128
[23864]: s19 renewal error -2 delta_length 10 last_success 160043
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160129
[2946]: s20 kill 24784 sig 15 count 16
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 41 now 160129 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[2946]: s20 kill 24784 sig 15 count 17
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[24471]: s21 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[24471]: s21 renewal error -2 delta_length 10 last_success 160045
Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 40 now 160130 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:46 nodehv01 sanlock[2946]: 2014-10-21 20:22:46+0200 160131
[2946]: s20 kill 24784 sig 15 count 18
Oct 21 20:22:46 nodehv01 wdmd[2934]: test failed rem 39 now 160131 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:47 nodehv01 sanlock[2946]: 2014-10-21 20:22:47+0200 160132
[2946]: s20 kill 24784 sig 15 count 19
Oct 21 20:22:47 nodehv01 wdmd[2934]: test failed rem 38 now 160132 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:48 nodehv01 sanlock[2946]: 2014-10-21 20:22:48+0200 160133
[2946]: s20 kill 24784 sig 15 count 20
Oct 21 20:22:48 nodehv01 wdmd[2934]: test failed rem 37 now 160133 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:49 nodehv01 sanlock[2946]: 2014-10-21 20:22:49+0200 160134
[2946]: s20 kill 24784 sig 15 count 21
Oct 21 20:22:49 nodehv01 wdmd[2934]: test failed rem 36 now 160134 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:50 nodehv01 sanlock[2946]: 2014-10-21 20:22:50+0200 160135
[2946]: s20 kill 24784 sig 15 count 22
Oct 21 20:22:50 nodehv01 wdmd[2934]: test failed rem 35 now 160135 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:51 nodehv01 sanlock[2946]: 2014-10-21 20:22:51+0200 160136
[2946]: s20 kill 24784 sig 15 count 23
Oct 21 20:22:51 nodehv01 wdmd[2934]: test failed rem 34 now 160136 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:52 nodehv01 sanlock[2946]: 2014-10-21 20:22:52+0200 160137
[2946]: s20 kill 24784 sig 15 count 24
Oct 21 20:22:52 nodehv01 wdmd[2934]: test failed rem 33 now 160137 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:53 nodehv01 sanlock[2946]: 2014-10-21 20:22:53+0200 160138
[2946]: s20 kill 24784 sig 15 count 25
Oct 21 20:22:53 nodehv01 wdmd[2934]: test failed rem 32 now 160138 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 1 0x7ffd08000910 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 2 0x7ffd08000960 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160139
[2946]: s20 kill 24784 sig 15 count 26
Oct 21 20:22:54 nodehv01 wdmd[2934]: test failed rem 31 now 160139 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[2946]: s20 kill 24784 sig 15 count 27
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 1 0x7ffcec000910 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 2 0x7ffcec000960 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 3 0x7ffcec0009b0 busy
Oct 21 20:22:55 nodehv01 wdmd[2934]: test failed rem 30 now 160140 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:56 nodehv01 sanlock[2946]: 2014-10-21 20:22:56+0200 160141
[2946]: s20 kill 24784 sig 15 count 28
Oct 21 20:22:56 nodehv01 wdmd[2934]: test failed rem 29 now 160141 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:57 nodehv01 sanlock[2946]: 2014-10-21 20:22:57+0200 160142
[2946]: s20 kill 24784 sig 15 count 29
Oct 21 20:22:57 nodehv01 wdmd[2934]: test failed rem 28 now 160142 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:58 nodehv01 sanlock[2946]: 2014-10-21 20:22:58+0200 160143
[2946]: s20 kill 24784 sig 15 count 30
Oct 21 20:22:58 nodehv01 wdmd[2934]: test failed rem 27 now 160143 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160143
[24471]: hosted-e aio collect
0x7ffcec0009b0:0x7ffcec0009c0:0x7ffd000e2000 result 1048576:0 close free
Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160144
[2946]: s20 kill 24784 sig 15 count 31
Oct 21 20:22:59 nodehv01 wdmd[2934]: test failed rem 26 now 160144 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:00 nodehv01 sanlock[2946]: 2014-10-21 20:23:00+0200 160145
[2946]: s20 kill 24784 sig 15 count 32
Oct 21 20:23:00 nodehv01 wdmd[2934]: test failed rem 25 now 160145 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146
[2946]: s20 kill 24784 sig 15 count 33
Oct 21 20:23:01 nodehv01 wdmd[2934]: test failed rem 24 now 160146 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146
[23864]: e3b7a377 aio collect
0x7ffd08000960:0x7ffd08000970:0x7ffd005ec000 result 1048576:0 close free
Oct 21 20:23:02 nodehv01 sanlock[2946]: 2014-10-21 20:23:02+0200 160147
[2946]: s20 kill 24784 sig 15 count 34
Oct 21 20:23:03 nodehv01 wdmd[2934]: test failed rem 23 now 160147 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160147
[24471]: hosted-e aio collect
0x7ffcec000960:0x7ffcec000970:0x7ffd003e8000 result 1048576:0 close free
Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160148
[2946]: s20 kill 24784 sig 15 count 35
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:04 nodehv01 wdmd[2934]: /dev/watchdog reopen
Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148
[2946]: dead 24784 ci 5 count 35
Oct 21 20:23:04 nodehv01 vdsm vds ERROR Received 00000011 on fileno 53
Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148
[2946]: s20 all pids clear
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleDisk at 0x2c4c110>#012Traceback (most
recent call last):#012  File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 323, in _sampleDisk#012   
diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)#012 
File "/usr/share/vdsm/virt/vm.py", line 670, in f#012    ret =
attr(*args, **kwargs)#012  File
"/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111,
in wrapper#012    ret = f(*args, **kwargs)#012  File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 1838, in
blockStats#012    if ret is None: raise libvirtError
('virDomainBlockStats() failed', dom=self)#012libvirtError: Unable to
read from monitor: Connection reset by peer
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleDiskLatency at 0x2c4a930>#012Traceback
(most recent call last):#012  File "/usr/share/vdsm/virt/sampling.py",
line 471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 337, in _sampleDiskLatency#012   
diskLatency[vmDrive.name] =
self._vm._dom.blockStatsFlags(#012AttributeError: 'NoneType' object has
no attribute 'blockStatsFlags'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleNet at 0x2c4c180>#012Traceback (most
recent call last):#012  File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 344, in _sampleNet#012   
netSamples[nic.name] =
self._vm._dom.interfaceStats(nic.name)#012AttributeError: 'NoneType'
object has no attribute 'interfaceStats'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleBalloon at 0x2c4c1f0>#012Traceback (most
recent call last):#012  File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 352, in _sampleBalloon#012    infos =
self._vm._dom.info()#012AttributeError: 'NoneType' object has no
attribute 'info'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleVcpuPinning at 0x2c4a9b0>#012Traceback
(most recent call last):#012  File "/usr/share/vdsm/virt/sampling.py",
line 471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 348, in _sampleVcpuPinning#012   
vCpuInfos = self._vm._dom.vcpus()#012AttributeError: 'NoneType' object
has no attribute 'vcpus'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleCpuTune at 0x2c4c2d0>#012Traceback (most
recent call last):#012  File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012    statsFunction()#012  File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012   
retValue = self._function(*args, **kwargs)#012  File
"/usr/share/vdsm/virt/vm.py", line 359, in _sampleCpuTune#012    infos =
self._vm._dom.schedulerParameters()#012AttributeError: 'NoneType' object
has no attribute 'schedulerParameters'
Oct 21 20:23:05 nodehv01 ntpd[3047]: Deleting interface #43 vnet3,
fe80::fc16:3eff:fe74:4156#123, interface stats: received=0, sent=0,
dropped=0, active_time=14116 secs
Oct 21 20:23:05 nodehv01 ntpd[3047]: peers refreshed
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 aio collect
0x7ffd08000910:0x7ffd08000920:0x7ffd008f2000 result 1048576:0 close free
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[23969]: bcfa7ec4 aio collect
0x7ffcf4000960:0x7ffcf4000970:0x7ffd007f0000 result 1048576:0 close free
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[24471]: hosted-e aio collect
0x7ffcec000910:0x7ffcec000920:0x7ffd006ee000 result 1048576:0 close free
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy
Oct 21 20:23:09 nodehv01 sanlock[2946]: 2014-10-21 20:23:09+0200 160154
[23864]: e3b7a377 aio collect
0x7ffd080008c0:0x7ffd080008d0:0x7ffd1005e000 result 1048576:0 close free
Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155
[23969]: bcfa7ec4 aio collect
0x7ffcf4000910:0x7ffcf4000920:0x7ffd009f4000 result 1048576:0 close free
Oct 21 20:23:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155
[24471]: hosted-e aio collect
0x7ffcec0008c0:0x7ffcec0008d0:0x7ffd00af6000 result 1048576:0 close free
Oct 21 20:23:12 nodehv01 sanlock[2946]: 2014-10-21 20:23:12+0200 160157
[23864]: e3b7a377 aio collect
0x7ffd080009b0:0x7ffd080009c0:0x7ffd002e6000 result 1048576:0 close free
Oct 21 20:24:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:25:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:26:05 nodehv01 ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine vm
died unexpectedly
Oct 21 20:26:06 nodehv01 kernel: device-mapper: table: 253:28:
multipath: error getting device

-- 
Daniel Helgenberger
m box bewegtbild GmbH

P: +49/30/2408781-22
F: +49/30/2408781-10

ACKERSTR. 19
D-10115 BERLIN

www.m-box.de  www.monkeymen.tv

Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767