On 22.10.2014 10:17, Martin Sivak wrote:
Hi,
Hello Martin,
I think there is something weird going on with your storage, this is the crash snippet
from the host that had the engine at the beginning:
This is what I suspected also;
though its hard to track down atm,
because when I cecked NFS was accessible just fine from both hosts. In
the end I have a snippet from /var/log/messages; I suppose it all starts
with the sanlock warnings?
Also I suppose:
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
this actually the HE being killed?
/var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21
20:22:33,919::task::866::Storage.TaskManager.Task::(_setError)
Task=`2ad31974-e1fc-4785-9423-ff3bd087a5aa`::Unexpected error
/var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21
20:22:33,934::dispatcher::79::Storage.Dispatcher::(wrapper) Connection timed out
/var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21
20:23:00,733::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched
domain 68aad705-7c9b-427a-a84c-6f32f23675b3
/var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21
20:23:00,734::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for
domain 68aad705-7c9b-427a-a84c-6f32f23675b3
/var/log/vdsm/vdsm.log:VM Channels Listener::ERROR::2014-10-21
20:23:04,258::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 53
The second host's VDSM lost the connection to storage domain at the same time..
20:23:09,950::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm is running on host 192.168.50.201 (id 1)
20:23:12,365::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
The engine VM was restarted right after the connection was restored:
20:25:54,336::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
20:26:20,572::hosted_engine::571::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock)
Acquired lock on host id 2
20:26:20,572::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
20:26:20,572::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 192.168.50.201 (id: 1, score: 2400)
20:26:30,606::states::459::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine down and local host has best score (2400), attempting to start engine VM
...
20:27:34,423::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Timeout cleared while transitioning <class
'ovirt_hosted_engine_ha.agent.states.EngineStarting'> -> <class
'ovirt_hosted_engine_ha.agent.states.EngineUp'>
20:27:34,430::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1413916054.43 type=state_transition detail=EngineStarting-EngineUp
hostname='nodehv02.lab.mbox.loc'
20:27:34,498::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition (EngineStarting-EngineUp) sent? sent
20:27:38,481::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
All was then well till the end of the log.
20:29:53,393::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm running on localhost
20:29:55,372::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
20:29:55,372::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 192.168.50.201 (id: 1, score: 0)
Hosted engine had nothing to do with the engine crash according to the log. On the
contrary, it properly re-started the VM once the cluster recovered from the storage issue.
Good to know :)
Can you give us more information about the setup? Storage type, topology, ...
Nothing fancy really. Hardware is old, though. Shared storage for engine
is NFS; some nfs, iscsi and fc domains are used for VMs. I have had a
crash of the NFS service; but all this might be a switch behaving badly
in high traffic situations in the end.
--
Martin Sivák
msivak(a)redhat.com
Red Hat Czech
RHEV-M SLA / Brno, CZ
----- Original Message -----
> Hello,
>
> since upgrading to the latest hosted-engine-ha I have the follwing problem:
>
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine vm died unexpectedly
>
> I suppose HA is forcing the engine down because liveliness check is
> failing. I attached a log compile from the latest incident, 2014-10-21
> 16:26:31,836. The 'host' logs are from the hosts the engine was running
> on, host2 the other HA host
> Interestingly this only happens when I was connected via a VNC console
> to one of my Winodws 2012 VMs.
>
>
> How can I further debug this?
> The engine log seems empty and also the HE does not seem to have any
> trouble when this happens. As precaustion / test I set my cluster to
> global maintenance.
>
> Thanks,
>
> vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch
> vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch
> vdsm-4.16.7-1.gitdb83943.el6.x86_64
> vdsm-python-4.16.7-1.gitdb83943.el6.noarch
> vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch
> vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch
> vdsm-cli-4.16.7-1.gitdb83943.el6.noarch
>
> ovirt-hosted-engine-ha-1.2.4-1.el6.noarch
> ovirt-release35-001-1.noarch
> ovirt-host-deploy-1.3.0-1.el6.noarch
> ovirt-hosted-engine-setup-1.2.1-1.el6.noarch
> ovirt-release34-1.0.3-1.noarch
> ovirt-engine-sdk-python-3.5.0.7-1.el6.noarch
>
>
> --
> Daniel Helgenberger
> m box bewegtbild GmbH
>
> P: +49/30/2408781-22
> F: +49/30/2408781-10
>
> ACKERSTR. 19
> D-10115 BERLIN
>
>
>
www.m-box.de www.monkeymen.tv
>
> Geschäftsführer: Martin Retschitzegger / Michaela Göllner
> Handeslregister: Amtsgericht Charlottenburg / HRB 112767
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
>
http://lists.ovirt.org/mailman/listinfo/users
>
Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074
[23864]: s19 renewal error -202 delta_length 10 last_success 160043
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:10 nodehv01 sanlock[2946]: 2014-10-21 20:22:10+0200 160094
[2946]: s20 check_our_lease warning 60 last_success 160034
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160095
[2946]: s20 check_our_lease warning 61 last_success 160034
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:12 nodehv01 sanlock[2946]: 2014-10-21 20:22:12+0200 160096
[2946]: s20 check_our_lease warning 62 last_success 160034
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[23969]: s20 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[23969]: s20 renewal error -202 delta_length 10 last_success 160034
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097
[2946]: s20 check_our_lease warning 63 last_success 160034
Oct 21 20:22:14 nodehv01 sanlock[2946]: 2014-10-21 20:22:14+0200 160098
[2946]: s20 check_our_lease warning 64 last_success 160034
Oct 21 20:22:15 nodehv01 sanlock[2946]: 2014-10-21 20:22:15+0200 160099
[2946]: s20 check_our_lease warning 65 last_success 160034
Oct 21 20:22:16 nodehv01 sanlock[2946]: 2014-10-21 20:22:16+0200 160100
[2946]: s20 check_our_lease warning 66 last_success 160034
Oct 21 20:22:17 nodehv01 sanlock[2946]: 2014-10-21 20:22:17+0200 160101
[2946]: s20 check_our_lease warning 67 last_success 160034
Oct 21 20:22:18 nodehv01 sanlock[2946]: 2014-10-21 20:22:18+0200 160102
[2946]: s20 check_our_lease warning 68 last_success 160034
Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103
[2946]: s20 check_our_lease warning 69 last_success 160034
Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103
[2946]: s19 check_our_lease warning 60 last_success 160043
Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104
[2946]: s20 check_our_lease warning 70 last_success 160034
Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104
[2946]: s19 check_our_lease warning 61 last_success 160043
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s21 check_our_lease warning 60 last_success 160045
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s20 check_our_lease warning 71 last_success 160034
Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105
[2946]: s19 check_our_lease warning 62 last_success 160043
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s21 check_our_lease warning 61 last_success 160045
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s20 check_our_lease warning 72 last_success 160034
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106
[2946]: s19 check_our_lease warning 63 last_success 160043
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107
[23864]: s19 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107
[23864]: s19 renewal error -202 delta_length 11 last_success 160043
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s21 check_our_lease warning 62 last_success 160045
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s20 check_our_lease warning 73 last_success 160034
Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107
[2946]: s19 check_our_lease warning 64 last_success 160043
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[24471]: s21 delta_renew read rv -202 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[24471]: s21 renewal error -202 delta_length 10 last_success 160045
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s21 check_our_lease warning 63 last_success 160045
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s20 check_our_lease warning 74 last_success 160034
Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108
[2946]: s19 check_our_lease warning 65 last_success 160043
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s21 check_our_lease warning 64 last_success 160045
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s20 check_our_lease warning 75 last_success 160034
Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109
[2946]: s19 check_our_lease warning 66 last_success 160043
Oct 21 20:22:25 nodehv01 wdmd[2934]: test warning now 160110 ping 160100
close 145768 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:25 nodehv01 wdmd[2934]: /dev/watchdog closed unclean
Oct 21 20:22:25 nodehv01 kernel: iTCO_wdt: Unexpected close, not
stopping watchdog!
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s21 check_our_lease warning 65 last_success 160045
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s20 check_our_lease warning 76 last_success 160034
Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110
[2946]: s19 check_our_lease warning 67 last_success 160043
Oct 21 20:22:26 nodehv01 wdmd[2934]: test warning now 160111 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s21 check_our_lease warning 66 last_success 160045
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s20 check_our_lease warning 77 last_success 160034
Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111
[2946]: s19 check_our_lease warning 68 last_success 160043
Oct 21 20:22:27 nodehv01 wdmd[2934]: test warning now 160112 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s21 check_our_lease warning 67 last_success 160045
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s20 check_our_lease warning 78 last_success 160034
Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112
[2946]: s19 check_our_lease warning 69 last_success 160043
Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100
close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s21 check_our_lease warning 68 last_success 160045
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s20 check_our_lease warning 79 last_success 160034
Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113
[2946]: s19 check_our_lease warning 70 last_success 160043
Oct 21 20:22:29 nodehv01 wdmd[2934]: test warning now 160114 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:29 nodehv01 wdmd[2934]: test failed rem 56 now 160114 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s21 check_our_lease warning 69 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s20 check_our_lease failed 80
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s20 kill 24784 sig 15 count 1
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s19 check_our_lease warning 71 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s21 check_our_lease warning 69 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114
[2946]: s19 check_our_lease warning 71 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s20 kill 24784 sig 15 count 2
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:30 nodehv01 wdmd[2934]: test failed rem 55 now 160115 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s21 check_our_lease warning 70 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115
[2946]: s19 check_our_lease warning 72 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s20 kill 24784 sig 15 count 3
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:31 nodehv01 wdmd[2934]: test failed rem 54 now 160116 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s21 check_our_lease warning 71 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116
[2946]: s19 check_our_lease warning 73 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s20 kill 24784 sig 15 count 4
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:33 nodehv01 wdmd[2934]: test failed rem 53 now 160117 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s21 check_our_lease warning 72 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117
[2946]: s19 check_our_lease warning 74 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s20 kill 24784 sig 15 count 5
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[23864]: s19 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118
[23864]: s19 renewal error -2 delta_length 11 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:34 nodehv01 wdmd[2934]: test failed rem 52 now 160118 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s21 check_our_lease warning 73 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118
[2946]: s19 check_our_lease warning 75 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s20 kill 24784 sig 15 count 6
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[24471]: s21 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[24471]: s21 renewal error -2 delta_length 10 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:35 nodehv01 wdmd[2934]: test failed rem 51 now 160119 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s21 check_our_lease warning 74 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119
[2946]: s19 check_our_lease warning 76 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s20 kill 24784 sig 15 count 7
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:36 nodehv01 wdmd[2934]: test failed rem 50 now 160120 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s21 check_our_lease warning 75 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120
[2946]: s19 check_our_lease warning 77 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s20 kill 24784 sig 15 count 8
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:37 nodehv01 wdmd[2934]: test failed rem 49 now 160121 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s21 check_our_lease warning 76 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121
[2946]: s19 check_our_lease warning 78 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s20 kill 24784 sig 15 count 9
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100
close 160110 renewal 160043 expire 160123 client 2946
sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 48 now 160122 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s21 check_our_lease warning 77 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122
[2946]: s19 check_our_lease warning 79 last_success 160043
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s20 kill 24784 sig 15 count 10
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s19 check_our_lease failed 80
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123
[2946]: s19 all pids clear
Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 47 now 160123 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160123 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123
[2946]: s21 check_our_lease warning 78 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s20 kill 24784 sig 15 count 11
Oct 21 20:22:39 nodehv01 wdmd[2934]: test failed rem 46 now 160124 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:39 nodehv01 wdmd[2934]: test warning now 160124 ping 160100
close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124
[2946]: s21 check_our_lease warning 79 last_success 160045
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s21 check_our_lease failed 80
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s20 kill 24784 sig 15 count 12
Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping
160100 close 160110 renewal 160045 expire 160125 client 2946
sanlock_hosted-engine:1
Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125
[2946]: s21 all pids clear
Oct 21 20:22:41 nodehv01 wdmd[2934]: test failed rem 44 now 160126 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:41 nodehv01 sanlock[2946]: 2014-10-21 20:22:41+0200 160126
[2946]: s20 kill 24784 sig 15 count 13
Oct 21 20:22:42 nodehv01 wdmd[2934]: test failed rem 43 now 160127 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:42 nodehv01 sanlock[2946]: 2014-10-21 20:22:42+0200 160127
[2946]: s20 kill 24784 sig 15 count 14
Oct 21 20:22:43 nodehv01 wdmd[2934]: test failed rem 42 now 160128 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:43 nodehv01 sanlock[2946]: 2014-10-21 20:22:43+0200 160128
[2946]: s20 kill 24784 sig 15 count 15
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128
[23864]: s19 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128
[23864]: s19 renewal error -2 delta_length 10 last_success 160043
Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160129
[2946]: s20 kill 24784 sig 15 count 16
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 41 now 160129 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[2946]: s20 kill 24784 sig 15 count 17
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[24471]: s21 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace
Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130
[24471]: s21 renewal error -2 delta_length 10 last_success 160045
Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 40 now 160130 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:46 nodehv01 sanlock[2946]: 2014-10-21 20:22:46+0200 160131
[2946]: s20 kill 24784 sig 15 count 18
Oct 21 20:22:46 nodehv01 wdmd[2934]: test failed rem 39 now 160131 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:47 nodehv01 sanlock[2946]: 2014-10-21 20:22:47+0200 160132
[2946]: s20 kill 24784 sig 15 count 19
Oct 21 20:22:47 nodehv01 wdmd[2934]: test failed rem 38 now 160132 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:48 nodehv01 sanlock[2946]: 2014-10-21 20:22:48+0200 160133
[2946]: s20 kill 24784 sig 15 count 20
Oct 21 20:22:48 nodehv01 wdmd[2934]: test failed rem 37 now 160133 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:49 nodehv01 sanlock[2946]: 2014-10-21 20:22:49+0200 160134
[2946]: s20 kill 24784 sig 15 count 21
Oct 21 20:22:49 nodehv01 wdmd[2934]: test failed rem 36 now 160134 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:50 nodehv01 sanlock[2946]: 2014-10-21 20:22:50+0200 160135
[2946]: s20 kill 24784 sig 15 count 22
Oct 21 20:22:50 nodehv01 wdmd[2934]: test failed rem 35 now 160135 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:51 nodehv01 sanlock[2946]: 2014-10-21 20:22:51+0200 160136
[2946]: s20 kill 24784 sig 15 count 23
Oct 21 20:22:51 nodehv01 wdmd[2934]: test failed rem 34 now 160136 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:52 nodehv01 sanlock[2946]: 2014-10-21 20:22:52+0200 160137
[2946]: s20 kill 24784 sig 15 count 24
Oct 21 20:22:52 nodehv01 wdmd[2934]: test failed rem 33 now 160137 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:53 nodehv01 sanlock[2946]: 2014-10-21 20:22:53+0200 160138
[2946]: s20 kill 24784 sig 15 count 25
Oct 21 20:22:53 nodehv01 wdmd[2934]: test failed rem 32 now 160138 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 1 0x7ffd08000910 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 2 0x7ffd08000960 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138
[23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy
Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160139
[2946]: s20 kill 24784 sig 15 count 26
Oct 21 20:22:54 nodehv01 wdmd[2934]: test failed rem 31 now 160139 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[23969]: s20 delta_renew read rv -2 offset 0
/rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[23969]: s20 renewal error -2 delta_length 10 last_success 160034
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[2946]: s20 kill 24784 sig 15 count 27
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 1 0x7ffcec000910 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 2 0x7ffcec000960 busy
Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140
[24471]: hosted-e close_task_aio 3 0x7ffcec0009b0 busy
Oct 21 20:22:55 nodehv01 wdmd[2934]: test failed rem 30 now 160140 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:56 nodehv01 sanlock[2946]: 2014-10-21 20:22:56+0200 160141
[2946]: s20 kill 24784 sig 15 count 28
Oct 21 20:22:56 nodehv01 wdmd[2934]: test failed rem 29 now 160141 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:57 nodehv01 sanlock[2946]: 2014-10-21 20:22:57+0200 160142
[2946]: s20 kill 24784 sig 15 count 29
Oct 21 20:22:57 nodehv01 wdmd[2934]: test failed rem 28 now 160142 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:58 nodehv01 sanlock[2946]: 2014-10-21 20:22:58+0200 160143
[2946]: s20 kill 24784 sig 15 count 30
Oct 21 20:22:58 nodehv01 wdmd[2934]: test failed rem 27 now 160143 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160143
[24471]: hosted-e aio collect
0x7ffcec0009b0:0x7ffcec0009c0:0x7ffd000e2000 result 1048576:0 close free
Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160144
[2946]: s20 kill 24784 sig 15 count 31
Oct 21 20:22:59 nodehv01 wdmd[2934]: test failed rem 26 now 160144 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:00 nodehv01 sanlock[2946]: 2014-10-21 20:23:00+0200 160145
[2946]: s20 kill 24784 sig 15 count 32
Oct 21 20:23:00 nodehv01 wdmd[2934]: test failed rem 25 now 160145 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146
[2946]: s20 kill 24784 sig 15 count 33
Oct 21 20:23:01 nodehv01 wdmd[2934]: test failed rem 24 now 160146 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146
[23864]: e3b7a377 aio collect
0x7ffd08000960:0x7ffd08000970:0x7ffd005ec000 result 1048576:0 close free
Oct 21 20:23:02 nodehv01 sanlock[2946]: 2014-10-21 20:23:02+0200 160147
[2946]: s20 kill 24784 sig 15 count 34
Oct 21 20:23:03 nodehv01 wdmd[2934]: test failed rem 23 now 160147 ping
160100 close 160110 renewal 160034 expire 160114 client 2946
sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1
Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160147
[24471]: hosted-e aio collect
0x7ffcec000960:0x7ffcec000970:0x7ffd003e8000 result 1048576:0 close free
Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160148
[2946]: s20 kill 24784 sig 15 count 35
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode
Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering
disabled state
Oct 21 20:23:04 nodehv01 wdmd[2934]: /dev/watchdog reopen
Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148
[2946]: dead 24784 ci 5 count 35
Oct 21 20:23:04 nodehv01 vdsm vds ERROR Received 00000011 on fileno 53
Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148
[2946]: s20 all pids clear
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleDisk at 0x2c4c110>#012Traceback (most
recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 323, in _sampleDisk#012
diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)#012
File "/usr/share/vdsm/virt/vm.py", line 670, in f#012 ret =
attr(*args, **kwargs)#012 File
"/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111,
in wrapper#012 ret = f(*args, **kwargs)#012 File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 1838, in
blockStats#012 if ret is None: raise libvirtError
('virDomainBlockStats() failed', dom=self)#012libvirtError: Unable to
read from monitor: Connection reset by peer
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleDiskLatency at 0x2c4a930>#012Traceback
(most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py",
line 471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 337, in _sampleDiskLatency#012
diskLatency[vmDrive.name] =
self._vm._dom.blockStatsFlags(#012AttributeError: 'NoneType' object has
no attribute 'blockStatsFlags'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleNet at 0x2c4c180>#012Traceback (most
recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 344, in _sampleNet#012
netSamples[nic.name] =
self._vm._dom.interfaceStats(nic.name)#012AttributeError: 'NoneType'
object has no attribute 'interfaceStats'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleBalloon at 0x2c4c1f0>#012Traceback (most
recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 352, in _sampleBalloon#012 infos =
self._vm._dom.info()#012AttributeError: 'NoneType' object has no
attribute 'info'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleVcpuPinning at 0x2c4a9b0>#012Traceback
(most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py",
line 471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 348, in _sampleVcpuPinning#012
vCpuInfos = self._vm._dom.vcpus()#012AttributeError: 'NoneType' object
has no attribute 'vcpus'
Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR
vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed:
<AdvancedStatsFunction _sampleCpuTune at 0x2c4c2d0>#012Traceback (most
recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line
471, in collect#012 statsFunction()#012 File
"/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012
retValue = self._function(*args, **kwargs)#012 File
"/usr/share/vdsm/virt/vm.py", line 359, in _sampleCpuTune#012 infos =
self._vm._dom.schedulerParameters()#012AttributeError: 'NoneType' object
has no attribute 'schedulerParameters'
Oct 21 20:23:05 nodehv01 ntpd[3047]: Deleting interface #43 vnet3,
fe80::fc16:3eff:fe74:4156#123, interface stats: received=0, sent=0,
dropped=0, active_time=14116 secs
Oct 21 20:23:05 nodehv01 ntpd[3047]: peers refreshed
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 aio collect
0x7ffd08000910:0x7ffd08000920:0x7ffd008f2000 result 1048576:0 close free
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy
Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150
[23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[23969]: bcfa7ec4 aio collect
0x7ffcf4000960:0x7ffcf4000970:0x7ffd007f0000 result 1048576:0 close free
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[24471]: hosted-e aio collect
0x7ffcec000910:0x7ffcec000920:0x7ffd006ee000 result 1048576:0 close free
Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151
[24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy
Oct 21 20:23:09 nodehv01 sanlock[2946]: 2014-10-21 20:23:09+0200 160154
[23864]: e3b7a377 aio collect
0x7ffd080008c0:0x7ffd080008d0:0x7ffd1005e000 result 1048576:0 close free
Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155
[23969]: bcfa7ec4 aio collect
0x7ffcf4000910:0x7ffcf4000920:0x7ffd009f4000 result 1048576:0 close free
Oct 21 20:23:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155
[24471]: hosted-e aio collect
0x7ffcec0008c0:0x7ffcec0008d0:0x7ffd00af6000 result 1048576:0 close free
Oct 21 20:23:12 nodehv01 sanlock[2946]: 2014-10-21 20:23:12+0200 160157
[23864]: e3b7a377 aio collect
0x7ffd080009b0:0x7ffd080009c0:0x7ffd002e6000 result 1048576:0 close free
Oct 21 20:24:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:25:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth
ERROR Failed to getVmStats: 'pid'
Oct 21 20:26:05 nodehv01 ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine vm
died unexpectedly
Oct 21 20:26:06 nodehv01 kernel: device-mapper: table: 253:28:
multipath: error getting device
--
Daniel Helgenberger
m box bewegtbild GmbH
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19
D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767