Hi, Hello Martin,
I think there is something weird going on with your storage, this is the crash snippet from the host that had the engine at the beginning: This is what I suspected also; though its hard to track down atm, because when I cecked NFS was accessible just fine from both hosts. In
On 22.10.2014 10:17, Martin Sivak wrote: the end I have a snippet from /var/log/messages; I suppose it all starts with the sanlock warnings? Also I suppose: Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering disabled state Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering disabled state this actually the HE being killed?
/var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21 20:22:33,919::task::866::Storage.TaskManager.Task::(_setError) Task=`2ad31974-e1fc-4785-9423-ff3bd087a5aa`::Unexpected error /var/log/vdsm/vdsm.log:Thread-162994::ERROR::2014-10-21 20:22:33,934::dispatcher::79::Storage.Dispatcher::(wrapper) Connection timed out /var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21 20:23:00,733::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 68aad705-7c9b-427a-a84c-6f32f23675b3 /var/log/vdsm/vdsm.log:Thread-62::ERROR::2014-10-21 20:23:00,734::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 68aad705-7c9b-427a-a84c-6f32f23675b3 /var/log/vdsm/vdsm.log:VM Channels Listener::ERROR::2014-10-21 20:23:04,258::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 53
The second host's VDSM lost the connection to storage domain at the same time..
20:23:09,950::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host 192.168.50.201 (id 1) 20:23:12,365::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING
The engine VM was restarted right after the connection was restored:
20:25:54,336::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING 20:26:20,572::hosted_engine::571::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Acquired lock on host id 2 20:26:20,572::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) 20:26:20,572::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 192.168.50.201 (id: 1, score: 2400) 20:26:30,606::states::459::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (2400), attempting to start engine VM
...
20:27:34,423::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineStarting'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUp'> 20:27:34,430::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1413916054.43 type=state_transition detail=EngineStarting-EngineUp hostname='nodehv02.lab.mbox.loc' 20:27:34,498::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineUp) sent? sent 20:27:38,481::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400)
All was then well till the end of the log.
20:29:53,393::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost 20:29:55,372::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) 20:29:55,372::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 192.168.50.201 (id: 1, score: 0)
Hosted engine had nothing to do with the engine crash according to the log. On the contrary, it properly re-started the VM once the cluster recovered from the storage issue.
Good to know :)
Can you give us more information about the setup? Storage type, topology, ...
Nothing fancy really. Hardware is old, though. Shared storage for engine is NFS; some nfs, iscsi and fc domains are used for VMs. I have had a crash of the NFS service; but all this might be a switch behaving badly in high traffic situations in the end.
-- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ
----- Original Message -----
Hello,
since upgrading to the latest hosted-engine-ha I have the follwing problem:
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm died unexpectedly
I suppose HA is forcing the engine down because liveliness check is failing. I attached a log compile from the latest incident, 2014-10-21 16:26:31,836. The 'host' logs are from the hosts the engine was running on, host2 the other HA host Interestingly this only happens when I was connected via a VNC console to one of my Winodws 2012 VMs.
How can I further debug this? The engine log seems empty and also the HE does not seem to have any trouble when this happens. As precaustion / test I set my cluster to global maintenance.
Thanks,
vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-4.16.7-1.gitdb83943.el6.x86_64 vdsm-python-4.16.7-1.gitdb83943.el6.noarch vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-cli-4.16.7-1.gitdb83943.el6.noarch
ovirt-hosted-engine-ha-1.2.4-1.el6.noarch ovirt-release35-001-1.noarch ovirt-host-deploy-1.3.0-1.el6.noarch ovirt-hosted-engine-setup-1.2.1-1.el6.noarch ovirt-release34-1.0.3-1.noarch ovirt-engine-sdk-python-3.5.0.7-1.el6.noarch
-- Daniel Helgenberger m box bewegtbild GmbH
P: +49/30/2408781-22 F: +49/30/2408781-10
ACKERSTR. 19 D-10115 BERLIN
www.m-box.de www.monkeymen.tv
Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064 [23969]: s20 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:21:40 nodehv01 sanlock[2946]: 2014-10-21 20:21:40+0200 160064 [23969]: s20 renewal error -202 delta_length 10 last_success 160034 Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074 [23864]: s19 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:21:49 nodehv01 sanlock[2946]: 2014-10-21 20:21:49+0200 160074 [23864]: s19 renewal error -202 delta_length 10 last_success 160043 Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075 [23969]: s20 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075 [23969]: s20 renewal error -202 delta_length 10 last_success 160034 Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075 [24471]: s21 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:21:51 nodehv01 sanlock[2946]: 2014-10-21 20:21:51+0200 160075 [24471]: s21 renewal error -202 delta_length 10 last_success 160045 Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085 [23864]: s19 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:22:00 nodehv01 sanlock[2946]: 2014-10-21 20:22:00+0200 160085 [23864]: s19 renewal error -202 delta_length 11 last_success 160043 Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086 [23969]: s20 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086 [23969]: s20 renewal error -202 delta_length 10 last_success 160034 Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086 [24471]: s21 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:22:02 nodehv01 sanlock[2946]: 2014-10-21 20:22:02+0200 160086 [24471]: s21 renewal error -202 delta_length 10 last_success 160045 Oct 21 20:22:10 nodehv01 sanlock[2946]: 2014-10-21 20:22:10+0200 160094 [2946]: s20 check_our_lease warning 60 last_success 160034 Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160095 [2946]: s20 check_our_lease warning 61 last_success 160034 Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096 [23864]: s19 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:22:11 nodehv01 sanlock[2946]: 2014-10-21 20:22:11+0200 160096 [23864]: s19 renewal error -202 delta_length 11 last_success 160043 Oct 21 20:22:12 nodehv01 sanlock[2946]: 2014-10-21 20:22:12+0200 160096 [2946]: s20 check_our_lease warning 62 last_success 160034 Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097 [23969]: s20 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097 [23969]: s20 renewal error -202 delta_length 10 last_success 160034 Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097 [24471]: s21 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097 [24471]: s21 renewal error -202 delta_length 10 last_success 160045 Oct 21 20:22:13 nodehv01 sanlock[2946]: 2014-10-21 20:22:13+0200 160097 [2946]: s20 check_our_lease warning 63 last_success 160034 Oct 21 20:22:14 nodehv01 sanlock[2946]: 2014-10-21 20:22:14+0200 160098 [2946]: s20 check_our_lease warning 64 last_success 160034 Oct 21 20:22:15 nodehv01 sanlock[2946]: 2014-10-21 20:22:15+0200 160099 [2946]: s20 check_our_lease warning 65 last_success 160034 Oct 21 20:22:16 nodehv01 sanlock[2946]: 2014-10-21 20:22:16+0200 160100 [2946]: s20 check_our_lease warning 66 last_success 160034 Oct 21 20:22:17 nodehv01 sanlock[2946]: 2014-10-21 20:22:17+0200 160101 [2946]: s20 check_our_lease warning 67 last_success 160034 Oct 21 20:22:18 nodehv01 sanlock[2946]: 2014-10-21 20:22:18+0200 160102 [2946]: s20 check_our_lease warning 68 last_success 160034 Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103 [2946]: s20 check_our_lease warning 69 last_success 160034 Oct 21 20:22:19 nodehv01 sanlock[2946]: 2014-10-21 20:22:19+0200 160103 [2946]: s19 check_our_lease warning 60 last_success 160043 Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104 [2946]: s20 check_our_lease warning 70 last_success 160034 Oct 21 20:22:20 nodehv01 sanlock[2946]: 2014-10-21 20:22:20+0200 160104 [2946]: s19 check_our_lease warning 61 last_success 160043 Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105 [2946]: s21 check_our_lease warning 60 last_success 160045 Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105 [2946]: s20 check_our_lease warning 71 last_success 160034 Oct 21 20:22:21 nodehv01 sanlock[2946]: 2014-10-21 20:22:21+0200 160105 [2946]: s19 check_our_lease warning 62 last_success 160043 Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106 [2946]: s21 check_our_lease warning 61 last_success 160045 Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106 [2946]: s20 check_our_lease warning 72 last_success 160034 Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160106 [2946]: s19 check_our_lease warning 63 last_success 160043 Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107 [23864]: s19 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:22:22 nodehv01 sanlock[2946]: 2014-10-21 20:22:22+0200 160107 [23864]: s19 renewal error -202 delta_length 11 last_success 160043 Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107 [2946]: s21 check_our_lease warning 62 last_success 160045 Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107 [2946]: s20 check_our_lease warning 73 last_success 160034 Oct 21 20:22:23 nodehv01 sanlock[2946]: 2014-10-21 20:22:23+0200 160107 [2946]: s19 check_our_lease warning 64 last_success 160043 Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [23969]: s20 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [23969]: s20 renewal error -2 delta_length 10 last_success 160034 Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [24471]: s21 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [24471]: s21 renewal error -202 delta_length 10 last_success 160045 Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [2946]: s21 check_our_lease warning 63 last_success 160045 Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [2946]: s20 check_our_lease warning 74 last_success 160034 Oct 21 20:22:24 nodehv01 sanlock[2946]: 2014-10-21 20:22:24+0200 160108 [2946]: s19 check_our_lease warning 65 last_success 160043 Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109 [2946]: s21 check_our_lease warning 64 last_success 160045 Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109 [2946]: s20 check_our_lease warning 75 last_success 160034 Oct 21 20:22:25 nodehv01 sanlock[2946]: 2014-10-21 20:22:25+0200 160109 [2946]: s19 check_our_lease warning 66 last_success 160043 Oct 21 20:22:25 nodehv01 wdmd[2934]: test warning now 160110 ping 160100 close 145768 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:25 nodehv01 wdmd[2934]: /dev/watchdog closed unclean Oct 21 20:22:25 nodehv01 kernel: iTCO_wdt: Unexpected close, not stopping watchdog! Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110 [2946]: s21 check_our_lease warning 65 last_success 160045 Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110 [2946]: s20 check_our_lease warning 76 last_success 160034 Oct 21 20:22:26 nodehv01 sanlock[2946]: 2014-10-21 20:22:26+0200 160110 [2946]: s19 check_our_lease warning 67 last_success 160043 Oct 21 20:22:26 nodehv01 wdmd[2934]: test warning now 160111 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111 [2946]: s21 check_our_lease warning 66 last_success 160045 Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111 [2946]: s20 check_our_lease warning 77 last_success 160034 Oct 21 20:22:27 nodehv01 sanlock[2946]: 2014-10-21 20:22:27+0200 160111 [2946]: s19 check_our_lease warning 68 last_success 160043 Oct 21 20:22:27 nodehv01 wdmd[2934]: test warning now 160112 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112 [2946]: s21 check_our_lease warning 67 last_success 160045 Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112 [2946]: s20 check_our_lease warning 78 last_success 160034 Oct 21 20:22:28 nodehv01 sanlock[2946]: 2014-10-21 20:22:28+0200 160112 [2946]: s19 check_our_lease warning 69 last_success 160043 Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:28 nodehv01 wdmd[2934]: test warning now 160113 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113 [2946]: s21 check_our_lease warning 68 last_success 160045 Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113 [2946]: s20 check_our_lease warning 79 last_success 160034 Oct 21 20:22:29 nodehv01 sanlock[2946]: 2014-10-21 20:22:29+0200 160113 [2946]: s19 check_our_lease warning 70 last_success 160043 Oct 21 20:22:29 nodehv01 wdmd[2934]: test warning now 160114 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:29 nodehv01 wdmd[2934]: test failed rem 56 now 160114 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s21 check_our_lease warning 69 last_success 160045 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s20 check_our_lease failed 80 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s20 kill 24784 sig 15 count 1 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s19 check_our_lease warning 71 last_success 160043 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s21 check_our_lease warning 69 last_success 160045 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160114 [2946]: s19 check_our_lease warning 71 last_success 160043 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115 [2946]: s21 check_our_lease warning 70 last_success 160045 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115 [2946]: s20 kill 24784 sig 15 count 2 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115 [2946]: s19 check_our_lease warning 72 last_success 160043 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115 [2946]: s21 check_our_lease warning 70 last_success 160045 Oct 21 20:22:30 nodehv01 sanlock[2946]: 2014-10-21 20:22:30+0200 160115 [2946]: s19 check_our_lease warning 72 last_success 160043 Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:30 nodehv01 wdmd[2934]: test failed rem 55 now 160115 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:30 nodehv01 wdmd[2934]: test warning now 160115 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s21 check_our_lease warning 70 last_success 160045 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s19 check_our_lease warning 72 last_success 160043 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s21 check_our_lease warning 70 last_success 160045 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s19 check_our_lease warning 72 last_success 160043 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s21 check_our_lease warning 70 last_success 160045 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160115 [2946]: s19 check_our_lease warning 72 last_success 160043 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116 [2946]: s21 check_our_lease warning 71 last_success 160045 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116 [2946]: s20 kill 24784 sig 15 count 3 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116 [2946]: s19 check_our_lease warning 73 last_success 160043 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116 [2946]: s21 check_our_lease warning 71 last_success 160045 Oct 21 20:22:31 nodehv01 sanlock[2946]: 2014-10-21 20:22:31+0200 160116 [2946]: s19 check_our_lease warning 73 last_success 160043 Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:31 nodehv01 wdmd[2934]: test failed rem 54 now 160116 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:31 nodehv01 wdmd[2934]: test warning now 160116 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s21 check_our_lease warning 71 last_success 160045 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s19 check_our_lease warning 73 last_success 160043 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s21 check_our_lease warning 71 last_success 160045 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s19 check_our_lease warning 73 last_success 160043 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s21 check_our_lease warning 71 last_success 160045 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160116 [2946]: s19 check_our_lease warning 73 last_success 160043 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117 [2946]: s21 check_our_lease warning 72 last_success 160045 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117 [2946]: s20 kill 24784 sig 15 count 4 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117 [2946]: s19 check_our_lease warning 74 last_success 160043 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117 [2946]: s21 check_our_lease warning 72 last_success 160045 Oct 21 20:22:32 nodehv01 sanlock[2946]: 2014-10-21 20:22:32+0200 160117 [2946]: s19 check_our_lease warning 74 last_success 160043 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s21 check_our_lease warning 72 last_success 160045 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s19 check_our_lease warning 74 last_success 160043 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s21 check_our_lease warning 72 last_success 160045 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s19 check_our_lease warning 74 last_success 160043 Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:33 nodehv01 wdmd[2934]: test failed rem 53 now 160117 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:33 nodehv01 wdmd[2934]: test warning now 160117 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s21 check_our_lease warning 72 last_success 160045 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160117 [2946]: s19 check_our_lease warning 74 last_success 160043 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [2946]: s21 check_our_lease warning 73 last_success 160045 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [2946]: s20 kill 24784 sig 15 count 5 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [2946]: s19 check_our_lease warning 75 last_success 160043 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [2946]: s21 check_our_lease warning 73 last_success 160045 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [2946]: s19 check_our_lease warning 75 last_success 160043 Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [23864]: s19 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:22:33 nodehv01 sanlock[2946]: 2014-10-21 20:22:33+0200 160118 [23864]: s19 renewal error -2 delta_length 11 last_success 160043 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s21 check_our_lease warning 73 last_success 160045 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s19 check_our_lease warning 75 last_success 160043 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s21 check_our_lease warning 73 last_success 160045 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s19 check_our_lease warning 75 last_success 160043 Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:34 nodehv01 wdmd[2934]: test failed rem 52 now 160118 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:34 nodehv01 wdmd[2934]: test warning now 160118 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s21 check_our_lease warning 73 last_success 160045 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160118 [2946]: s19 check_our_lease warning 75 last_success 160043 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [23969]: s20 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [23969]: s20 renewal error -2 delta_length 10 last_success 160034 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [2946]: s21 check_our_lease warning 74 last_success 160045 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [2946]: s20 kill 24784 sig 15 count 6 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [2946]: s19 check_our_lease warning 76 last_success 160043 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [2946]: s21 check_our_lease warning 74 last_success 160045 Oct 21 20:22:34 nodehv01 sanlock[2946]: 2014-10-21 20:22:34+0200 160119 [2946]: s19 check_our_lease warning 76 last_success 160043 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s21 check_our_lease warning 74 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s19 check_our_lease warning 76 last_success 160043 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [24471]: s21 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [24471]: s21 renewal error -2 delta_length 10 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s21 check_our_lease warning 74 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s19 check_our_lease warning 76 last_success 160043 Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:35 nodehv01 wdmd[2934]: test failed rem 51 now 160119 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:35 nodehv01 wdmd[2934]: test warning now 160119 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s21 check_our_lease warning 74 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160119 [2946]: s19 check_our_lease warning 76 last_success 160043 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120 [2946]: s21 check_our_lease warning 75 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120 [2946]: s20 kill 24784 sig 15 count 7 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120 [2946]: s19 check_our_lease warning 77 last_success 160043 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120 [2946]: s21 check_our_lease warning 75 last_success 160045 Oct 21 20:22:35 nodehv01 sanlock[2946]: 2014-10-21 20:22:35+0200 160120 [2946]: s19 check_our_lease warning 77 last_success 160043 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s21 check_our_lease warning 75 last_success 160045 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s19 check_our_lease warning 77 last_success 160043 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s21 check_our_lease warning 75 last_success 160045 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s19 check_our_lease warning 77 last_success 160043 Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:36 nodehv01 wdmd[2934]: test failed rem 50 now 160120 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:36 nodehv01 wdmd[2934]: test warning now 160120 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s21 check_our_lease warning 75 last_success 160045 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160120 [2946]: s19 check_our_lease warning 77 last_success 160043 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121 [2946]: s21 check_our_lease warning 76 last_success 160045 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121 [2946]: s20 kill 24784 sig 15 count 8 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121 [2946]: s19 check_our_lease warning 78 last_success 160043 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121 [2946]: s21 check_our_lease warning 76 last_success 160045 Oct 21 20:22:36 nodehv01 sanlock[2946]: 2014-10-21 20:22:36+0200 160121 [2946]: s19 check_our_lease warning 78 last_success 160043 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s21 check_our_lease warning 76 last_success 160045 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s19 check_our_lease warning 78 last_success 160043 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s21 check_our_lease warning 76 last_success 160045 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s19 check_our_lease warning 78 last_success 160043 Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:37 nodehv01 wdmd[2934]: test failed rem 49 now 160121 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:37 nodehv01 wdmd[2934]: test warning now 160121 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s21 check_our_lease warning 76 last_success 160045 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160121 [2946]: s19 check_our_lease warning 78 last_success 160043 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122 [2946]: s21 check_our_lease warning 77 last_success 160045 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122 [2946]: s20 kill 24784 sig 15 count 9 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122 [2946]: s19 check_our_lease warning 79 last_success 160043 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122 [2946]: s21 check_our_lease warning 77 last_success 160045 Oct 21 20:22:37 nodehv01 sanlock[2946]: 2014-10-21 20:22:37+0200 160122 [2946]: s19 check_our_lease warning 79 last_success 160043 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s21 check_our_lease warning 77 last_success 160045 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s19 check_our_lease warning 79 last_success 160043 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s21 check_our_lease warning 77 last_success 160045 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s19 check_our_lease warning 79 last_success 160043 Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100 close 160110 renewal 160043 expire 160123 client 2946 sanlock_e3b7a377-22d5-47b4-b753-7b42db00ce61:1 Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 48 now 160122 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160122 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s21 check_our_lease warning 77 last_success 160045 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160122 [2946]: s19 check_our_lease warning 79 last_success 160043 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123 [2946]: s21 check_our_lease warning 78 last_success 160045 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123 [2946]: s20 kill 24784 sig 15 count 10 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123 [2946]: s19 check_our_lease failed 80 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123 [2946]: s21 check_our_lease warning 78 last_success 160045 Oct 21 20:22:38 nodehv01 sanlock[2946]: 2014-10-21 20:22:38+0200 160123 [2946]: s19 all pids clear Oct 21 20:22:38 nodehv01 wdmd[2934]: test failed rem 47 now 160123 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:38 nodehv01 wdmd[2934]: test warning now 160123 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123 [2946]: s21 check_our_lease warning 78 last_success 160045 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123 [2946]: s21 check_our_lease warning 78 last_success 160045 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160123 [2946]: s21 check_our_lease warning 78 last_success 160045 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124 [2946]: s21 check_our_lease warning 79 last_success 160045 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124 [2946]: s20 kill 24784 sig 15 count 11 Oct 21 20:22:39 nodehv01 wdmd[2934]: test failed rem 46 now 160124 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:39 nodehv01 wdmd[2934]: test warning now 160124 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:39 nodehv01 sanlock[2946]: 2014-10-21 20:22:39+0200 160124 [2946]: s21 check_our_lease warning 79 last_success 160045 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124 [2946]: s21 check_our_lease warning 79 last_success 160045 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124 [2946]: s21 check_our_lease warning 79 last_success 160045 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160124 [2946]: s21 check_our_lease warning 79 last_success 160045 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125 [2946]: s21 check_our_lease failed 80 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125 [2946]: s20 kill 24784 sig 15 count 12 Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:40 nodehv01 wdmd[2934]: test failed rem 45 now 160125 ping 160100 close 160110 renewal 160045 expire 160125 client 2946 sanlock_hosted-engine:1 Oct 21 20:22:40 nodehv01 sanlock[2946]: 2014-10-21 20:22:40+0200 160125 [2946]: s21 all pids clear Oct 21 20:22:41 nodehv01 wdmd[2934]: test failed rem 44 now 160126 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:41 nodehv01 sanlock[2946]: 2014-10-21 20:22:41+0200 160126 [2946]: s20 kill 24784 sig 15 count 13 Oct 21 20:22:42 nodehv01 wdmd[2934]: test failed rem 43 now 160127 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:42 nodehv01 sanlock[2946]: 2014-10-21 20:22:42+0200 160127 [2946]: s20 kill 24784 sig 15 count 14 Oct 21 20:22:43 nodehv01 wdmd[2934]: test failed rem 42 now 160128 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:43 nodehv01 sanlock[2946]: 2014-10-21 20:22:43+0200 160128 [2946]: s20 kill 24784 sig 15 count 15 Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128 [23864]: s19 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-data01/e3b7a377-22d5-47b4-b753-7b42db00ce61/dom_md/ids Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160128 [23864]: s19 renewal error -2 delta_length 10 last_success 160043 Oct 21 20:22:44 nodehv01 sanlock[2946]: 2014-10-21 20:22:44+0200 160129 [2946]: s20 kill 24784 sig 15 count 16 Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129 [23969]: s20 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160129 [23969]: s20 renewal error -2 delta_length 10 last_success 160034 Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 41 now 160129 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130 [2946]: s20 kill 24784 sig 15 count 17 Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130 [24471]: s21 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/ha_agent/hosted-engine.lockspace Oct 21 20:22:45 nodehv01 sanlock[2946]: 2014-10-21 20:22:45+0200 160130 [24471]: s21 renewal error -2 delta_length 10 last_success 160045 Oct 21 20:22:45 nodehv01 wdmd[2934]: test failed rem 40 now 160130 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:46 nodehv01 sanlock[2946]: 2014-10-21 20:22:46+0200 160131 [2946]: s20 kill 24784 sig 15 count 18 Oct 21 20:22:46 nodehv01 wdmd[2934]: test failed rem 39 now 160131 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:47 nodehv01 sanlock[2946]: 2014-10-21 20:22:47+0200 160132 [2946]: s20 kill 24784 sig 15 count 19 Oct 21 20:22:47 nodehv01 wdmd[2934]: test failed rem 38 now 160132 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:48 nodehv01 sanlock[2946]: 2014-10-21 20:22:48+0200 160133 [2946]: s20 kill 24784 sig 15 count 20 Oct 21 20:22:48 nodehv01 wdmd[2934]: test failed rem 37 now 160133 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:49 nodehv01 sanlock[2946]: 2014-10-21 20:22:49+0200 160134 [2946]: s20 kill 24784 sig 15 count 21 Oct 21 20:22:49 nodehv01 wdmd[2934]: test failed rem 36 now 160134 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:50 nodehv01 sanlock[2946]: 2014-10-21 20:22:50+0200 160135 [2946]: s20 kill 24784 sig 15 count 22 Oct 21 20:22:50 nodehv01 wdmd[2934]: test failed rem 35 now 160135 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:51 nodehv01 sanlock[2946]: 2014-10-21 20:22:51+0200 160136 [2946]: s20 kill 24784 sig 15 count 23 Oct 21 20:22:51 nodehv01 wdmd[2934]: test failed rem 34 now 160136 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:52 nodehv01 sanlock[2946]: 2014-10-21 20:22:52+0200 160137 [2946]: s20 kill 24784 sig 15 count 24 Oct 21 20:22:52 nodehv01 wdmd[2934]: test failed rem 33 now 160137 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:53 nodehv01 sanlock[2946]: 2014-10-21 20:22:53+0200 160138 [2946]: s20 kill 24784 sig 15 count 25 Oct 21 20:22:53 nodehv01 wdmd[2934]: test failed rem 32 now 160138 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138 [23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138 [23864]: e3b7a377 close_task_aio 1 0x7ffd08000910 busy Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138 [23864]: e3b7a377 close_task_aio 2 0x7ffd08000960 busy Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160138 [23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy Oct 21 20:22:54 nodehv01 sanlock[2946]: 2014-10-21 20:22:54+0200 160139 [2946]: s20 kill 24784 sig 15 count 26 Oct 21 20:22:54 nodehv01 wdmd[2934]: test failed rem 31 now 160139 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [23969]: s20 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.50.11:_volumes_tank_ovirt-vdsmd/bcfa7ec4-5278-44d8-9f31-682f2d9de91d/dom_md/ids Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [23969]: s20 renewal error -2 delta_length 10 last_success 160034 Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [2946]: s20 kill 24784 sig 15 count 27 Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [24471]: hosted-e close_task_aio 1 0x7ffcec000910 busy Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [24471]: hosted-e close_task_aio 2 0x7ffcec000960 busy Oct 21 20:22:55 nodehv01 sanlock[2946]: 2014-10-21 20:22:55+0200 160140 [24471]: hosted-e close_task_aio 3 0x7ffcec0009b0 busy Oct 21 20:22:55 nodehv01 wdmd[2934]: test failed rem 30 now 160140 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:56 nodehv01 sanlock[2946]: 2014-10-21 20:22:56+0200 160141 [2946]: s20 kill 24784 sig 15 count 28 Oct 21 20:22:56 nodehv01 wdmd[2934]: test failed rem 29 now 160141 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:57 nodehv01 sanlock[2946]: 2014-10-21 20:22:57+0200 160142 [2946]: s20 kill 24784 sig 15 count 29 Oct 21 20:22:57 nodehv01 wdmd[2934]: test failed rem 28 now 160142 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:58 nodehv01 sanlock[2946]: 2014-10-21 20:22:58+0200 160143 [2946]: s20 kill 24784 sig 15 count 30 Oct 21 20:22:58 nodehv01 wdmd[2934]: test failed rem 27 now 160143 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160143 [24471]: hosted-e aio collect 0x7ffcec0009b0:0x7ffcec0009c0:0x7ffd000e2000 result 1048576:0 close free Oct 21 20:22:59 nodehv01 sanlock[2946]: 2014-10-21 20:22:59+0200 160144 [2946]: s20 kill 24784 sig 15 count 31 Oct 21 20:22:59 nodehv01 wdmd[2934]: test failed rem 26 now 160144 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:23:00 nodehv01 sanlock[2946]: 2014-10-21 20:23:00+0200 160145 [2946]: s20 kill 24784 sig 15 count 32 Oct 21 20:23:00 nodehv01 wdmd[2934]: test failed rem 25 now 160145 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146 [2946]: s20 kill 24784 sig 15 count 33 Oct 21 20:23:01 nodehv01 wdmd[2934]: test failed rem 24 now 160146 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:23:01 nodehv01 sanlock[2946]: 2014-10-21 20:23:01+0200 160146 [23864]: e3b7a377 aio collect 0x7ffd08000960:0x7ffd08000970:0x7ffd005ec000 result 1048576:0 close free Oct 21 20:23:02 nodehv01 sanlock[2946]: 2014-10-21 20:23:02+0200 160147 [2946]: s20 kill 24784 sig 15 count 34 Oct 21 20:23:03 nodehv01 wdmd[2934]: test failed rem 23 now 160147 ping 160100 close 160110 renewal 160034 expire 160114 client 2946 sanlock_bcfa7ec4-5278-44d8-9f31-682f2d9de91d:1 Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160147 [24471]: hosted-e aio collect 0x7ffcec000960:0x7ffcec000970:0x7ffd003e8000 result 1048576:0 close free Oct 21 20:23:03 nodehv01 sanlock[2946]: 2014-10-21 20:23:03+0200 160148 [2946]: s20 kill 24784 sig 15 count 35 Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering disabled state Oct 21 20:23:03 nodehv01 kernel: device vnet3 left promiscuous mode Oct 21 20:23:03 nodehv01 kernel: ovirtmgmt: port 3(vnet3) entering disabled state Oct 21 20:23:04 nodehv01 wdmd[2934]: /dev/watchdog reopen Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148 [2946]: dead 24784 ci 5 count 35 Oct 21 20:23:04 nodehv01 vdsm vds ERROR Received 00000011 on fileno 53 Oct 21 20:23:04 nodehv01 sanlock[2946]: 2014-10-21 20:23:04+0200 160148 [2946]: s20 all pids clear Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x2c4c110>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 323, in _sampleDisk#012 diskSamples[vmDrive.name] = self._vm._dom.blockStats(vmDrive.name)#012 File "/usr/share/vdsm/virt/vm.py", line 670, in f#012 ret = attr(*args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper#012 ret = f(*args, **kwargs)#012 File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1838, in blockStats#012 if ret is None: raise libvirtError ('virDomainBlockStats() failed', dom=self)#012libvirtError: Unable to read from monitor: Connection reset by peer Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleDiskLatency at 0x2c4a930>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 337, in _sampleDiskLatency#012 diskLatency[vmDrive.name] = self._vm._dom.blockStatsFlags(#012AttributeError: 'NoneType' object has no attribute 'blockStatsFlags' Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x2c4c180>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 344, in _sampleNet#012 netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)#012AttributeError: 'NoneType' object has no attribute 'interfaceStats' Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleBalloon at 0x2c4c1f0>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 352, in _sampleBalloon#012 infos = self._vm._dom.info()#012AttributeError: 'NoneType' object has no attribute 'info' Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleVcpuPinning at 0x2c4a9b0>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 348, in _sampleVcpuPinning#012 vCpuInfos = self._vm._dom.vcpus()#012AttributeError: 'NoneType' object has no attribute 'vcpus' Oct 21 20:23:04 nodehv01 vdsm vm.Vm ERROR vmId=`e68a11c8-1251-4c13-9e3b-3847bbb4fa3d`::Stats function failed: <AdvancedStatsFunction _sampleCpuTune at 0x2c4c2d0>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/virt/sampling.py", line 471, in collect#012 statsFunction()#012 File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/virt/vm.py", line 359, in _sampleCpuTune#012 infos = self._vm._dom.schedulerParameters()#012AttributeError: 'NoneType' object has no attribute 'schedulerParameters' Oct 21 20:23:05 nodehv01 ntpd[3047]: Deleting interface #43 vnet3, fe80::fc16:3eff:fe74:4156#123, interface stats: received=0, sent=0, dropped=0, active_time=14116 secs Oct 21 20:23:05 nodehv01 ntpd[3047]: peers refreshed Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150 [23864]: e3b7a377 aio collect 0x7ffd08000910:0x7ffd08000920:0x7ffd008f2000 result 1048576:0 close free Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150 [23864]: e3b7a377 close_task_aio 0 0x7ffd080008c0 busy Oct 21 20:23:05 nodehv01 sanlock[2946]: 2014-10-21 20:23:05+0200 160150 [23864]: e3b7a377 close_task_aio 3 0x7ffd080009b0 busy Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151 [23969]: bcfa7ec4 aio collect 0x7ffcf4000960:0x7ffcf4000970:0x7ffd007f0000 result 1048576:0 close free Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151 [24471]: hosted-e aio collect 0x7ffcec000910:0x7ffcec000920:0x7ffd006ee000 result 1048576:0 close free Oct 21 20:23:07 nodehv01 sanlock[2946]: 2014-10-21 20:23:07+0200 160151 [24471]: hosted-e close_task_aio 0 0x7ffcec0008c0 busy Oct 21 20:23:09 nodehv01 sanlock[2946]: 2014-10-21 20:23:09+0200 160154 [23864]: e3b7a377 aio collect 0x7ffd080008c0:0x7ffd080008d0:0x7ffd1005e000 result 1048576:0 close free Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155 [23969]: bcfa7ec4 aio collect 0x7ffcf4000910:0x7ffcf4000920:0x7ffd009f4000 result 1048576:0 close free Oct 21 20:23:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats: 'pid' Oct 21 20:23:11 nodehv01 sanlock[2946]: 2014-10-21 20:23:11+0200 160155 [24471]: hosted-e aio collect 0x7ffcec0008c0:0x7ffcec0008d0:0x7ffd00af6000 result 1048576:0 close free Oct 21 20:23:12 nodehv01 sanlock[2946]: 2014-10-21 20:23:12+0200 160157 [23864]: e3b7a377 aio collect 0x7ffd080009b0:0x7ffd080009c0:0x7ffd002e6000 result 1048576:0 close free Oct 21 20:24:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats: 'pid' Oct 21 20:25:11 nodehv01 ovirt-ha-broker cpu_load_no_engine.EngineHealth ERROR Failed to getVmStats: 'pid' Oct 21 20:26:05 nodehv01 ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine vm died unexpectedly Oct 21 20:26:06 nodehv01 kernel: device-mapper: table: 253:28: multipath: error getting device -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767