Feb  9 03:40:01 vm02 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1919" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb  9 17:27:48 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:27:58 vm02 sanlock[2561]: 2014-02-09 17:27:58-0600 261285 [16497]: s5 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:27:58 vm02 sanlock[2561]: 2014-02-09 17:27:58-0600 261285 [16497]: s5 renewal error -202 delta_length 10 last_success 261254
Feb  9 17:27:58 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:27:59 vm02 kernel: ------------[ cut here ]------------
Feb  9 17:27:59 vm02 kernel: WARNING: at kernel/softirq.c:159 local_bh_enable_ip+0x7d/0xb0() (Not tainted)
Feb  9 17:27:59 vm02 kernel: Hardware name: H8DMT-IBX
Feb  9 17:27:59 vm02 kernel: Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xprtrdma ebt_arp nfs fscache auth_rpcgss nfs_acl bonding ebtable_nat ebtables softdog lockd sunrpc powernow_k8 freq_table mperf 8021q garp bridge stp llc xt_physdev ipt_REJECT xt_comment xt_multiport ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 dm_round_robin dm_multipath vhost_net macvtap macvlan tun kvm_amd kvm microcode serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd igb dca i2c_algo_bit ptp pps_core mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core sg i2c_nforce2 i2c_core ext4 jbd2 mbcache raid1 sd_mod crc_t10dif sata_nv ata_generic pata_acpi pata_amd dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]
Feb  9 17:27:59 vm02 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-431.3.1.el6.x86_64 #1
Feb  9 17:27:59 vm02 kernel: Call Trace:
Feb  9 17:27:59 vm02 kernel: <IRQ>  [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Feb  9 17:27:59 vm02 kernel: [<ffffffff81071e7a>] ? warn_slowpath_null+0x1a/0x20
Feb  9 17:27:59 vm02 kernel: [<ffffffff8107a3ed>] ? local_bh_enable_ip+0x7d/0xb0
Feb  9 17:27:59 vm02 kernel: [<ffffffff8152a4fb>] ? _spin_unlock_bh+0x1b/0x20
Feb  9 17:27:59 vm02 kernel: [<ffffffffa04504f0>] ? rpc_wake_up_status+0x70/0x80 [sunrpc]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa044979c>] ? xprt_wake_pending_tasks+0x2c/0x30 [sunrpc]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa05562fc>] ? rpcrdma_conn_func+0x9c/0xb0 [xprtrdma]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa0559450>] ? rpcrdma_qp_async_error_upcall+0x40/0x80 [xprtrdma]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa019a1cb>] ? mlx4_ib_qp_event+0x8b/0x100 [mlx4_ib]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa0138c54>] ? mlx4_qp_event+0x74/0xf0 [mlx4_core]
Feb  9 17:27:59 vm02 kernel: [<ffffffffa0126057>] ? mlx4_eq_int+0x557/0xcb0 [mlx4_core]
Feb  9 17:27:59 vm02 kernel: [<ffffffff8109f352>] ? enqueue_hrtimer+0x82/0xd0
Feb  9 17:27:59 vm02 kernel: [<ffffffffa01267c4>] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core]
Feb  9 17:27:59 vm02 kernel: [<ffffffff810e6ed0>] ? handle_IRQ_event+0x60/0x170
Feb  9 17:27:59 vm02 kernel: [<ffffffff810e982e>] ? handle_edge_irq+0xde/0x180
Feb  9 17:27:59 vm02 kernel: [<ffffffff8100faf9>] ? handle_irq+0x49/0xa0
Feb  9 17:27:59 vm02 kernel: [<ffffffff81530fec>] ? do_IRQ+0x6c/0xf0
Feb  9 17:27:59 vm02 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Feb  9 17:27:59 vm02 kernel: <EOI>  [<ffffffff8103eacb>] ? native_safe_halt+0xb/0x10
Feb  9 17:27:59 vm02 kernel: [<ffffffff810167bd>] ? default_idle+0x4d/0xb0
Feb  9 17:27:59 vm02 kernel: [<ffffffff810168bd>] ? c1e_idle+0x9d/0x120
Feb  9 17:27:59 vm02 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Feb  9 17:27:59 vm02 kernel: [<ffffffff8150cc1a>] ? rest_init+0x7a/0x80
Feb  9 17:27:59 vm02 kernel: [<ffffffff81c26f8f>] ? start_kernel+0x424/0x430
Feb  9 17:27:59 vm02 kernel: [<ffffffff81c2633a>] ? x86_64_start_reservations+0x125/0x129
Feb  9 17:27:59 vm02 kernel: [<ffffffff81c26453>] ? x86_64_start_kernel+0x115/0x124
Feb  9 17:27:59 vm02 kernel: ---[ end trace d69aebd5ff46a669 ]---
Feb  9 17:27:59 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:28:09 vm02 sanlock[2561]: 2014-02-09 17:28:09-0600 261296 [16497]: s5 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:28:09 vm02 sanlock[2561]: 2014-02-09 17:28:09-0600 261296 [16497]: s5 renewal error -202 delta_length 11 last_success 261254
Feb  9 17:28:19 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:28:20 vm02 sanlock[2561]: 2014-02-09 17:28:20-0600 261307 [16497]: s5 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:28:20 vm02 sanlock[2561]: 2014-02-09 17:28:20-0600 261307 [16497]: s5 renewal error -202 delta_length 11 last_success 261254
Feb  9 17:28:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:28:27 vm02 sanlock[2561]: 2014-02-09 17:28:27-0600 261314 [2561]: s5 check_our_lease warning 60 last_success 261254
Feb  9 17:28:28 vm02 sanlock[2561]: 2014-02-09 17:28:28-0600 261315 [2561]: s5 check_our_lease warning 61 last_success 261254
Feb  9 17:28:29 vm02 sanlock[2561]: 2014-02-09 17:28:29-0600 261316 [2561]: s5 check_our_lease warning 62 last_success 261254
Feb  9 17:28:30 vm02 sanlock[2561]: 2014-02-09 17:28:30-0600 261317 [2561]: s5 check_our_lease warning 63 last_success 261254
Feb  9 17:28:31 vm02 sanlock[2561]: 2014-02-09 17:28:31-0600 261318 [16497]: s5 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:28:31 vm02 sanlock[2561]: 2014-02-09 17:28:31-0600 261318 [16497]: s5 renewal error -202 delta_length 11 last_success 261254
Feb  9 17:28:31 vm02 sanlock[2561]: 2014-02-09 17:28:31-0600 261318 [2561]: s5 check_our_lease warning 64 last_success 261254
Feb  9 17:28:32 vm02 sanlock[2561]: 2014-02-09 17:28:32-0600 261319 [2561]: s5 check_our_lease warning 65 last_success 261254
Feb  9 17:28:33 vm02 sanlock[2561]: 2014-02-09 17:28:33-0600 261320 [2561]: s5 check_our_lease warning 66 last_success 261254
Feb  9 17:28:34 vm02 sanlock[2561]: 2014-02-09 17:28:34-0600 261321 [2561]: s5 check_our_lease warning 67 last_success 261254
Feb  9 17:28:35 vm02 sanlock[2561]: 2014-02-09 17:28:35-0600 261322 [2561]: s5 check_our_lease warning 68 last_success 261254
Feb  9 17:28:36 vm02 sanlock[2561]: 2014-02-09 17:28:36-0600 261323 [2561]: s5 check_our_lease warning 69 last_success 261254
Feb  9 17:28:37 vm02 sanlock[2561]: 2014-02-09 17:28:37-0600 261324 [2561]: s5 check_our_lease warning 70 last_success 261254
Feb  9 17:28:38 vm02 sanlock[2561]: 2014-02-09 17:28:38-0600 261325 [2561]: s5 check_our_lease warning 71 last_success 261254
Feb  9 17:28:39 vm02 sanlock[2561]: 2014-02-09 17:28:39-0600 261326 [2561]: s5 check_our_lease warning 72 last_success 261254
Feb  9 17:28:40 vm02 sanlock[2561]: 2014-02-09 17:28:40-0600 261327 [2561]: s5 check_our_lease warning 73 last_success 261254
Feb  9 17:28:41 vm02 wdmd[2550]: test warning now 261328 ping 261318 close 0 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:41 vm02 wdmd[2550]: /dev/watchdog closed unclean
Feb  9 17:28:41 vm02 kernel: SoftDog: Unexpected close, not stopping watchdog!
Feb  9 17:28:41 vm02 sanlock[2561]: 2014-02-09 17:28:41-0600 261328 [2561]: s5 check_our_lease warning 74 last_success 261254
Feb  9 17:28:42 vm02 sanlock[2561]: 2014-02-09 17:28:42-0600 261329 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:28:42 vm02 sanlock[2561]: 2014-02-09 17:28:42-0600 261329 [16497]: s5 renewal error -2 delta_length 11 last_success 261254
Feb  9 17:28:42 vm02 wdmd[2550]: test warning now 261329 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:42 vm02 sanlock[2561]: 2014-02-09 17:28:42-0600 261329 [2561]: s5 check_our_lease warning 75 last_success 261254
Feb  9 17:28:43 vm02 wdmd[2550]: test warning now 261330 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:43 vm02 sanlock[2561]: 2014-02-09 17:28:43-0600 261330 [2561]: s5 check_our_lease warning 76 last_success 261254
Feb  9 17:28:44 vm02 wdmd[2550]: test warning now 261331 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:44 vm02 sanlock[2561]: 2014-02-09 17:28:44-0600 261331 [2561]: s5 check_our_lease warning 77 last_success 261254
Feb  9 17:28:45 vm02 wdmd[2550]: test warning now 261332 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:45 vm02 sanlock[2561]: 2014-02-09 17:28:45-0600 261332 [2561]: s5 check_our_lease warning 78 last_success 261254
Feb  9 17:28:46 vm02 wdmd[2550]: test warning now 261333 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:46 vm02 sanlock[2561]: 2014-02-09 17:28:46-0600 261333 [2561]: s5 check_our_lease warning 79 last_success 261254
Feb  9 17:28:47 vm02 wdmd[2550]: test failed rem 54 now 261334 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:47 vm02 sanlock[2561]: 2014-02-09 17:28:47-0600 261334 [2561]: s5 check_our_lease failed 80
Feb  9 17:28:47 vm02 sanlock[2561]: 2014-02-09 17:28:47-0600 261334 [2561]: s5 kill 15270 sig 15 count 1
Feb  9 17:28:48 vm02 sanlock[2561]: 2014-02-09 17:28:48-0600 261335 [2561]: s5 kill 15270 sig 15 count 2
Feb  9 17:28:48 vm02 wdmd[2550]: test failed rem 53 now 261335 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:49 vm02 sanlock[2561]: 2014-02-09 17:28:49-0600 261336 [2561]: s5 kill 15270 sig 15 count 3
Feb  9 17:28:49 vm02 wdmd[2550]: test failed rem 52 now 261336 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:50 vm02 sanlock[2561]: 2014-02-09 17:28:50-0600 261337 [2561]: s5 kill 15270 sig 15 count 4
Feb  9 17:28:50 vm02 wdmd[2550]: test failed rem 51 now 261337 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:50 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:28:50 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:28:50 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:28:51 vm02 sanlock[2561]: 2014-02-09 17:28:51-0600 261338 [2561]: s5 kill 15270 sig 15 count 5
Feb  9 17:28:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:28:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:28:51 vm02 wdmd[2550]: test failed rem 50 now 261338 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:52 vm02 sanlock[2561]: 2014-02-09 17:28:52-0600 261339 [2561]: s5 kill 15270 sig 15 count 6
Feb  9 17:28:52 vm02 wdmd[2550]: test failed rem 49 now 261339 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:52 vm02 sanlock[2561]: 2014-02-09 17:28:52-0600 261339 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:28:52 vm02 sanlock[2561]: 2014-02-09 17:28:52-0600 261339 [16497]: s5 renewal error -2 delta_length 10 last_success 261254
Feb  9 17:28:53 vm02 sanlock[2561]: 2014-02-09 17:28:53-0600 261340 [2561]: s5 kill 15270 sig 15 count 7
Feb  9 17:28:53 vm02 wdmd[2550]: test failed rem 48 now 261340 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:54 vm02 sanlock[2561]: 2014-02-09 17:28:54-0600 261341 [2561]: s5 kill 15270 sig 15 count 8
Feb  9 17:28:54 vm02 wdmd[2550]: test failed rem 47 now 261341 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:55 vm02 sanlock[2561]: 2014-02-09 17:28:55-0600 261342 [2561]: s5 kill 15270 sig 15 count 9
Feb  9 17:28:55 vm02 wdmd[2550]: test failed rem 46 now 261342 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:56 vm02 sanlock[2561]: 2014-02-09 17:28:56-0600 261343 [2561]: s5 kill 15270 sig 15 count 10
Feb  9 17:28:56 vm02 wdmd[2550]: test failed rem 45 now 261343 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:57 vm02 sanlock[2561]: 2014-02-09 17:28:57-0600 261344 [2561]: s5 kill 15270 sig 15 count 11
Feb  9 17:28:57 vm02 wdmd[2550]: test failed rem 44 now 261344 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:58 vm02 sanlock[2561]: 2014-02-09 17:28:58-0600 261345 [2561]: s5 kill 15270 sig 15 count 12
Feb  9 17:28:58 vm02 wdmd[2550]: test failed rem 43 now 261345 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:28:59 vm02 sanlock[2561]: 2014-02-09 17:28:59-0600 261346 [2561]: s5 kill 15270 sig 15 count 13
Feb  9 17:28:59 vm02 wdmd[2550]: test failed rem 42 now 261346 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:00 vm02 sanlock[2561]: 2014-02-09 17:29:00-0600 261347 [2561]: s5 kill 15270 sig 15 count 14
Feb  9 17:29:00 vm02 wdmd[2550]: test failed rem 41 now 261347 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:01 vm02 sanlock[2561]: 2014-02-09 17:29:01-0600 261348 [2561]: s5 kill 15270 sig 15 count 15
Feb  9 17:29:01 vm02 wdmd[2550]: test failed rem 40 now 261348 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:02 vm02 sanlock[2561]: 2014-02-09 17:29:02-0600 261349 [2561]: s5 kill 15270 sig 15 count 16
Feb  9 17:29:02 vm02 wdmd[2550]: test failed rem 39 now 261349 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:03 vm02 sanlock[2561]: 2014-02-09 17:29:03-0600 261350 [2561]: s5 kill 15270 sig 15 count 17
Feb  9 17:29:03 vm02 sanlock[2561]: 2014-02-09 17:29:03-0600 261350 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:29:03 vm02 sanlock[2561]: 2014-02-09 17:29:03-0600 261350 [16497]: s5 renewal error -2 delta_length 10 last_success 261254
Feb  9 17:29:03 vm02 wdmd[2550]: test failed rem 38 now 261350 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:04 vm02 sanlock[2561]: 2014-02-09 17:29:04-0600 261351 [2561]: s5 kill 15270 sig 15 count 18
Feb  9 17:29:04 vm02 wdmd[2550]: test failed rem 37 now 261351 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:05 vm02 sanlock[2561]: 2014-02-09 17:29:05-0600 261352 [2561]: s5 kill 15270 sig 15 count 19
Feb  9 17:29:05 vm02 wdmd[2550]: test failed rem 36 now 261352 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:06 vm02 sanlock[2561]: 2014-02-09 17:29:06-0600 261353 [2561]: s5 kill 15270 sig 15 count 20
Feb  9 17:29:06 vm02 wdmd[2550]: test failed rem 35 now 261353 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:07 vm02 sanlock[2561]: 2014-02-09 17:29:07-0600 261354 [2561]: s5 kill 15270 sig 15 count 21
Feb  9 17:29:07 vm02 wdmd[2550]: test failed rem 34 now 261354 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:08 vm02 sanlock[2561]: 2014-02-09 17:29:08-0600 261355 [2561]: s5 kill 15270 sig 15 count 22
Feb  9 17:29:08 vm02 wdmd[2550]: test failed rem 33 now 261355 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:09 vm02 sanlock[2561]: 2014-02-09 17:29:09-0600 261356 [2561]: s5 kill 15270 sig 15 count 23
Feb  9 17:29:09 vm02 wdmd[2550]: test failed rem 32 now 261356 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:10 vm02 sanlock[2561]: 2014-02-09 17:29:10-0600 261357 [2561]: s5 kill 15270 sig 15 count 24
Feb  9 17:29:10 vm02 wdmd[2550]: test failed rem 31 now 261357 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:11 vm02 sanlock[2561]: 2014-02-09 17:29:11-0600 261358 [2561]: s5 kill 15270 sig 15 count 25
Feb  9 17:29:11 vm02 wdmd[2550]: test failed rem 30 now 261358 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:12 vm02 sanlock[2561]: 2014-02-09 17:29:12-0600 261359 [2561]: s5 kill 15270 sig 15 count 26
Feb  9 17:29:12 vm02 wdmd[2550]: test failed rem 29 now 261359 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:13 vm02 sanlock[2561]: 2014-02-09 17:29:13-0600 261360 [2561]: s5 kill 15270 sig 15 count 27
Feb  9 17:29:13 vm02 wdmd[2550]: test failed rem 28 now 261360 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:13 vm02 sanlock[2561]: 2014-02-09 17:29:13-0600 261360 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:29:13 vm02 sanlock[2561]: 2014-02-09 17:29:13-0600 261360 [16497]: s5 renewal error -2 delta_length 10 last_success 261254
Feb  9 17:29:14 vm02 sanlock[2561]: 2014-02-09 17:29:14-0600 261361 [2561]: s5 kill 15270 sig 15 count 28
Feb  9 17:29:14 vm02 wdmd[2550]: test failed rem 27 now 261361 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:15 vm02 sanlock[2561]: 2014-02-09 17:29:15-0600 261362 [2561]: s5 kill 15270 sig 15 count 29
Feb  9 17:29:15 vm02 wdmd[2550]: test failed rem 26 now 261362 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:16 vm02 sanlock[2561]: 2014-02-09 17:29:16-0600 261363 [2561]: s5 kill 15270 sig 15 count 30
Feb  9 17:29:16 vm02 wdmd[2550]: test failed rem 25 now 261363 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:17 vm02 sanlock[2561]: 2014-02-09 17:29:17-0600 261364 [2561]: s5 kill 15270 sig 15 count 31
Feb  9 17:29:17 vm02 wdmd[2550]: test failed rem 24 now 261364 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:18 vm02 sanlock[2561]: 2014-02-09 17:29:18-0600 261365 [2561]: s5 kill 15270 sig 15 count 32
Feb  9 17:29:18 vm02 wdmd[2550]: test failed rem 23 now 261365 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:19 vm02 sanlock[2561]: 2014-02-09 17:29:19-0600 261366 [2561]: s5 kill 15270 sig 15 count 33
Feb  9 17:29:19 vm02 wdmd[2550]: test failed rem 22 now 261366 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:20 vm02 sanlock[2561]: 2014-02-09 17:29:20-0600 261367 [2561]: s5 kill 15270 sig 15 count 34
Feb  9 17:29:20 vm02 wdmd[2550]: test failed rem 21 now 261367 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:21 vm02 sanlock[2561]: 2014-02-09 17:29:21-0600 261368 [2561]: s5 kill 15270 sig 15 count 35
Feb  9 17:29:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:29:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:29:21 vm02 wdmd[2550]: test failed rem 20 now 261368 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:22 vm02 sanlock[2561]: 2014-02-09 17:29:22-0600 261369 [2561]: s5 kill 15270 sig 15 count 36
Feb  9 17:29:22 vm02 wdmd[2550]: test failed rem 19 now 261369 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:23 vm02 sanlock[2561]: 2014-02-09 17:29:23-0600 261370 [2561]: s5 kill 15270 sig 15 count 37
Feb  9 17:29:23 vm02 wdmd[2550]: test failed rem 18 now 261370 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:24 vm02 sanlock[2561]: 2014-02-09 17:29:24-0600 261371 [2561]: s5 kill 15270 sig 15 count 38
Feb  9 17:29:24 vm02 sanlock[2561]: 2014-02-09 17:29:24-0600 261371 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:29:24 vm02 sanlock[2561]: 2014-02-09 17:29:24-0600 261371 [16497]: s5 renewal error -2 delta_length 10 last_success 261254
Feb  9 17:29:24 vm02 wdmd[2550]: test failed rem 17 now 261371 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:25 vm02 sanlock[2561]: 2014-02-09 17:29:25-0600 261372 [2561]: s5 kill 15270 sig 15 count 39
Feb  9 17:29:25 vm02 wdmd[2550]: test failed rem 16 now 261372 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:26 vm02 sanlock[2561]: 2014-02-09 17:29:26-0600 261373 [2561]: s5 kill 15270 sig 15 count 40
Feb  9 17:29:26 vm02 wdmd[2550]: test failed rem 15 now 261373 ping 261318 close 261328 renewal 261254 expire 261334 client 2561 sanlock_7c49750d-7eae-4cd2-9b63-1dc71f357b88:2
Feb  9 17:29:27 vm02 sanlock[2561]: 2014-02-09 17:29:27-0600 261374 [2561]: s5 kill 15270 sig 9 count 41
Feb  9 17:29:27 vm02 sanlock[2561]: 2014-02-09 17:29:27-0600 261374 [2561]: dead 15270 ci 4 count 41
Feb  9 17:29:27 vm02 respawn: slave '/usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid' died, respawning slave
Feb  9 17:29:27 vm02 sanlock[2561]: 2014-02-09 17:29:27-0600 261374 [2561]: s5 all pids clear
Feb  9 17:29:27 vm02 wdmd[2550]: /dev/watchdog reopen
Feb  9 17:29:28 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:29:28 vm02 vdsm vds WARNING Unable to load the json rpc server module. Please make sure it is installed.
Feb  9 17:29:28 vm02 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::Unknown type found, device: '{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}' found
Feb  9 17:29:28 vm02 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::Unknown type found, device: '{'device': 'unix', 'alias': 'channel1', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '2'}}' found
Feb  9 17:29:29 vm02 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::_readPauseCode unsupported by libvirt vm
Feb  9 17:29:29 vm02 kernel: ata1: soft resetting link
Feb  9 17:29:29 vm02 kernel: ata2: EH complete
Feb  9 17:29:29 vm02 kernel: ata5: hard resetting link
Feb  9 17:29:29 vm02 kernel: ata6: hard resetting link
Feb  9 17:29:29 vm02 kernel: ata4: hard resetting link
Feb  9 17:29:29 vm02 kernel: ata4: nv: skipping hardreset on occupied port
Feb  9 17:29:29 vm02 kernel: ata3: hard resetting link
Feb  9 17:29:29 vm02 kernel: ata3: nv: skipping hardreset on occupied port
Feb  9 17:29:29 vm02 kernel: ata1: EH complete
Feb  9 17:29:30 vm02 kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:29:30 vm02 kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:29:30 vm02 kernel: ata4.00: configured for UDMA/133
Feb  9 17:29:30 vm02 kernel: ata4: EH complete
Feb  9 17:29:30 vm02 kernel: ata3.00: configured for UDMA/133
Feb  9 17:29:30 vm02 kernel: ata3: EH complete
Feb  9 17:29:30 vm02 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Feb  9 17:29:30 vm02 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Feb  9 17:29:30 vm02 kernel: ata5: EH complete
Feb  9 17:29:30 vm02 kernel: ata6: EH complete
Feb  9 17:29:31 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:29:31 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:29:31 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:29:31 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:29:32 vm02 kernel: ata1: soft resetting link
Feb  9 17:29:32 vm02 kernel: ata2: EH complete
Feb  9 17:29:32 vm02 kernel: ata3: hard resetting link
Feb  9 17:29:32 vm02 kernel: ata3: nv: skipping hardreset on occupied port
Feb  9 17:29:32 vm02 kernel: ata4: hard resetting link
Feb  9 17:29:32 vm02 kernel: ata4: nv: skipping hardreset on occupied port
Feb  9 17:29:32 vm02 kernel: ata5: hard resetting link
Feb  9 17:29:32 vm02 kernel: ata6: hard resetting link
Feb  9 17:29:33 vm02 kernel: ata1: EH complete
Feb  9 17:29:33 vm02 kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:29:33 vm02 kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:29:33 vm02 kernel: ata3.00: configured for UDMA/133
Feb  9 17:29:33 vm02 kernel: ata3: EH complete
Feb  9 17:29:33 vm02 kernel: ata4.00: configured for UDMA/133
Feb  9 17:29:33 vm02 kernel: ata4: EH complete
Feb  9 17:29:33 vm02 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Feb  9 17:29:33 vm02 kernel: ata5: EH complete
Feb  9 17:29:33 vm02 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Feb  9 17:29:33 vm02 kernel: ata6: EH complete
Feb  9 17:29:33 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:29:34 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/import_export: rdma,port=20049
Feb  9 17:29:34 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/iso: rdma,port=20049
Feb  9 17:29:34 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/data: rdma,port=20049
Feb  9 17:29:34 vm02 sanlock[2561]: 2014-02-09 17:29:34-0600 261381 [16497]: s5 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data/7c49750d-7eae-4cd2-9b63-1dc71f357b88/dom_md/ids
Feb  9 17:29:34 vm02 sanlock[2561]: 2014-02-09 17:29:34-0600 261381 [16497]: s5 renewal error -2 delta_length 10 last_success 261254
Feb  9 17:29:34 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:29:34 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:29:34 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:29:34 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:29:37 vm02 vdsm TaskManager.Task ERROR Task=`65a7e7ed-6744-4675-972d-9768166735be`::Unexpected error#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/task.py", line 857, in _run#012    return fn(*args, **kargs)#012  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012    res = f(*args, **kwargs)#012  File "/usr/share/vdsm/storage/hsm.py", line 2110, in getAllTasksStatuses#012    raise se.SpmStatusError()#012SpmStatusError: Not SPM: ()
Feb  9 17:29:44 vm02 sanlock[2561]: 2014-02-09 17:29:44-0600 261391 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:29:44 vm02 sanlock[2561]: 2014-02-09 17:29:44-0600 261391 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:29:44 vm02 sanlock[2561]: 2014-02-09 17:29:44-0600 261391 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:29:44 vm02 sanlock[2561]: 2014-02-09 17:29:44-0600 261391 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:29:45 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:29:45 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:29:45 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:29:45 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:29:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:29:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:29:54 vm02 sanlock[2561]: 2014-02-09 17:29:54-0600 261401 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:29:54 vm02 sanlock[2561]: 2014-02-09 17:29:54-0600 261401 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:29:54 vm02 sanlock[2561]: 2014-02-09 17:29:54-0600 261401 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:29:54 vm02 sanlock[2561]: 2014-02-09 17:29:54-0600 261401 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:04 vm02 sanlock[2561]: 2014-02-09 17:30:04-0600 261411 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:04 vm02 sanlock[2561]: 2014-02-09 17:30:04-0600 261411 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:04 vm02 sanlock[2561]: 2014-02-09 17:30:04-0600 261411 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:04 vm02 sanlock[2561]: 2014-02-09 17:30:04-0600 261411 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:14 vm02 sanlock[2561]: 2014-02-09 17:30:14-0600 261421 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:14 vm02 sanlock[2561]: 2014-02-09 17:30:14-0600 261421 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:14 vm02 sanlock[2561]: 2014-02-09 17:30:14-0600 261421 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:14 vm02 sanlock[2561]: 2014-02-09 17:30:14-0600 261421 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:30:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:30:24 vm02 sanlock[2561]: 2014-02-09 17:30:24-0600 261431 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:24 vm02 sanlock[2561]: 2014-02-09 17:30:24-0600 261431 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:24 vm02 sanlock[2561]: 2014-02-09 17:30:24-0600 261431 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:24 vm02 sanlock[2561]: 2014-02-09 17:30:24-0600 261431 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:34 vm02 sanlock[2561]: 2014-02-09 17:30:34-0600 261441 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:34 vm02 sanlock[2561]: 2014-02-09 17:30:34-0600 261441 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:34 vm02 sanlock[2561]: 2014-02-09 17:30:34-0600 261441 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:34 vm02 sanlock[2561]: 2014-02-09 17:30:34-0600 261441 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:35 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:30:35 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:30:35 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:30:35 vm02 kernel: ata1: soft resetting link
Feb  9 17:30:35 vm02 kernel: ata2: EH complete
Feb  9 17:30:35 vm02 kernel: ata1: EH complete
Feb  9 17:30:36 vm02 kernel: ata3: hard resetting link
Feb  9 17:30:36 vm02 kernel: ata3: nv: skipping hardreset on occupied port
Feb  9 17:30:36 vm02 kernel: ata4: hard resetting link
Feb  9 17:30:36 vm02 kernel: ata4: nv: skipping hardreset on occupied port
Feb  9 17:30:36 vm02 kernel: ata5: hard resetting link
Feb  9 17:30:36 vm02 kernel: ata6: hard resetting link
Feb  9 17:30:36 vm02 kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:30:36 vm02 kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:30:36 vm02 kernel: ata3.00: configured for UDMA/133
Feb  9 17:30:36 vm02 kernel: ata3: EH complete
Feb  9 17:30:36 vm02 kernel: ata4.00: configured for UDMA/133
Feb  9 17:30:36 vm02 kernel: ata4: EH complete
Feb  9 17:30:37 vm02 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Feb  9 17:30:37 vm02 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Feb  9 17:30:37 vm02 kernel: ata5: EH complete
Feb  9 17:30:37 vm02 kernel: ata6: EH complete
Feb  9 17:30:38 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:30:38 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:30:38 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:30:38 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:30:38 vm02 udevd-work[12770]: setfilecon /dev/dm-2 failed: No such file or directory
Feb  9 17:30:44 vm02 sanlock[2561]: 2014-02-09 17:30:44-0600 261451 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:44 vm02 sanlock[2561]: 2014-02-09 17:30:44-0600 261451 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:44 vm02 sanlock[2561]: 2014-02-09 17:30:44-0600 261451 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:44 vm02 sanlock[2561]: 2014-02-09 17:30:44-0600 261451 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:30:48 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:30:48 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:30:48 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:30:48 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:30:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:30:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:30:54 vm02 sanlock[2561]: 2014-02-09 17:30:54-0600 261461 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:30:54 vm02 sanlock[2561]: 2014-02-09 17:30:54-0600 261461 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:30:54 vm02 sanlock[2561]: 2014-02-09 17:30:54-0600 261461 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:30:54 vm02 sanlock[2561]: 2014-02-09 17:30:54-0600 261461 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:04 vm02 sanlock[2561]: 2014-02-09 17:31:04-0600 261471 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:04 vm02 sanlock[2561]: 2014-02-09 17:31:04-0600 261471 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:04 vm02 sanlock[2561]: 2014-02-09 17:31:04-0600 261471 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:04 vm02 sanlock[2561]: 2014-02-09 17:31:04-0600 261471 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:14 vm02 sanlock[2561]: 2014-02-09 17:31:14-0600 261481 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:14 vm02 sanlock[2561]: 2014-02-09 17:31:14-0600 261481 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:14 vm02 sanlock[2561]: 2014-02-09 17:31:14-0600 261481 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:14 vm02 sanlock[2561]: 2014-02-09 17:31:14-0600 261481 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:31:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:31:24 vm02 sanlock[2561]: 2014-02-09 17:31:24-0600 261491 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:24 vm02 sanlock[2561]: 2014-02-09 17:31:24-0600 261491 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:24 vm02 sanlock[2561]: 2014-02-09 17:31:24-0600 261491 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:24 vm02 sanlock[2561]: 2014-02-09 17:31:24-0600 261491 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:34 vm02 sanlock[2561]: 2014-02-09 17:31:34-0600 261501 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:34 vm02 sanlock[2561]: 2014-02-09 17:31:34-0600 261501 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:34 vm02 sanlock[2561]: 2014-02-09 17:31:34-0600 261501 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:34 vm02 sanlock[2561]: 2014-02-09 17:31:34-0600 261501 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:38 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:31:38 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:31:38 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:31:38 vm02 vdsm TaskManager.Task ERROR Task=`0c4a3078-e9d3-426b-869a-df59c3a1c8ca`::Unexpected error#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/task.py", line 857, in _run#012    return fn(*args, **kargs)#012  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012    res = f(*args, **kwargs)#012  File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool#012    masterVersion, options)#012  File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool#012    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 699, in connect#012    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild#012    masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain#012    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)#012StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=7c49750d-7eae-4cd2-9b63-1dc71f357b88'
Feb  9 17:31:44 vm02 sanlock[2561]: 2014-02-09 17:31:44-0600 261511 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:44 vm02 sanlock[2561]: 2014-02-09 17:31:44-0600 261511 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:44 vm02 sanlock[2561]: 2014-02-09 17:31:44-0600 261511 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:44 vm02 sanlock[2561]: 2014-02-09 17:31:44-0600 261511 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:31:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:31:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:31:54 vm02 sanlock[2561]: 2014-02-09 17:31:54-0600 261521 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:31:54 vm02 sanlock[2561]: 2014-02-09 17:31:54-0600 261521 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:31:54 vm02 sanlock[2561]: 2014-02-09 17:31:54-0600 261521 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:31:54 vm02 sanlock[2561]: 2014-02-09 17:31:54-0600 261521 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:04 vm02 sanlock[2561]: 2014-02-09 17:32:04-0600 261531 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:04 vm02 sanlock[2561]: 2014-02-09 17:32:04-0600 261531 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:04 vm02 sanlock[2561]: 2014-02-09 17:32:04-0600 261531 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:04 vm02 sanlock[2561]: 2014-02-09 17:32:04-0600 261531 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:14 vm02 sanlock[2561]: 2014-02-09 17:32:14-0600 261541 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:14 vm02 sanlock[2561]: 2014-02-09 17:32:14-0600 261541 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:14 vm02 sanlock[2561]: 2014-02-09 17:32:14-0600 261541 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:14 vm02 sanlock[2561]: 2014-02-09 17:32:14-0600 261541 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:32:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:32:24 vm02 sanlock[2561]: 2014-02-09 17:32:24-0600 261551 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:24 vm02 sanlock[2561]: 2014-02-09 17:32:24-0600 261551 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:24 vm02 sanlock[2561]: 2014-02-09 17:32:24-0600 261551 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:24 vm02 sanlock[2561]: 2014-02-09 17:32:24-0600 261551 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:34 vm02 sanlock[2561]: 2014-02-09 17:32:34-0600 261561 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:34 vm02 sanlock[2561]: 2014-02-09 17:32:34-0600 261561 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:34 vm02 sanlock[2561]: 2014-02-09 17:32:34-0600 261561 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:34 vm02 sanlock[2561]: 2014-02-09 17:32:34-0600 261561 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:44 vm02 sanlock[2561]: 2014-02-09 17:32:44-0600 261571 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:44 vm02 sanlock[2561]: 2014-02-09 17:32:44-0600 261571 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:44 vm02 sanlock[2561]: 2014-02-09 17:32:44-0600 261571 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:44 vm02 sanlock[2561]: 2014-02-09 17:32:44-0600 261571 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:32:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:32:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:32:54 vm02 sanlock[2561]: 2014-02-09 17:32:54-0600 261581 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:32:54 vm02 sanlock[2561]: 2014-02-09 17:32:54-0600 261581 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:32:54 vm02 sanlock[2561]: 2014-02-09 17:32:54-0600 261581 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:32:54 vm02 sanlock[2561]: 2014-02-09 17:32:54-0600 261581 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:04 vm02 sanlock[2561]: 2014-02-09 17:33:04-0600 261591 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:04 vm02 sanlock[2561]: 2014-02-09 17:33:04-0600 261591 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:04 vm02 sanlock[2561]: 2014-02-09 17:33:04-0600 261591 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:04 vm02 sanlock[2561]: 2014-02-09 17:33:04-0600 261591 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:14 vm02 sanlock[2561]: 2014-02-09 17:33:14-0600 261601 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:14 vm02 sanlock[2561]: 2014-02-09 17:33:14-0600 261601 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:14 vm02 sanlock[2561]: 2014-02-09 17:33:14-0600 261601 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:14 vm02 sanlock[2561]: 2014-02-09 17:33:14-0600 261601 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:33:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:33:24 vm02 sanlock[2561]: 2014-02-09 17:33:24-0600 261611 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:24 vm02 sanlock[2561]: 2014-02-09 17:33:24-0600 261611 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:24 vm02 sanlock[2561]: 2014-02-09 17:33:24-0600 261611 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:24 vm02 sanlock[2561]: 2014-02-09 17:33:24-0600 261611 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:34 vm02 sanlock[2561]: 2014-02-09 17:33:34-0600 261621 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:34 vm02 sanlock[2561]: 2014-02-09 17:33:34-0600 261621 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:34 vm02 sanlock[2561]: 2014-02-09 17:33:34-0600 261621 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:34 vm02 sanlock[2561]: 2014-02-09 17:33:34-0600 261621 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:44 vm02 sanlock[2561]: 2014-02-09 17:33:44-0600 261631 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:44 vm02 sanlock[2561]: 2014-02-09 17:33:44-0600 261631 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:44 vm02 sanlock[2561]: 2014-02-09 17:33:44-0600 261631 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:44 vm02 sanlock[2561]: 2014-02-09 17:33:44-0600 261631 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:33:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:33:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:33:54 vm02 sanlock[2561]: 2014-02-09 17:33:54-0600 261641 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:33:54 vm02 sanlock[2561]: 2014-02-09 17:33:54-0600 261641 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:33:54 vm02 sanlock[2561]: 2014-02-09 17:33:54-0600 261641 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:33:54 vm02 sanlock[2561]: 2014-02-09 17:33:54-0600 261641 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:04 vm02 sanlock[2561]: 2014-02-09 17:34:04-0600 261651 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:04 vm02 sanlock[2561]: 2014-02-09 17:34:04-0600 261651 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:04 vm02 sanlock[2561]: 2014-02-09 17:34:04-0600 261651 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:04 vm02 sanlock[2561]: 2014-02-09 17:34:04-0600 261651 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:14 vm02 sanlock[2561]: 2014-02-09 17:34:14-0600 261661 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:14 vm02 sanlock[2561]: 2014-02-09 17:34:14-0600 261661 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:14 vm02 sanlock[2561]: 2014-02-09 17:34:14-0600 261661 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:14 vm02 sanlock[2561]: 2014-02-09 17:34:14-0600 261661 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:34:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:34:24 vm02 sanlock[2561]: 2014-02-09 17:34:24-0600 261671 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:24 vm02 sanlock[2561]: 2014-02-09 17:34:24-0600 261671 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:24 vm02 sanlock[2561]: 2014-02-09 17:34:24-0600 261671 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:24 vm02 sanlock[2561]: 2014-02-09 17:34:24-0600 261671 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:34 vm02 sanlock[2561]: 2014-02-09 17:34:34-0600 261681 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:34 vm02 sanlock[2561]: 2014-02-09 17:34:34-0600 261681 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:34 vm02 sanlock[2561]: 2014-02-09 17:34:34-0600 261681 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:34 vm02 sanlock[2561]: 2014-02-09 17:34:34-0600 261681 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:44 vm02 sanlock[2561]: 2014-02-09 17:34:44-0600 261691 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:44 vm02 sanlock[2561]: 2014-02-09 17:34:44-0600 261691 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:44 vm02 sanlock[2561]: 2014-02-09 17:34:44-0600 261691 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:44 vm02 sanlock[2561]: 2014-02-09 17:34:44-0600 261691 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:34:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:34:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:34:54 vm02 sanlock[2561]: 2014-02-09 17:34:54-0600 261701 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:34:54 vm02 sanlock[2561]: 2014-02-09 17:34:54-0600 261701 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:34:54 vm02 sanlock[2561]: 2014-02-09 17:34:54-0600 261701 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:34:54 vm02 sanlock[2561]: 2014-02-09 17:34:54-0600 261701 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:00 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:35:02 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:35:03 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/import_export: rdma,port=20049
Feb  9 17:35:03 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/iso: rdma,port=20049
Feb  9 17:35:03 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/data: rdma,port=20049
Feb  9 17:35:03 vm02 kernel: ata1: soft resetting link
Feb  9 17:35:03 vm02 kernel: ata2: EH complete
Feb  9 17:35:03 vm02 kernel: ata3: hard resetting link
Feb  9 17:35:03 vm02 kernel: ata3: nv: skipping hardreset on occupied port
Feb  9 17:35:03 vm02 kernel: ata4: hard resetting link
Feb  9 17:35:03 vm02 kernel: ata4: nv: skipping hardreset on occupied port
Feb  9 17:35:03 vm02 kernel: ata5: hard resetting link
Feb  9 17:35:03 vm02 kernel: ata6: hard resetting link
Feb  9 17:35:03 vm02 kernel: ata1: EH complete
Feb  9 17:35:03 vm02 kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:35:03 vm02 kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:35:03 vm02 kernel: ata3.00: configured for UDMA/133
Feb  9 17:35:03 vm02 kernel: ata3: EH complete
Feb  9 17:35:03 vm02 kernel: ata4.00: configured for UDMA/133
Feb  9 17:35:03 vm02 kernel: ata4: EH complete
Feb  9 17:35:04 vm02 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Feb  9 17:35:04 vm02 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Feb  9 17:35:04 vm02 kernel: ata5: EH complete
Feb  9 17:35:04 vm02 kernel: ata6: EH complete
Feb  9 17:35:04 vm02 sanlock[2561]: 2014-02-09 17:35:04-0600 261711 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:04 vm02 sanlock[2561]: 2014-02-09 17:35:04-0600 261711 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:04 vm02 sanlock[2561]: 2014-02-09 17:35:04-0600 261711 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:04 vm02 sanlock[2561]: 2014-02-09 17:35:04-0600 261711 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:05 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:35:05 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:35:05 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:35:05 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:35:14 vm02 sanlock[2561]: 2014-02-09 17:35:14-0600 261721 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:14 vm02 sanlock[2561]: 2014-02-09 17:35:14-0600 261721 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:14 vm02 sanlock[2561]: 2014-02-09 17:35:14-0600 261721 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:14 vm02 sanlock[2561]: 2014-02-09 17:35:14-0600 261721 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:15 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:35:15 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:35:15 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:35:15 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:35:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:35:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:35:24 vm02 sanlock[2561]: 2014-02-09 17:35:24-0600 261731 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:24 vm02 sanlock[2561]: 2014-02-09 17:35:24-0600 261731 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:24 vm02 sanlock[2561]: 2014-02-09 17:35:24-0600 261731 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:24 vm02 sanlock[2561]: 2014-02-09 17:35:24-0600 261731 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:34 vm02 sanlock[2561]: 2014-02-09 17:35:34-0600 261741 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:34 vm02 sanlock[2561]: 2014-02-09 17:35:34-0600 261741 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:34 vm02 sanlock[2561]: 2014-02-09 17:35:34-0600 261741 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:34 vm02 sanlock[2561]: 2014-02-09 17:35:34-0600 261741 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:44 vm02 sanlock[2561]: 2014-02-09 17:35:44-0600 261751 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:44 vm02 sanlock[2561]: 2014-02-09 17:35:44-0600 261751 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:44 vm02 sanlock[2561]: 2014-02-09 17:35:44-0600 261751 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:44 vm02 sanlock[2561]: 2014-02-09 17:35:44-0600 261751 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:35:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:35:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:35:54 vm02 sanlock[2561]: 2014-02-09 17:35:54-0600 261761 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:35:54 vm02 sanlock[2561]: 2014-02-09 17:35:54-0600 261761 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:35:54 vm02 sanlock[2561]: 2014-02-09 17:35:54-0600 261761 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:35:54 vm02 sanlock[2561]: 2014-02-09 17:35:54-0600 261761 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:04 vm02 sanlock[2561]: 2014-02-09 17:36:04-0600 261771 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:04 vm02 sanlock[2561]: 2014-02-09 17:36:04-0600 261771 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:04 vm02 sanlock[2561]: 2014-02-09 17:36:04-0600 261771 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:04 vm02 sanlock[2561]: 2014-02-09 17:36:04-0600 261771 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:05 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:36:05 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:36:05 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:36:05 vm02 vdsm TaskManager.Task ERROR Task=`2dbf55bc-b776-4af5-bed0-b17b08e96813`::Unexpected error#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/task.py", line 857, in _run#012    return fn(*args, **kargs)#012  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012    res = f(*args, **kwargs)#012  File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool#012    masterVersion, options)#012  File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool#012    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 699, in connect#012    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild#012    masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain#012    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)#012StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=7c49750d-7eae-4cd2-9b63-1dc71f357b88'
Feb  9 17:36:14 vm02 sanlock[2561]: 2014-02-09 17:36:14-0600 261781 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:14 vm02 sanlock[2561]: 2014-02-09 17:36:14-0600 261781 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:14 vm02 sanlock[2561]: 2014-02-09 17:36:14-0600 261781 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:14 vm02 sanlock[2561]: 2014-02-09 17:36:14-0600 261781 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:36:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:36:24 vm02 sanlock[2561]: 2014-02-09 17:36:24-0600 261791 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:24 vm02 sanlock[2561]: 2014-02-09 17:36:24-0600 261791 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:24 vm02 sanlock[2561]: 2014-02-09 17:36:24-0600 261791 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:24 vm02 sanlock[2561]: 2014-02-09 17:36:24-0600 261791 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:34 vm02 sanlock[2561]: 2014-02-09 17:36:34-0600 261801 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:34 vm02 sanlock[2561]: 2014-02-09 17:36:34-0600 261801 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:34 vm02 sanlock[2561]: 2014-02-09 17:36:34-0600 261801 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:34 vm02 sanlock[2561]: 2014-02-09 17:36:34-0600 261801 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:44 vm02 sanlock[2561]: 2014-02-09 17:36:44-0600 261811 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:44 vm02 sanlock[2561]: 2014-02-09 17:36:44-0600 261811 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:44 vm02 sanlock[2561]: 2014-02-09 17:36:44-0600 261811 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:44 vm02 sanlock[2561]: 2014-02-09 17:36:44-0600 261811 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:36:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:36:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:36:54 vm02 sanlock[2561]: 2014-02-09 17:36:54-0600 261821 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:36:54 vm02 sanlock[2561]: 2014-02-09 17:36:54-0600 261821 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:36:54 vm02 sanlock[2561]: 2014-02-09 17:36:54-0600 261821 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:36:54 vm02 sanlock[2561]: 2014-02-09 17:36:54-0600 261821 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:04 vm02 sanlock[2561]: 2014-02-09 17:37:04-0600 261831 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:04 vm02 sanlock[2561]: 2014-02-09 17:37:04-0600 261831 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:04 vm02 sanlock[2561]: 2014-02-09 17:37:04-0600 261831 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:04 vm02 sanlock[2561]: 2014-02-09 17:37:04-0600 261831 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:14 vm02 sanlock[2561]: 2014-02-09 17:37:14-0600 261841 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:14 vm02 sanlock[2561]: 2014-02-09 17:37:14-0600 261841 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:14 vm02 sanlock[2561]: 2014-02-09 17:37:14-0600 261841 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:14 vm02 sanlock[2561]: 2014-02-09 17:37:14-0600 261841 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:37:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:37:24 vm02 sanlock[2561]: 2014-02-09 17:37:24-0600 261851 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:24 vm02 sanlock[2561]: 2014-02-09 17:37:24-0600 261851 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:24 vm02 sanlock[2561]: 2014-02-09 17:37:24-0600 261851 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:24 vm02 sanlock[2561]: 2014-02-09 17:37:24-0600 261851 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:34 vm02 sanlock[2561]: 2014-02-09 17:37:34-0600 261861 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:34 vm02 sanlock[2561]: 2014-02-09 17:37:34-0600 261861 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:34 vm02 sanlock[2561]: 2014-02-09 17:37:34-0600 261861 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:34 vm02 sanlock[2561]: 2014-02-09 17:37:34-0600 261861 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:44 vm02 sanlock[2561]: 2014-02-09 17:37:44-0600 261871 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:44 vm02 sanlock[2561]: 2014-02-09 17:37:44-0600 261871 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:44 vm02 sanlock[2561]: 2014-02-09 17:37:44-0600 261871 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:44 vm02 sanlock[2561]: 2014-02-09 17:37:44-0600 261871 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:37:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:37:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:37:54 vm02 sanlock[2561]: 2014-02-09 17:37:54-0600 261881 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:37:54 vm02 sanlock[2561]: 2014-02-09 17:37:54-0600 261881 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:37:54 vm02 sanlock[2561]: 2014-02-09 17:37:54-0600 261881 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:37:54 vm02 sanlock[2561]: 2014-02-09 17:37:54-0600 261881 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:04 vm02 sanlock[2561]: 2014-02-09 17:38:04-0600 261891 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:04 vm02 sanlock[2561]: 2014-02-09 17:38:04-0600 261891 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:04 vm02 sanlock[2561]: 2014-02-09 17:38:04-0600 261891 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:04 vm02 sanlock[2561]: 2014-02-09 17:38:04-0600 261891 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:14 vm02 sanlock[2561]: 2014-02-09 17:38:14-0600 261901 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:14 vm02 sanlock[2561]: 2014-02-09 17:38:14-0600 261901 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:14 vm02 sanlock[2561]: 2014-02-09 17:38:14-0600 261901 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:14 vm02 sanlock[2561]: 2014-02-09 17:38:14-0600 261901 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:38:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:38:24 vm02 sanlock[2561]: 2014-02-09 17:38:24-0600 261911 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:24 vm02 sanlock[2561]: 2014-02-09 17:38:24-0600 261911 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:24 vm02 sanlock[2561]: 2014-02-09 17:38:24-0600 261911 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:24 vm02 sanlock[2561]: 2014-02-09 17:38:24-0600 261911 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:34 vm02 sanlock[2561]: 2014-02-09 17:38:34-0600 261921 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:34 vm02 sanlock[2561]: 2014-02-09 17:38:34-0600 261921 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:34 vm02 sanlock[2561]: 2014-02-09 17:38:34-0600 261921 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:34 vm02 sanlock[2561]: 2014-02-09 17:38:34-0600 261921 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:44 vm02 sanlock[2561]: 2014-02-09 17:38:44-0600 261931 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:44 vm02 sanlock[2561]: 2014-02-09 17:38:44-0600 261931 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:44 vm02 sanlock[2561]: 2014-02-09 17:38:44-0600 261931 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:44 vm02 sanlock[2561]: 2014-02-09 17:38:44-0600 261931 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:38:45 vm02 vdsm vm.Vm WARNING vmId=`741f9811-db68-4dc4-a88a-7cb9be576e57`::Migration is stuck: Hasn't progressed in 300.045619965 seconds. Aborting.
Feb  9 17:38:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:38:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:38:54 vm02 sanlock[2561]: 2014-02-09 17:38:54-0600 261941 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:38:54 vm02 sanlock[2561]: 2014-02-09 17:38:54-0600 261941 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:38:54 vm02 sanlock[2561]: 2014-02-09 17:38:54-0600 261941 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:38:54 vm02 sanlock[2561]: 2014-02-09 17:38:54-0600 261941 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:04 vm02 sanlock[2561]: 2014-02-09 17:39:04-0600 261951 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:04 vm02 sanlock[2561]: 2014-02-09 17:39:04-0600 261951 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:04 vm02 sanlock[2561]: 2014-02-09 17:39:04-0600 261951 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:04 vm02 sanlock[2561]: 2014-02-09 17:39:04-0600 261951 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:14 vm02 sanlock[2561]: 2014-02-09 17:39:14-0600 261961 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:14 vm02 sanlock[2561]: 2014-02-09 17:39:14-0600 261961 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:14 vm02 sanlock[2561]: 2014-02-09 17:39:14-0600 261961 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:14 vm02 sanlock[2561]: 2014-02-09 17:39:14-0600 261961 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:39:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:39:24 vm02 sanlock[2561]: 2014-02-09 17:39:24-0600 261971 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:24 vm02 sanlock[2561]: 2014-02-09 17:39:24-0600 261971 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:24 vm02 sanlock[2561]: 2014-02-09 17:39:24-0600 261971 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:24 vm02 sanlock[2561]: 2014-02-09 17:39:24-0600 261971 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:34 vm02 sanlock[2561]: 2014-02-09 17:39:34-0600 261981 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:34 vm02 sanlock[2561]: 2014-02-09 17:39:34-0600 261981 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:34 vm02 sanlock[2561]: 2014-02-09 17:39:34-0600 261981 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:34 vm02 sanlock[2561]: 2014-02-09 17:39:34-0600 261981 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:44 vm02 sanlock[2561]: 2014-02-09 17:39:44-0600 261991 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:44 vm02 sanlock[2561]: 2014-02-09 17:39:44-0600 261991 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:44 vm02 sanlock[2561]: 2014-02-09 17:39:44-0600 261991 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:44 vm02 sanlock[2561]: 2014-02-09 17:39:44-0600 261991 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:39:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:39:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:39:54 vm02 sanlock[2561]: 2014-02-09 17:39:54-0600 262001 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:39:54 vm02 sanlock[2561]: 2014-02-09 17:39:54-0600 262001 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:39:54 vm02 sanlock[2561]: 2014-02-09 17:39:54-0600 262001 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:39:54 vm02 sanlock[2561]: 2014-02-09 17:39:54-0600 262001 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:00 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:40:03 vm02 vdsm root ERROR cannot read ib0 speed#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 297, in speed#012    s = int(speedFile.read())#012IOError: [Errno 22] Invalid argument
Feb  9 17:40:04 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/import_export: rdma,port=20049
Feb  9 17:40:04 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/iso: rdma,port=20049
Feb  9 17:40:04 vm02 vdsm root WARNING Using deprecated nfs_mount_options from vdsm.conf to mount 192.168.211.245:/tank/ovirt/data: rdma,port=20049
Feb  9 17:40:04 vm02 kernel: ata1: soft resetting link
Feb  9 17:40:04 vm02 kernel: ata2: EH complete
Feb  9 17:40:04 vm02 kernel: ata3: hard resetting link
Feb  9 17:40:04 vm02 kernel: ata3: nv: skipping hardreset on occupied port
Feb  9 17:40:04 vm02 kernel: ata4: hard resetting link
Feb  9 17:40:04 vm02 kernel: ata4: nv: skipping hardreset on occupied port
Feb  9 17:40:04 vm02 kernel: ata5: hard resetting link
Feb  9 17:40:04 vm02 kernel: ata6: hard resetting link
Feb  9 17:40:04 vm02 kernel: ata1: EH complete
Feb  9 17:40:04 vm02 kernel: ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:40:04 vm02 kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Feb  9 17:40:04 vm02 kernel: ata3.00: configured for UDMA/133
Feb  9 17:40:04 vm02 kernel: ata3: EH complete
Feb  9 17:40:04 vm02 kernel: ata4.00: configured for UDMA/133
Feb  9 17:40:04 vm02 kernel: ata4: EH complete
Feb  9 17:40:04 vm02 sanlock[2561]: 2014-02-09 17:40:04-0600 262011 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:04 vm02 sanlock[2561]: 2014-02-09 17:40:04-0600 262011 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:04 vm02 sanlock[2561]: 2014-02-09 17:40:04-0600 262011 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:04 vm02 sanlock[2561]: 2014-02-09 17:40:04-0600 262011 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:04 vm02 kernel: ata5: SATA link down (SStatus 0 SControl 300)
Feb  9 17:40:04 vm02 kernel: ata5: EH complete
Feb  9 17:40:04 vm02 kernel: ata6: SATA link down (SStatus 0 SControl 300)
Feb  9 17:40:04 vm02 kernel: ata6: EH complete
Feb  9 17:40:06 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:40:06 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:40:06 vm02 kernel: device-mapper: table: 253:2: multipath: error getting device
Feb  9 17:40:06 vm02 kernel: device-mapper: ioctl: error adding target to table
Feb  9 17:40:14 vm02 sanlock[2561]: 2014-02-09 17:40:14-0600 262021 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:14 vm02 sanlock[2561]: 2014-02-09 17:40:14-0600 262021 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:14 vm02 sanlock[2561]: 2014-02-09 17:40:14-0600 262021 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:14 vm02 sanlock[2561]: 2014-02-09 17:40:14-0600 262021 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:16 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:40:16 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:40:16 vm02 cpuspeed: Disabling performance cpu frequency scaling governor
Feb  9 17:40:16 vm02 cpuspeed: Enabling performance cpu frequency scaling governor
Feb  9 17:40:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:40:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:40:24 vm02 sanlock[2561]: 2014-02-09 17:40:24-0600 262031 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:24 vm02 sanlock[2561]: 2014-02-09 17:40:24-0600 262031 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:24 vm02 sanlock[2561]: 2014-02-09 17:40:24-0600 262031 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:24 vm02 sanlock[2561]: 2014-02-09 17:40:24-0600 262031 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:34 vm02 sanlock[2561]: 2014-02-09 17:40:34-0600 262041 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:34 vm02 sanlock[2561]: 2014-02-09 17:40:34-0600 262041 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:34 vm02 sanlock[2561]: 2014-02-09 17:40:34-0600 262041 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:34 vm02 sanlock[2561]: 2014-02-09 17:40:34-0600 262041 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:44 vm02 sanlock[2561]: 2014-02-09 17:40:44-0600 262051 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:44 vm02 sanlock[2561]: 2014-02-09 17:40:44-0600 262051 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:44 vm02 sanlock[2561]: 2014-02-09 17:40:44-0600 262051 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:44 vm02 sanlock[2561]: 2014-02-09 17:40:44-0600 262051 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:40:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:40:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:40:54 vm02 sanlock[2561]: 2014-02-09 17:40:54-0600 262061 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:40:54 vm02 sanlock[2561]: 2014-02-09 17:40:54-0600 262061 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:40:54 vm02 sanlock[2561]: 2014-02-09 17:40:54-0600 262061 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:40:54 vm02 sanlock[2561]: 2014-02-09 17:40:54-0600 262061 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:04 vm02 sanlock[2561]: 2014-02-09 17:41:04-0600 262071 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:04 vm02 sanlock[2561]: 2014-02-09 17:41:04-0600 262071 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:04 vm02 sanlock[2561]: 2014-02-09 17:41:04-0600 262071 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:04 vm02 sanlock[2561]: 2014-02-09 17:41:04-0600 262071 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:06 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_data timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:41:06 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_iso timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:41:06 vm02 vdsm scanDomains WARNING Metadata collection for domain path /rhev/data-center/mnt/192.168.211.245:_tank_ovirt_import__export timedout#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles#012    sd.DOMAIN_META_DATA))#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction#012    *args, **kwargs)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction#012    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)#012  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll#012    raise Timeout()#012Timeout
Feb  9 17:41:06 vm02 vdsm TaskManager.Task ERROR Task=`715c11f2-6953-4cb3-bbd0-a204e01f52c4`::Unexpected error#012Traceback (most recent call last):#012  File "/usr/share/vdsm/storage/task.py", line 857, in _run#012    return fn(*args, **kargs)#012  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012    res = f(*args, **kwargs)#012  File "/usr/share/vdsm/storage/hsm.py", line 1008, in connectStoragePool#012    masterVersion, options)#012  File "/usr/share/vdsm/storage/hsm.py", line 1062, in _connectStoragePool#012    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 699, in connect#012    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1244, in __rebuild#012    masterVersion=masterVersion)#012  File "/usr/share/vdsm/storage/sp.py", line 1603, in getMasterDomain#012    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)#012StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=7c49750d-7eae-4cd2-9b63-1dc71f357b88'
Feb  9 17:41:14 vm02 sanlock[2561]: 2014-02-09 17:41:14-0600 262081 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:14 vm02 sanlock[2561]: 2014-02-09 17:41:14-0600 262081 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:14 vm02 sanlock[2561]: 2014-02-09 17:41:14-0600 262081 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:14 vm02 sanlock[2561]: 2014-02-09 17:41:14-0600 262081 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:41:21 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:41:24 vm02 sanlock[2561]: 2014-02-09 17:41:24-0600 262091 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:24 vm02 sanlock[2561]: 2014-02-09 17:41:24-0600 262091 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:24 vm02 sanlock[2561]: 2014-02-09 17:41:24-0600 262091 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:24 vm02 sanlock[2561]: 2014-02-09 17:41:24-0600 262091 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:34 vm02 sanlock[2561]: 2014-02-09 17:41:34-0600 262101 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:34 vm02 sanlock[2561]: 2014-02-09 17:41:34-0600 262101 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:34 vm02 sanlock[2561]: 2014-02-09 17:41:34-0600 262101 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:34 vm02 sanlock[2561]: 2014-02-09 17:41:34-0600 262101 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:44 vm02 sanlock[2561]: 2014-02-09 17:41:44-0600 262111 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:44 vm02 sanlock[2561]: 2014-02-09 17:41:44-0600 262111 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:44 vm02 sanlock[2561]: 2014-02-09 17:41:44-0600 262111 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:44 vm02 sanlock[2561]: 2014-02-09 17:41:44-0600 262111 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:41:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb  9 17:41:51 vm02 kernel: rpcrdma: connection to 192.168.211.245:20049 closed (-103)
Feb  9 17:41:54 vm02 sanlock[2561]: 2014-02-09 17:41:54-0600 262121 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:41:54 vm02 sanlock[2561]: 2014-02-09 17:41:54-0600 262121 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:41:54 vm02 sanlock[2561]: 2014-02-09 17:41:54-0600 262121 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:41:54 vm02 sanlock[2561]: 2014-02-09 17:41:54-0600 262121 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
Feb  9 17:42:04 vm02 sanlock[2561]: 2014-02-09 17:42:04-0600 262131 [16497]: 7c49750d close_task_aio 0 0x7f18600008c0 busy
Feb  9 17:42:04 vm02 sanlock[2561]: 2014-02-09 17:42:04-0600 262131 [16497]: 7c49750d close_task_aio 1 0x7f1860000910 busy
Feb  9 17:42:04 vm02 sanlock[2561]: 2014-02-09 17:42:04-0600 262131 [16497]: 7c49750d close_task_aio 2 0x7f1860000960 busy
Feb  9 17:42:04 vm02 sanlock[2561]: 2014-02-09 17:42:04-0600 262131 [16497]: 7c49750d close_task_aio 3 0x7f18600009b0 busy
