MainThread::INFO::2014-11-28 15:18:04,437::vdsm::131::vds::(run) (PID: 12805) I am the actual vdsm 4.14.17-0.el7 cti11.ctisandbox.cfcc.edu (3.10.0-123.9.3.el7.x86_64)
MainThread::DEBUG::2014-11-28 15:18:04,438::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-11-28 15:18:04,438::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::DEBUG::2014-11-28 15:18:04,922::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-11-28 15:18:04,950::multipath::151::Storage.Multipath::(isEnabled) multipath Defaulting to False
MainThread::DEBUG::2014-11-28 15:18:04,951::misc::459::Storage.Misc::(rotateFiles) dir: /etc, prefixName: multipath.conf, versions: 5
MainThread::DEBUG::2014-11-28 15:18:04,951::misc::480::Storage.Misc::(rotateFiles) versions found: []
MainThread::DEBUG::2014-11-28 15:18:04,952::multipath::169::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/cp /tmp/tmp8YtEAg /etc/multipath.conf' (cwd None)
MainThread::DEBUG::2014-11-28 15:18:04,963::multipath::169::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:18:04,964::multipath::175::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /sbin/multipath -F' (cwd None)
MainThread::DEBUG::2014-11-28 15:18:04,975::multipath::175::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:18:04,975::multipath::178::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/vdsm-tool service-reload multipathd' (cwd None)
MainThread::DEBUG::2014-11-28 15:18:05,260::multipath::178::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:18:05,261::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-11-28 15:18:05,280::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:18:05,281::hsm::458::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-11-28 15:18:05,286::hsm::490::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-11-28 15:18:05,286::hsm::491::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-11-28 15:18:05,286::hsm::493::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-11-28 15:18:05,287::hsm::536::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-11-28 15:18:05,288::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
MainThread::INFO::2014-11-28 15:18:05,289::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-11-28 15:18:05,289::misc::740::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-11-28 15:18:05,321::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
MainThread::DEBUG::2014-11-28 15:18:05,322::task::595::TaskManager.Task::(_updateState) Task=`4658fe99-8352-479f-a796-2a22bab8ab33`::moving from state init -> state preparing
storageRefresh::DEBUG::2014-11-28 15:18:05,322::misc::740::SamplingMethod::(__call__) Got in to sampling method
MainThread::INFO::2014-11-28 15:18:05,322::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x19bc5f0>>)
storageRefresh::DEBUG::2014-11-28 15:18:05,322::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
MainThread::INFO::2014-11-28 15:18:05,322::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
storageRefresh::DEBUG::2014-11-28 15:18:05,323::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
MainThread::DEBUG::2014-11-28 15:18:05,323::task::1185::TaskManager.Task::(prepare) Task=`4658fe99-8352-479f-a796-2a22bab8ab33`::finished: None
MainThread::DEBUG::2014-11-28 15:18:05,323::task::595::TaskManager.Task::(_updateState) Task=`4658fe99-8352-479f-a796-2a22bab8ab33`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 15:18:05,323::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 15:18:05,324::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 15:18:05,327::task::990::TaskManager.Task::(_decref) Task=`4658fe99-8352-479f-a796-2a22bab8ab33`::ref 0 aborting False
MainThread::INFO::2014-11-28 15:18:05,328::momIF::49::MOM::(__init__) Starting up MOM
MainThread::INFO::2014-11-28 15:18:05,330::vmChannels::191::vds::(settimeout) Setting channels' timeout to 30 seconds.
clientIFinit::DEBUG::2014-11-28 15:18:05,352::libvirtconnection::145::root::(get) trying to connect libvirt
VM Channels Listener::INFO::2014-11-28 15:18:05,353::vmChannels::174::vds::(run) Starting VM channels listener thread.
storageRefresh::DEBUG::2014-11-28 15:18:05,353::misc::748::SamplingMethod::(__call__) Returning last result
MainThread::WARNING::2014-11-28 15:18:05,355::clientIF::180::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-11-28 15:18:05,356::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:18:05,421::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:18:05,424::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,427::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,429::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,431::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,434::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,437::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,438::misc::748::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-11-28 15:18:05,439::lvm::325::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:18:05,447::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [362856cb]
Thread-13::DEBUG::2014-11-28 15:18:05,449::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}}
storageRefresh::DEBUG::2014-11-28 15:18:05,452::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:18:05,476::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:18:05,477::lvm::350::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,478::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,480::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:18:05,502::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:18:05,504::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,507::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:18:05,527::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:18:05,529::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,532::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:18:05,552::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:18:05,553::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:18:05,554::hsm::393::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-11-28 15:18:05,973::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [362856cb]
Thread-13::DEBUG::2014-11-28 15:18:05,974::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:18:05,979::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [362856cb]
Thread-13::DEBUG::2014-11-28 15:18:06,015::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:06,056::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:18:06,092::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {}, 'bridges': {';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'em2', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '255.255.0.0', 'addr': '172.16.0.11', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'IPV6_PEERDNS': 'yes', 'IPV6INIT': 'yes', 'UUID': '63737da1-d4c1-4015-bf29-043f3105a768', 'IPADDR': '172.16.0.11', 'DNS1': '10.255.100.3', 'GATEWAY': '172.16.0.1', 'DEFROUTE': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': 'B8:AC:6F:91:12:D6', 'PREFIX': '16', 'BOOTPROTO': 'none', 'IPV6_DEFROUTE': 'yes', 'DNS2': '10.255.100.1', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'IPV6_PEERROUTES': 'yes', 'TYPE': 'Ethernet', 'ONBOOT': 'yes', 'NAME': 'em2'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': 'ea:57:dd:9b:4b:7d', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.977', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:18:06,180::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setupNetworks with ({'ovirtmgmt': {'nic': 'em2', 'ipaddr': '172.16.0.11', 'mtu': '9000', 'netmask': '255.255.0.0', 'STP': 'no', 'bridged': 'true', 'gateway': '172.16.0.1', 'defaultRoute': True}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [6c6641f2]
Thread-14::DEBUG::2014-11-28 15:18:06,184::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [6c6641f2]
Thread-14::DEBUG::2014-11-28 15:18:06,184::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:18:12,978::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [6c6641f2]
Thread-14::DEBUG::2014-11-28 15:18:12,978::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:18:12,984::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [6c6641f2]
Thread-14::DEBUG::2014-11-28 15:18:12,984::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:18:13,488::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [6c6641f2]
Thread-14::DEBUG::2014-11-28 15:18:13,488::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:18:13,577::BindingXMLRPC::1074::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:18:13,992::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [6c6641f2]
Thread-13::DEBUG::2014-11-28 15:18:13,993::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:13,998::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:18:14,017::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:12:d6', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'True', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': 'ea:57:dd:9b:4b:7d', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.977', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:18:14,102::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setSafeNetworkConfig with () {} flowID [484a1936]
Thread-13::DEBUG::2014-11-28 15:18:14,111::BindingXMLRPC::1074::vds::(wrapper) return setSafeNetworkConfig with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:18:16,932::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [7dd4b541]
Thread-13::DEBUG::2014-11-28 15:18:16,932::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:16,938::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:18:16,958::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:12:d6', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': 'ea:57:dd:9b:4b:7d', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.977', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:18:16,974::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [7dd4b541]
Thread-13::DEBUG::2014-11-28 15:18:17,014::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0046-4710-8038-CAC04F325031', 'systemSerialNumber': 'JFG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 15:18:17,052::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [73fca071]
Thread-13::DEBUG::2014-11-28 15:18:17,053::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:17,058::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:18:17,077::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:12:d6', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': 'ea:57:dd:9b:4b:7d', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.977', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:18:17,091::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [73fca071]
Thread-13::DEBUG::2014-11-28 15:18:17,092::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0046-4710-8038-CAC04F325031', 'systemSerialNumber': 'JFG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 15:18:17,224::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [16ea4c6a]
Thread-13::DEBUG::2014-11-28 15:18:17,224::task::595::TaskManager.Task::(_updateState) Task=`22f3364a-137d-416e-b19c-a9fd2cbdef26`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:18:17,225::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:18:17,230::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::ERROR::2014-11-28 15:18:17,321::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::ERROR::2014-11-28 15:18:17,322::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::DEBUG::2014-11-28 15:18:17,323::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {}
Thread-13::DEBUG::2014-11-28 15:18:17,328::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.2:/data/bs02store /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:17,539::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:18:17,541::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:18:17,541::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:18:17,547::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.99:/mnt/storage /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 15:18:17,577::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-13::DEBUG::2014-11-28 15:18:17,578::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 15:18:17,579::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:18:17,585::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-13::ERROR::2014-11-28 15:20:28,698::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 15:20:28,698::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 15:20:28,699::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:20:28,705::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 15:20:28,738::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-13::DEBUG::2014-11-28 15:20:28,739::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
Thread-13::DEBUG::2014-11-28 15:20:28,739::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:20:28,740::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:20:28,740::task::1185::TaskManager.Task::(prepare) Task=`22f3364a-137d-416e-b19c-a9fd2cbdef26`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:20:28,740::task::595::TaskManager.Task::(_updateState) Task=`22f3364a-137d-416e-b19c-a9fd2cbdef26`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:20:28,740::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:20:28,740::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:20:28,741::task::990::TaskManager.Task::(_decref) Task=`22f3364a-137d-416e-b19c-a9fd2cbdef26`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:20:28,843::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 15:20:28,843::task::595::TaskManager.Task::(_updateState) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:20:28,843::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24236, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 15:20:28,844::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`74b454d4-2034-4f0e-ad22-f56b88e23f51`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:20:28,844::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:20:28,844::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:20:28,845::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`74b454d4-2034-4f0e-ad22-f56b88e23f51`::Granted request
Thread-13::DEBUG::2014-11-28 15:20:28,845::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`d2f25f93-1d5a-49dd-9426-f78a66db0c94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:20:28,845::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:20:28,846::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:20:28,846::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`d2f25f93-1d5a-49dd-9426-f78a66db0c94`::Granted request
Thread-13::INFO::2014-11-28 15:20:28,846::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x7fc238091b00
Thread-13::INFO::2014-11-28 15:20:28,846::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24236)
Thread-13::DEBUG::2014-11-28 15:20:28,847::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,847::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,847::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,847::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,848::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,848::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,848::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 15:20:28,848::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:20:28,849::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 15:20:28,849::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:20:28,849::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 15:20:28,849::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 15:20:28,861::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:20:28,863::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 15:20:28,965::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:20:28,966::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,966::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,967::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,967::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,967::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,967::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:20:28,968::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:20:29,132::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-13::DEBUG::2014-11-28 15:20:29,279::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 15:20:29,282::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-13::DEBUG::2014-11-28 15:21:29,283::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 15:21:29,283::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 15:21:29,284::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:21:29,284::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 15:21:29,284::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-11-28 15:21:29,284::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 15:21:29,285::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:21:29,285::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::ERROR::2014-11-28 15:21:29,285::task::866::TaskManager.Task::(_setError) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:21:29,288::task::885::TaskManager.Task::(_run) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::Task._run: f43dcbf3-7446-482c-aa20-783d81044fde ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24236, None) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:21:29,288::task::1211::TaskManager.Task::(stop) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:21:29,288::task::990::TaskManager.Task::(_decref) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::ref 1 aborting True
Thread-13::DEBUG::2014-11-28 15:21:29,288::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-13::INFO::2014-11-28 15:21:29,289::task::1168::TaskManager.Task::(prepare) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::aborting: Task is aborted: u'' - code 100
Thread-13::DEBUG::2014-11-28 15:21:29,289::task::1173::TaskManager.Task::(prepare) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::Prepare: aborted: 
Thread-13::DEBUG::2014-11-28 15:21:29,289::task::990::TaskManager.Task::(_decref) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:21:29,289::task::925::TaskManager.Task::(_doAbort) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:21:29,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:21:29,290::task::595::TaskManager.Task::(_updateState) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:21:29,290::task::550::TaskManager.Task::(__state_aborting) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:21:29,290::task::595::TaskManager.Task::(_updateState) Task=`f43dcbf3-7446-482c-aa20-783d81044fde`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:21:29,290::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:21:29,290::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:21:29,290::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-13::DEBUG::2014-11-28 15:21:29,299::task::595::TaskManager.Task::(_updateState) Task=`02047806-cff3-4592-9208-1919d9b66026`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:21:29,299::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:21:29,299::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:21:29,299::task::1185::TaskManager.Task::(prepare) Task=`02047806-cff3-4592-9208-1919d9b66026`::finished: {}
Thread-13::DEBUG::2014-11-28 15:21:29,299::task::595::TaskManager.Task::(_updateState) Task=`02047806-cff3-4592-9208-1919d9b66026`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:21:29,299::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:21:29,300::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:21:29,300::task::990::TaskManager.Task::(_decref) Task=`02047806-cff3-4592-9208-1919d9b66026`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:21:29,314::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': False},) {} flowID [16ea4c6a]
Thread-13::DEBUG::2014-11-28 15:21:29,384::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:21:35,455::task::595::TaskManager.Task::(_updateState) Task=`96d9e872-ab12-42cb-933d-791af32b0fcf`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:21:35,456::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:21:35,456::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:21:35,456::task::1185::TaskManager.Task::(prepare) Task=`96d9e872-ab12-42cb-933d-791af32b0fcf`::finished: {}
Thread-13::DEBUG::2014-11-28 15:21:35,456::task::595::TaskManager.Task::(_updateState) Task=`96d9e872-ab12-42cb-933d-791af32b0fcf`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:21:35,457::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:21:35,457::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:21:35,457::task::990::TaskManager.Task::(_decref) Task=`96d9e872-ab12-42cb-933d-791af32b0fcf`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:21:50,571::task::595::TaskManager.Task::(_updateState) Task=`5ca119c2-5cd1-4ff9-8084-c615403b8b48`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:21:50,571::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:21:50,571::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:21:50,571::task::1185::TaskManager.Task::(prepare) Task=`5ca119c2-5cd1-4ff9-8084-c615403b8b48`::finished: {}
Thread-13::DEBUG::2014-11-28 15:21:50,571::task::595::TaskManager.Task::(_updateState) Task=`5ca119c2-5cd1-4ff9-8084-c615403b8b48`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:21:50,572::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:21:50,572::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:21:50,572::task::990::TaskManager.Task::(_decref) Task=`5ca119c2-5cd1-4ff9-8084-c615403b8b48`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:05,663::task::595::TaskManager.Task::(_updateState) Task=`6561fabe-2945-4280-95d9-ea9d7e42c9d7`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:05,663::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:22:05,663::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:22:05,664::task::1185::TaskManager.Task::(prepare) Task=`6561fabe-2945-4280-95d9-ea9d7e42c9d7`::finished: {}
Thread-13::DEBUG::2014-11-28 15:22:05,664::task::595::TaskManager.Task::(_updateState) Task=`6561fabe-2945-4280-95d9-ea9d7e42c9d7`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:05,664::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:05,664::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:05,664::task::990::TaskManager.Task::(_decref) Task=`6561fabe-2945-4280-95d9-ea9d7e42c9d7`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,091::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7c3ec0e1]
Thread-13::DEBUG::2014-11-28 15:22:11,091::task::595::TaskManager.Task::(_updateState) Task=`4d585e21-9204-46ab-8c0d-d70e6e45ca19`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,092::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,098::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-13::DEBUG::2014-11-28 15:22:11,099::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
Thread-13::DEBUG::2014-11-28 15:22:11,099::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:22:11,100::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,100::task::1185::TaskManager.Task::(prepare) Task=`4d585e21-9204-46ab-8c0d-d70e6e45ca19`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,100::task::595::TaskManager.Task::(_updateState) Task=`4d585e21-9204-46ab-8c0d-d70e6e45ca19`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:11,100::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:11,100::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:11,101::task::990::TaskManager.Task::(_decref) Task=`4d585e21-9204-46ab-8c0d-d70e6e45ca19`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,255::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [657b2c33]
Thread-13::DEBUG::2014-11-28 15:22:11,255::task::595::TaskManager.Task::(_updateState) Task=`52ab024b-859f-4372-aeb3-3a10b5d7dda0`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,256::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,262::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:22:11,263::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:22:11,263::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:22:11,263::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,263::task::1185::TaskManager.Task::(prepare) Task=`52ab024b-859f-4372-aeb3-3a10b5d7dda0`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,264::task::595::TaskManager.Task::(_updateState) Task=`52ab024b-859f-4372-aeb3-3a10b5d7dda0`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:11,264::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:11,264::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:11,264::task::990::TaskManager.Task::(_decref) Task=`52ab024b-859f-4372-aeb3-3a10b5d7dda0`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,488::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [45361b98]
Thread-13::DEBUG::2014-11-28 15:22:11,488::task::595::TaskManager.Task::(_updateState) Task=`4a26b6a9-195b-4c67-add3-cdd6e3ab0b98`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,489::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,495::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-18::DEBUG::2014-11-28 15:22:20,767::task::595::TaskManager.Task::(_updateState) Task=`65880224-2827-4381-b012-87248070c909`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:22:20,768::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:22:20,768::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:22:20,768::task::1185::TaskManager.Task::(prepare) Task=`65880224-2827-4381-b012-87248070c909`::finished: {}
Thread-18::DEBUG::2014-11-28 15:22:20,768::task::595::TaskManager.Task::(_updateState) Task=`65880224-2827-4381-b012-87248070c909`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:22:20,768::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:22:20,769::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:22:20,769::task::990::TaskManager.Task::(_decref) Task=`65880224-2827-4381-b012-87248070c909`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:22:35,881::task::595::TaskManager.Task::(_updateState) Task=`1e670d60-0446-4b85-8539-55981d50e8e4`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:22:35,882::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:22:35,882::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:22:35,882::task::1185::TaskManager.Task::(prepare) Task=`1e670d60-0446-4b85-8539-55981d50e8e4`::finished: {}
Thread-18::DEBUG::2014-11-28 15:22:35,882::task::595::TaskManager.Task::(_updateState) Task=`1e670d60-0446-4b85-8539-55981d50e8e4`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:22:35,883::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:22:35,883::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:22:35,883::task::990::TaskManager.Task::(_decref) Task=`1e670d60-0446-4b85-8539-55981d50e8e4`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:22:42,524::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-18::DEBUG::2014-11-28 15:22:42,525::task::595::TaskManager.Task::(_updateState) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:22:42,525::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-18::ERROR::2014-11-28 15:22:42,525::task::866::TaskManager.Task::(_setError) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-18::DEBUG::2014-11-28 15:22:42,526::task::885::TaskManager.Task::(_run) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::Task._run: 5ae19174-c5a9-4278-bb1f-53bbd7dc22a2 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-18::DEBUG::2014-11-28 15:22:42,526::task::1211::TaskManager.Task::(stop) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::stopping in state preparing (force False)
Thread-18::DEBUG::2014-11-28 15:22:42,526::task::990::TaskManager.Task::(_decref) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::ref 1 aborting True
Thread-18::INFO::2014-11-28 15:22:42,526::task::1168::TaskManager.Task::(prepare) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-18::DEBUG::2014-11-28 15:22:42,527::task::1173::TaskManager.Task::(prepare) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::Prepare: aborted: Unknown pool id, pool not connected
Thread-18::DEBUG::2014-11-28 15:22:42,527::task::990::TaskManager.Task::(_decref) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::ref 0 aborting True
Thread-18::DEBUG::2014-11-28 15:22:42,527::task::925::TaskManager.Task::(_doAbort) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::Task._doAbort: force False
Thread-18::DEBUG::2014-11-28 15:22:42,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:22:42,527::task::595::TaskManager.Task::(_updateState) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::moving from state preparing -> state aborting
Thread-18::DEBUG::2014-11-28 15:22:42,527::task::550::TaskManager.Task::(__state_aborting) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::_aborting: recover policy none
Thread-18::DEBUG::2014-11-28 15:22:42,528::task::595::TaskManager.Task::(_updateState) Task=`5ae19174-c5a9-4278-bb1f-53bbd7dc22a2`::moving from state aborting -> state failed
Thread-18::DEBUG::2014-11-28 15:22:42,528::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:22:42,528::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::ERROR::2014-11-28 15:22:42,528::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-18::DEBUG::2014-11-28 15:22:42,542::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-18::DEBUG::2014-11-28 15:22:42,543::task::595::TaskManager.Task::(_updateState) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:22:42,543::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24236, domainsMap=None, options=None)
Thread-18::DEBUG::2014-11-28 15:22:42,543::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`1fbd82cc-6dfb-4e82-a542-df1726881e99`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-18::DEBUG::2014-11-28 15:22:42,543::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-18::DEBUG::2014-11-28 15:22:42,544::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-18::DEBUG::2014-11-28 15:22:42,544::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`1fbd82cc-6dfb-4e82-a542-df1726881e99`::Granted request
Thread-18::DEBUG::2014-11-28 15:22:42,544::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`661609c6-1012-473d-8cf7-5670752bb6b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-18::DEBUG::2014-11-28 15:22:42,545::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-18::DEBUG::2014-11-28 15:22:42,545::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-18::DEBUG::2014-11-28 15:22:42,545::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`661609c6-1012-473d-8cf7-5670752bb6b1`::Granted request
Thread-18::INFO::2014-11-28 15:22:42,546::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x7fc23807c0e0
Thread-18::INFO::2014-11-28 15:22:42,546::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24236)
Thread-18::DEBUG::2014-11-28 15:22:42,547::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,547::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,547::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,548::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,548::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,548::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,548::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-18::DEBUG::2014-11-28 15:22:42,549::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-18::DEBUG::2014-11-28 15:22:42,549::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-18::DEBUG::2014-11-28 15:22:42,549::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-18::DEBUG::2014-11-28 15:22:42,549::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-18::DEBUG::2014-11-28 15:22:42,549::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-18::DEBUG::2014-11-28 15:22:42,562::misc::748::SamplingMethod::(__call__) Returning last result
Thread-18::DEBUG::2014-11-28 15:22:42,564::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-18::DEBUG::2014-11-28 15:22:42,588::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-18::DEBUG::2014-11-28 15:22:42,588::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,588::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,589::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,589::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,589::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,589::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-18::DEBUG::2014-11-28 15:22:42,590::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:23:11,495::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:23:11,496::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:23:11,496::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:23:11,496::task::1185::TaskManager.Task::(prepare) Task=`4a26b6a9-195b-4c67-add3-cdd6e3ab0b98`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:23:11,496::task::595::TaskManager.Task::(_updateState) Task=`4a26b6a9-195b-4c67-add3-cdd6e3ab0b98`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:11,496::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:11,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:11,497::task::990::TaskManager.Task::(_decref) Task=`4a26b6a9-195b-4c67-add3-cdd6e3ab0b98`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:17,530::task::595::TaskManager.Task::(_updateState) Task=`ccefd77e-64ad-4095-9312-9f2ead6a1ae5`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:23:17,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:23:17,530::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:23:17,530::task::1185::TaskManager.Task::(prepare) Task=`ccefd77e-64ad-4095-9312-9f2ead6a1ae5`::finished: {}
Thread-13::DEBUG::2014-11-28 15:23:17,530::task::595::TaskManager.Task::(_updateState) Task=`ccefd77e-64ad-4095-9312-9f2ead6a1ae5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:17,531::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:17,531::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:17,531::task::990::TaskManager.Task::(_decref) Task=`ccefd77e-64ad-4095-9312-9f2ead6a1ae5`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:32,622::task::595::TaskManager.Task::(_updateState) Task=`70fdc07e-fa65-4ab7-b363-4c4369125c8c`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:23:32,622::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:23:32,622::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:23:32,622::task::1185::TaskManager.Task::(prepare) Task=`70fdc07e-fa65-4ab7-b363-4c4369125c8c`::finished: {}
Thread-13::DEBUG::2014-11-28 15:23:32,623::task::595::TaskManager.Task::(_updateState) Task=`70fdc07e-fa65-4ab7-b363-4c4369125c8c`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:32,623::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:32,623::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:32,623::task::990::TaskManager.Task::(_decref) Task=`70fdc07e-fa65-4ab7-b363-4c4369125c8c`::ref 0 aborting False
Thread-20::WARNING::2014-11-28 15:23:42,593::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-21::WARNING::2014-11-28 15:23:42,593::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-18::ERROR::2014-11-28 15:23:42,598::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-18::DEBUG::2014-11-28 15:23:42,598::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-18::DEBUG::2014-11-28 15:23:42,598::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-18::DEBUG::2014-11-28 15:23:42,598::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-18::DEBUG::2014-11-28 15:23:42,598::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-18::DEBUG::2014-11-28 15:23:42,599::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-18::DEBUG::2014-11-28 15:23:42,599::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-18::DEBUG::2014-11-28 15:23:42,599::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-18::DEBUG::2014-11-28 15:23:42,599::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-18::ERROR::2014-11-28 15:23:42,599::task::866::TaskManager.Task::(_setError) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-18::DEBUG::2014-11-28 15:23:42,600::task::885::TaskManager.Task::(_run) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::Task._run: 0717c4a4-736c-45a0-94b9-1e28e2e865f1 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24236, None) {} failed - stopping task
Thread-18::DEBUG::2014-11-28 15:23:42,600::task::1211::TaskManager.Task::(stop) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::stopping in state preparing (force False)
Thread-18::DEBUG::2014-11-28 15:23:42,600::task::990::TaskManager.Task::(_decref) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::ref 1 aborting True
Thread-18::INFO::2014-11-28 15:23:42,600::task::1168::TaskManager.Task::(prepare) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-18::DEBUG::2014-11-28 15:23:42,601::task::1173::TaskManager.Task::(prepare) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::Prepare: aborted: Cannot find master domain
Thread-18::DEBUG::2014-11-28 15:23:42,601::task::990::TaskManager.Task::(_decref) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::ref 0 aborting True
Thread-18::DEBUG::2014-11-28 15:23:42,601::task::925::TaskManager.Task::(_doAbort) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::Task._doAbort: force False
Thread-18::DEBUG::2014-11-28 15:23:42,601::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:23:42,601::task::595::TaskManager.Task::(_updateState) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::moving from state preparing -> state aborting
Thread-18::DEBUG::2014-11-28 15:23:42,601::task::550::TaskManager.Task::(__state_aborting) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::_aborting: recover policy none
Thread-18::DEBUG::2014-11-28 15:23:42,602::task::595::TaskManager.Task::(_updateState) Task=`0717c4a4-736c-45a0-94b9-1e28e2e865f1`::moving from state aborting -> state failed
Thread-18::DEBUG::2014-11-28 15:23:42,602::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:23:42,602::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::ERROR::2014-11-28 15:23:42,602::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-18::DEBUG::2014-11-28 15:23:47,718::task::595::TaskManager.Task::(_updateState) Task=`96f76d03-60c2-456a-a8f0-7a7f7f4d9fc7`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:23:47,718::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:23:47,718::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:23:47,718::task::1185::TaskManager.Task::(prepare) Task=`96f76d03-60c2-456a-a8f0-7a7f7f4d9fc7`::finished: {}
Thread-18::DEBUG::2014-11-28 15:23:47,719::task::595::TaskManager.Task::(_updateState) Task=`96f76d03-60c2-456a-a8f0-7a7f7f4d9fc7`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:23:47,719::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:23:47,719::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:23:47,719::task::990::TaskManager.Task::(_decref) Task=`96f76d03-60c2-456a-a8f0-7a7f7f4d9fc7`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:23:52,633::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-18::DEBUG::2014-11-28 15:23:52,634::task::595::TaskManager.Task::(_updateState) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:23:52,634::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-18::ERROR::2014-11-28 15:23:52,634::task::866::TaskManager.Task::(_setError) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-18::DEBUG::2014-11-28 15:23:52,635::task::885::TaskManager.Task::(_run) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::Task._run: 231ce465-bea4-4e5e-ae9b-7d41887db772 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-18::DEBUG::2014-11-28 15:23:52,635::task::1211::TaskManager.Task::(stop) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::stopping in state preparing (force False)
Thread-18::DEBUG::2014-11-28 15:23:52,635::task::990::TaskManager.Task::(_decref) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::ref 1 aborting True
Thread-18::INFO::2014-11-28 15:23:52,635::task::1168::TaskManager.Task::(prepare) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-18::DEBUG::2014-11-28 15:23:52,636::task::1173::TaskManager.Task::(prepare) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::Prepare: aborted: Unknown pool id, pool not connected
Thread-18::DEBUG::2014-11-28 15:23:52,636::task::990::TaskManager.Task::(_decref) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::ref 0 aborting True
Thread-18::DEBUG::2014-11-28 15:23:52,636::task::925::TaskManager.Task::(_doAbort) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::Task._doAbort: force False
Thread-18::DEBUG::2014-11-28 15:23:52,636::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:23:52,636::task::595::TaskManager.Task::(_updateState) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::moving from state preparing -> state aborting
Thread-18::DEBUG::2014-11-28 15:23:52,636::task::550::TaskManager.Task::(__state_aborting) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::_aborting: recover policy none
Thread-18::DEBUG::2014-11-28 15:23:52,637::task::595::TaskManager.Task::(_updateState) Task=`231ce465-bea4-4e5e-ae9b-7d41887db772`::moving from state aborting -> state failed
Thread-18::DEBUG::2014-11-28 15:23:52,637::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:23:52,637::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::ERROR::2014-11-28 15:23:52,637::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-18::DEBUG::2014-11-28 15:23:52,640::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-18::DEBUG::2014-11-28 15:23:52,641::task::595::TaskManager.Task::(_updateState) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:23:52,641::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-18::ERROR::2014-11-28 15:23:52,641::task::866::TaskManager.Task::(_setError) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2124, in getAllTasksStatuses
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-18::DEBUG::2014-11-28 15:23:52,642::task::885::TaskManager.Task::(_run) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::Task._run: 487f54e9-c00d-481f-9129-beafd29f812d () {} failed - stopping task
Thread-18::DEBUG::2014-11-28 15:23:52,642::task::1211::TaskManager.Task::(stop) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::stopping in state preparing (force False)
Thread-18::DEBUG::2014-11-28 15:23:52,642::task::990::TaskManager.Task::(_decref) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::ref 1 aborting True
Thread-18::INFO::2014-11-28 15:23:52,642::task::1168::TaskManager.Task::(prepare) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::aborting: Task is aborted: 'Not SPM' - code 654
Thread-18::DEBUG::2014-11-28 15:23:52,642::task::1173::TaskManager.Task::(prepare) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::Prepare: aborted: Not SPM
Thread-18::DEBUG::2014-11-28 15:23:52,642::task::990::TaskManager.Task::(_decref) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::ref 0 aborting True
Thread-18::DEBUG::2014-11-28 15:23:52,643::task::925::TaskManager.Task::(_doAbort) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::Task._doAbort: force False
Thread-18::DEBUG::2014-11-28 15:23:52,643::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:23:52,643::task::595::TaskManager.Task::(_updateState) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::moving from state preparing -> state aborting
Thread-18::DEBUG::2014-11-28 15:23:52,643::task::550::TaskManager.Task::(__state_aborting) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::_aborting: recover policy none
Thread-18::DEBUG::2014-11-28 15:23:52,643::task::595::TaskManager.Task::(_updateState) Task=`487f54e9-c00d-481f-9129-beafd29f812d`::moving from state aborting -> state failed
Thread-18::DEBUG::2014-11-28 15:23:52,644::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:23:52,644::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::ERROR::2014-11-28 15:23:52,644::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-18::DEBUG::2014-11-28 15:24:02,806::task::595::TaskManager.Task::(_updateState) Task=`f4de4c1e-ba69-4053-8c4f-2c6b2867aac7`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:02,806::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:24:02,806::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:24:02,807::task::1185::TaskManager.Task::(prepare) Task=`f4de4c1e-ba69-4053-8c4f-2c6b2867aac7`::finished: {}
Thread-18::DEBUG::2014-11-28 15:24:02,807::task::595::TaskManager.Task::(_updateState) Task=`f4de4c1e-ba69-4053-8c4f-2c6b2867aac7`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:24:02,807::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:24:02,807::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:24:02,807::task::990::TaskManager.Task::(_decref) Task=`f4de4c1e-ba69-4053-8c4f-2c6b2867aac7`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:24:17,895::task::595::TaskManager.Task::(_updateState) Task=`415fa4ae-f4c8-4922-8d96-591d93d68dbc`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:17,896::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:24:17,896::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:24:17,896::task::1185::TaskManager.Task::(prepare) Task=`415fa4ae-f4c8-4922-8d96-591d93d68dbc`::finished: {}
Thread-18::DEBUG::2014-11-28 15:24:17,896::task::595::TaskManager.Task::(_updateState) Task=`415fa4ae-f4c8-4922-8d96-591d93d68dbc`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:24:17,896::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:24:17,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:24:17,897::task::990::TaskManager.Task::(_decref) Task=`415fa4ae-f4c8-4922-8d96-591d93d68dbc`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:24:33,033::task::595::TaskManager.Task::(_updateState) Task=`25eb6f34-f0f0-4f8a-bfcb-941020be4708`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:33,033::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:24:33,033::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:24:33,033::task::1185::TaskManager.Task::(prepare) Task=`25eb6f34-f0f0-4f8a-bfcb-941020be4708`::finished: {}
Thread-18::DEBUG::2014-11-28 15:24:33,033::task::595::TaskManager.Task::(_updateState) Task=`25eb6f34-f0f0-4f8a-bfcb-941020be4708`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:24:33,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:24:33,034::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:24:33,034::task::990::TaskManager.Task::(_decref) Task=`25eb6f34-f0f0-4f8a-bfcb-941020be4708`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:24:48,123::task::595::TaskManager.Task::(_updateState) Task=`58de8bad-191a-4e1d-b161-147e68bb1028`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:48,123::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:24:48,123::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:24:48,124::task::1185::TaskManager.Task::(prepare) Task=`58de8bad-191a-4e1d-b161-147e68bb1028`::finished: {}
Thread-18::DEBUG::2014-11-28 15:24:48,124::task::595::TaskManager.Task::(_updateState) Task=`58de8bad-191a-4e1d-b161-147e68bb1028`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:24:48,124::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:24:48,124::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:24:48,124::task::990::TaskManager.Task::(_decref) Task=`58de8bad-191a-4e1d-b161-147e68bb1028`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:24:59,687::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5b643901]
Thread-18::DEBUG::2014-11-28 15:24:59,688::task::595::TaskManager.Task::(_updateState) Task=`09c1d2e3-bdad-463b-b408-c12dca555260`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:59,688::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-18::DEBUG::2014-11-28 15:24:59,695::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-18::ERROR::2014-11-28 15:24:59,714::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-18::ERROR::2014-11-28 15:24:59,715::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-18::DEBUG::2014-11-28 15:24:59,715::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-18::INFO::2014-11-28 15:24:59,715::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-18::DEBUG::2014-11-28 15:24:59,715::task::1185::TaskManager.Task::(prepare) Task=`09c1d2e3-bdad-463b-b408-c12dca555260`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-18::DEBUG::2014-11-28 15:24:59,716::task::595::TaskManager.Task::(_updateState) Task=`09c1d2e3-bdad-463b-b408-c12dca555260`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:24:59,716::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:24:59,716::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:24:59,716::task::990::TaskManager.Task::(_decref) Task=`09c1d2e3-bdad-463b-b408-c12dca555260`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:24:59,871::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [28e65d04]
Thread-18::DEBUG::2014-11-28 15:24:59,871::task::595::TaskManager.Task::(_updateState) Task=`7dd7724d-5cec-4978-aee8-33e863550f88`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:24:59,871::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-18::DEBUG::2014-11-28 15:24:59,895::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-22::DEBUG::2014-11-28 15:25:03,271::task::595::TaskManager.Task::(_updateState) Task=`ab06009f-2d7f-4aec-a182-866a5504893b`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:03,272::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:25:03,272::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:25:03,272::task::1185::TaskManager.Task::(prepare) Task=`ab06009f-2d7f-4aec-a182-866a5504893b`::finished: {}
Thread-22::DEBUG::2014-11-28 15:25:03,272::task::595::TaskManager.Task::(_updateState) Task=`ab06009f-2d7f-4aec-a182-866a5504893b`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:25:03,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:25:03,272::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:25:03,273::task::990::TaskManager.Task::(_decref) Task=`ab06009f-2d7f-4aec-a182-866a5504893b`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:25:18,363::task::595::TaskManager.Task::(_updateState) Task=`bb6b303e-b4d9-41dd-86e9-265b2cf70df2`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:18,363::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:25:18,364::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:25:18,364::task::1185::TaskManager.Task::(prepare) Task=`bb6b303e-b4d9-41dd-86e9-265b2cf70df2`::finished: {}
Thread-22::DEBUG::2014-11-28 15:25:18,364::task::595::TaskManager.Task::(_updateState) Task=`bb6b303e-b4d9-41dd-86e9-265b2cf70df2`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:25:18,364::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:25:18,364::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:25:18,365::task::990::TaskManager.Task::(_decref) Task=`bb6b303e-b4d9-41dd-86e9-265b2cf70df2`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:25:33,455::task::595::TaskManager.Task::(_updateState) Task=`6ea1dbb7-2a00-4523-a4bf-06e79709cf5b`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:33,456::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:25:33,456::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:25:33,456::task::1185::TaskManager.Task::(prepare) Task=`6ea1dbb7-2a00-4523-a4bf-06e79709cf5b`::finished: {}
Thread-22::DEBUG::2014-11-28 15:25:33,456::task::595::TaskManager.Task::(_updateState) Task=`6ea1dbb7-2a00-4523-a4bf-06e79709cf5b`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:25:33,456::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:25:33,457::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:25:33,457::task::990::TaskManager.Task::(_decref) Task=`6ea1dbb7-2a00-4523-a4bf-06e79709cf5b`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:25:48,546::task::595::TaskManager.Task::(_updateState) Task=`2144118f-75d9-46db-aec4-e887851814f6`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:48,546::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:25:48,546::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:25:48,546::task::1185::TaskManager.Task::(prepare) Task=`2144118f-75d9-46db-aec4-e887851814f6`::finished: {}
Thread-22::DEBUG::2014-11-28 15:25:48,546::task::595::TaskManager.Task::(_updateState) Task=`2144118f-75d9-46db-aec4-e887851814f6`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:25:48,547::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:25:48,547::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:25:48,547::task::990::TaskManager.Task::(_decref) Task=`2144118f-75d9-46db-aec4-e887851814f6`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:25:53,170::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-22::DEBUG::2014-11-28 15:25:53,170::task::595::TaskManager.Task::(_updateState) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:53,171::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-22::ERROR::2014-11-28 15:25:53,171::task::866::TaskManager.Task::(_setError) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-22::DEBUG::2014-11-28 15:25:53,171::task::885::TaskManager.Task::(_run) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::Task._run: 490b9964-c410-47f3-8da8-b9b470a4a75c ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 15:25:53,171::task::1211::TaskManager.Task::(stop) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 15:25:53,172::task::990::TaskManager.Task::(_decref) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::ref 1 aborting True
Thread-22::INFO::2014-11-28 15:25:53,172::task::1168::TaskManager.Task::(prepare) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-22::DEBUG::2014-11-28 15:25:53,172::task::1173::TaskManager.Task::(prepare) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::Prepare: aborted: Unknown pool id, pool not connected
Thread-22::DEBUG::2014-11-28 15:25:53,172::task::990::TaskManager.Task::(_decref) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::ref 0 aborting True
Thread-22::DEBUG::2014-11-28 15:25:53,172::task::925::TaskManager.Task::(_doAbort) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::Task._doAbort: force False
Thread-22::DEBUG::2014-11-28 15:25:53,173::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:25:53,173::task::595::TaskManager.Task::(_updateState) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::moving from state preparing -> state aborting
Thread-22::DEBUG::2014-11-28 15:25:53,173::task::550::TaskManager.Task::(__state_aborting) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::_aborting: recover policy none
Thread-22::DEBUG::2014-11-28 15:25:53,173::task::595::TaskManager.Task::(_updateState) Task=`490b9964-c410-47f3-8da8-b9b470a4a75c`::moving from state aborting -> state failed
Thread-22::DEBUG::2014-11-28 15:25:53,173::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:25:53,174::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::ERROR::2014-11-28 15:25:53,174::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-22::DEBUG::2014-11-28 15:25:53,186::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-22::DEBUG::2014-11-28 15:25:53,187::task::595::TaskManager.Task::(_updateState) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:25:53,187::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24237, domainsMap=None, options=None)
Thread-22::DEBUG::2014-11-28 15:25:53,188::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`c9b6bd05-a5d4-44be-bf13-3bdce01f3f79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-22::DEBUG::2014-11-28 15:25:53,188::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-22::DEBUG::2014-11-28 15:25:53,188::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-22::DEBUG::2014-11-28 15:25:53,188::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`c9b6bd05-a5d4-44be-bf13-3bdce01f3f79`::Granted request
Thread-22::DEBUG::2014-11-28 15:25:53,189::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`dd67984e-9eda-45c4-b053-82143ad973bb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-22::DEBUG::2014-11-28 15:25:53,189::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-22::DEBUG::2014-11-28 15:25:53,189::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-22::DEBUG::2014-11-28 15:25:53,189::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`dd67984e-9eda-45c4-b053-82143ad973bb`::Granted request
Thread-22::INFO::2014-11-28 15:25:53,190::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x7fc2380e3b90
Thread-22::INFO::2014-11-28 15:25:53,190::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24237)
Thread-22::DEBUG::2014-11-28 15:25:53,190::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,191::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,191::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,191::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,191::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,192::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,192::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-22::DEBUG::2014-11-28 15:25:53,192::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 15:25:53,192::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-22::DEBUG::2014-11-28 15:25:53,192::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 15:25:53,193::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-22::DEBUG::2014-11-28 15:25:53,193::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-22::DEBUG::2014-11-28 15:25:53,208::misc::748::SamplingMethod::(__call__) Returning last result
Thread-22::DEBUG::2014-11-28 15:25:53,211::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-22::DEBUG::2014-11-28 15:25:53,243::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-22::DEBUG::2014-11-28 15:25:53,244::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,244::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,244::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,244::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,245::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,245::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 15:25:53,245::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::WARNING::2014-11-28 15:26:53,258::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-25::WARNING::2014-11-28 15:26:53,259::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-25::DEBUG::2014-11-28 15:26:53,260::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-22::ERROR::2014-11-28 15:26:53,261::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-22::DEBUG::2014-11-28 15:26:53,261::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-22::DEBUG::2014-11-28 15:26:53,261::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-22::DEBUG::2014-11-28 15:26:53,262::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-22::DEBUG::2014-11-28 15:26:53,262::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-22::DEBUG::2014-11-28 15:26:53,262::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-22::DEBUG::2014-11-28 15:26:53,262::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-22::DEBUG::2014-11-28 15:26:53,262::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-22::DEBUG::2014-11-28 15:26:53,263::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-22::ERROR::2014-11-28 15:26:53,263::task::866::TaskManager.Task::(_setError) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-22::DEBUG::2014-11-28 15:26:53,263::task::885::TaskManager.Task::(_run) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::Task._run: 484f3165-64a0-4812-bbdc-577ce98e7450 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24237, None) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 15:26:53,264::task::1211::TaskManager.Task::(stop) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 15:26:53,264::task::990::TaskManager.Task::(_decref) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::ref 1 aborting True
Thread-22::INFO::2014-11-28 15:26:53,264::task::1168::TaskManager.Task::(prepare) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-22::DEBUG::2014-11-28 15:26:53,264::task::1173::TaskManager.Task::(prepare) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::Prepare: aborted: Cannot find master domain
Thread-22::DEBUG::2014-11-28 15:26:53,264::task::990::TaskManager.Task::(_decref) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::ref 0 aborting True
Thread-22::DEBUG::2014-11-28 15:26:53,265::task::925::TaskManager.Task::(_doAbort) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::Task._doAbort: force False
Thread-22::DEBUG::2014-11-28 15:26:53,265::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:26:53,265::task::595::TaskManager.Task::(_updateState) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::moving from state preparing -> state aborting
Thread-22::DEBUG::2014-11-28 15:26:53,265::task::550::TaskManager.Task::(__state_aborting) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::_aborting: recover policy none
Thread-22::DEBUG::2014-11-28 15:26:53,265::task::595::TaskManager.Task::(_updateState) Task=`484f3165-64a0-4812-bbdc-577ce98e7450`::moving from state aborting -> state failed
Thread-22::DEBUG::2014-11-28 15:26:53,265::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:26:53,266::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::ERROR::2014-11-28 15:26:53,266::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-22::DEBUG::2014-11-28 15:26:53,295::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-22::DEBUG::2014-11-28 15:26:53,295::task::595::TaskManager.Task::(_updateState) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:26:53,295::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-22::ERROR::2014-11-28 15:26:53,295::task::866::TaskManager.Task::(_setError) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-22::DEBUG::2014-11-28 15:26:53,296::task::885::TaskManager.Task::(_run) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::Task._run: 21296366-6300-4ef9-959b-02b564ea8f1c ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 15:26:53,296::task::1211::TaskManager.Task::(stop) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 15:26:53,296::task::990::TaskManager.Task::(_decref) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::ref 1 aborting True
Thread-22::INFO::2014-11-28 15:26:53,296::task::1168::TaskManager.Task::(prepare) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-22::DEBUG::2014-11-28 15:26:53,297::task::1173::TaskManager.Task::(prepare) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::Prepare: aborted: Unknown pool id, pool not connected
Thread-22::DEBUG::2014-11-28 15:26:53,297::task::990::TaskManager.Task::(_decref) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::ref 0 aborting True
Thread-22::DEBUG::2014-11-28 15:26:53,297::task::925::TaskManager.Task::(_doAbort) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::Task._doAbort: force False
Thread-22::DEBUG::2014-11-28 15:26:53,297::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:26:53,297::task::595::TaskManager.Task::(_updateState) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::moving from state preparing -> state aborting
Thread-22::DEBUG::2014-11-28 15:26:53,297::task::550::TaskManager.Task::(__state_aborting) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::_aborting: recover policy none
Thread-22::DEBUG::2014-11-28 15:26:53,298::task::595::TaskManager.Task::(_updateState) Task=`21296366-6300-4ef9-959b-02b564ea8f1c`::moving from state aborting -> state failed
Thread-22::DEBUG::2014-11-28 15:26:53,298::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:26:53,298::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::ERROR::2014-11-28 15:26:53,298::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-22::DEBUG::2014-11-28 15:27:02,310::task::595::TaskManager.Task::(_updateState) Task=`98f98ff4-4f92-4d27-94d8-9102423d57ad`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:27:02,310::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:27:02,310::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:27:02,311::task::1185::TaskManager.Task::(prepare) Task=`98f98ff4-4f92-4d27-94d8-9102423d57ad`::finished: {}
Thread-22::DEBUG::2014-11-28 15:27:02,311::task::595::TaskManager.Task::(_updateState) Task=`98f98ff4-4f92-4d27-94d8-9102423d57ad`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:27:02,311::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:27:02,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:27:02,311::task::990::TaskManager.Task::(_decref) Task=`98f98ff4-4f92-4d27-94d8-9102423d57ad`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 15:27:11,010::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-18::ERROR::2014-11-28 15:27:11,011::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-18::DEBUG::2014-11-28 15:27:11,011::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-18::INFO::2014-11-28 15:27:11,012::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-18::DEBUG::2014-11-28 15:27:11,012::task::1185::TaskManager.Task::(prepare) Task=`7dd7724d-5cec-4978-aee8-33e863550f88`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-18::DEBUG::2014-11-28 15:27:11,012::task::595::TaskManager.Task::(_updateState) Task=`7dd7724d-5cec-4978-aee8-33e863550f88`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:27:11,012::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:27:11,012::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:27:11,013::task::990::TaskManager.Task::(_decref) Task=`7dd7724d-5cec-4978-aee8-33e863550f88`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:27:11,163::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2bd26ce9]
Thread-18::DEBUG::2014-11-28 15:27:11,163::task::595::TaskManager.Task::(_updateState) Task=`6511c867-4a2d-47b0-8c5a-2eafd641604d`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:27:11,163::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-18::DEBUG::2014-11-28 15:27:11,169::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-18::DEBUG::2014-11-28 15:27:11,317::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-18::DEBUG::2014-11-28 15:27:11,318::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-18::INFO::2014-11-28 15:27:11,318::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-18::DEBUG::2014-11-28 15:27:11,318::task::1185::TaskManager.Task::(prepare) Task=`6511c867-4a2d-47b0-8c5a-2eafd641604d`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-18::DEBUG::2014-11-28 15:27:11,318::task::595::TaskManager.Task::(_updateState) Task=`6511c867-4a2d-47b0-8c5a-2eafd641604d`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:27:11,318::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:27:11,319::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:27:11,319::task::990::TaskManager.Task::(_decref) Task=`6511c867-4a2d-47b0-8c5a-2eafd641604d`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:27:11,402::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [479113eb]
Thread-22::DEBUG::2014-11-28 15:27:11,402::task::595::TaskManager.Task::(_updateState) Task=`130cf5e4-5f0d-46bd-9c42-bbf9c9000621`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:27:11,403::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:27:11,409::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-18::DEBUG::2014-11-28 15:27:17,425::task::595::TaskManager.Task::(_updateState) Task=`b7f6bf48-11a3-4648-8736-441a7adb630c`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:27:17,425::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:27:17,425::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:27:17,425::task::1185::TaskManager.Task::(prepare) Task=`b7f6bf48-11a3-4648-8736-441a7adb630c`::finished: {}
Thread-18::DEBUG::2014-11-28 15:27:17,426::task::595::TaskManager.Task::(_updateState) Task=`b7f6bf48-11a3-4648-8736-441a7adb630c`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:27:17,426::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:27:17,426::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:27:17,426::task::990::TaskManager.Task::(_decref) Task=`b7f6bf48-11a3-4648-8736-441a7adb630c`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:27:32,515::task::595::TaskManager.Task::(_updateState) Task=`ed456c43-fc9a-4ef7-9214-2b42ab283766`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:27:32,515::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:27:32,516::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:27:32,516::task::1185::TaskManager.Task::(prepare) Task=`ed456c43-fc9a-4ef7-9214-2b42ab283766`::finished: {}
Thread-18::DEBUG::2014-11-28 15:27:32,516::task::595::TaskManager.Task::(_updateState) Task=`ed456c43-fc9a-4ef7-9214-2b42ab283766`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:27:32,516::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:27:32,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:27:32,516::task::990::TaskManager.Task::(_decref) Task=`ed456c43-fc9a-4ef7-9214-2b42ab283766`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:27:47,604::task::595::TaskManager.Task::(_updateState) Task=`e86f343f-aaa4-462d-acbc-d5bc9a0d2374`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:27:47,606::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:27:47,606::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:27:47,607::task::1185::TaskManager.Task::(prepare) Task=`e86f343f-aaa4-462d-acbc-d5bc9a0d2374`::finished: {}
Thread-18::DEBUG::2014-11-28 15:27:47,608::task::595::TaskManager.Task::(_updateState) Task=`e86f343f-aaa4-462d-acbc-d5bc9a0d2374`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:27:47,609::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:27:47,610::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:27:47,611::task::990::TaskManager.Task::(_decref) Task=`e86f343f-aaa4-462d-acbc-d5bc9a0d2374`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:28:02,710::task::595::TaskManager.Task::(_updateState) Task=`d83b321a-7caf-4a76-9508-e41cebc40284`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:28:02,711::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:28:02,711::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:28:02,712::task::1185::TaskManager.Task::(prepare) Task=`d83b321a-7caf-4a76-9508-e41cebc40284`::finished: {}
Thread-18::DEBUG::2014-11-28 15:28:02,714::task::595::TaskManager.Task::(_updateState) Task=`d83b321a-7caf-4a76-9508-e41cebc40284`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:28:02,715::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:28:02,715::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:28:02,715::task::990::TaskManager.Task::(_decref) Task=`d83b321a-7caf-4a76-9508-e41cebc40284`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:28:11,409::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:28:11,410::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:28:11,410::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:28:11,410::task::1185::TaskManager.Task::(prepare) Task=`130cf5e4-5f0d-46bd-9c42-bbf9c9000621`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:28:11,410::task::595::TaskManager.Task::(_updateState) Task=`130cf5e4-5f0d-46bd-9c42-bbf9c9000621`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:28:11,411::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:28:11,411::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:28:11,411::task::990::TaskManager.Task::(_decref) Task=`130cf5e4-5f0d-46bd-9c42-bbf9c9000621`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:28:11,505::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5f466e14]
Thread-22::DEBUG::2014-11-28 15:28:11,506::task::595::TaskManager.Task::(_updateState) Task=`4f8de92e-3ade-47b7-893b-008898625db1`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:28:11,506::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:28:11,512::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-18::DEBUG::2014-11-28 15:28:17,808::task::595::TaskManager.Task::(_updateState) Task=`88abe07a-4ed0-4e37-8bf3-a72e63d84898`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:28:17,808::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:28:17,808::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:28:17,808::task::1185::TaskManager.Task::(prepare) Task=`88abe07a-4ed0-4e37-8bf3-a72e63d84898`::finished: {}
Thread-18::DEBUG::2014-11-28 15:28:17,809::task::595::TaskManager.Task::(_updateState) Task=`88abe07a-4ed0-4e37-8bf3-a72e63d84898`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:28:17,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:28:17,809::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:28:17,809::task::990::TaskManager.Task::(_decref) Task=`88abe07a-4ed0-4e37-8bf3-a72e63d84898`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:28:32,894::task::595::TaskManager.Task::(_updateState) Task=`e46411be-e74e-47bf-ba2a-b3220e6f027f`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:28:32,894::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:28:32,894::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:28:32,895::task::1185::TaskManager.Task::(prepare) Task=`e46411be-e74e-47bf-ba2a-b3220e6f027f`::finished: {}
Thread-18::DEBUG::2014-11-28 15:28:32,895::task::595::TaskManager.Task::(_updateState) Task=`e46411be-e74e-47bf-ba2a-b3220e6f027f`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:28:32,895::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:28:32,895::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:28:32,895::task::990::TaskManager.Task::(_decref) Task=`e46411be-e74e-47bf-ba2a-b3220e6f027f`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:28:47,982::task::595::TaskManager.Task::(_updateState) Task=`2a7837f1-25c6-4255-9c08-e0bcc6a757bb`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:28:47,982::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:28:47,982::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:28:47,983::task::1185::TaskManager.Task::(prepare) Task=`2a7837f1-25c6-4255-9c08-e0bcc6a757bb`::finished: {}
Thread-18::DEBUG::2014-11-28 15:28:47,983::task::595::TaskManager.Task::(_updateState) Task=`2a7837f1-25c6-4255-9c08-e0bcc6a757bb`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:28:47,983::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:28:47,983::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:28:47,983::task::990::TaskManager.Task::(_decref) Task=`2a7837f1-25c6-4255-9c08-e0bcc6a757bb`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:29:03,068::task::595::TaskManager.Task::(_updateState) Task=`8a59ffff-ed86-4edf-a3b9-d9f4e8dfc2e3`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:29:03,068::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:29:03,068::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:29:03,068::task::1185::TaskManager.Task::(prepare) Task=`8a59ffff-ed86-4edf-a3b9-d9f4e8dfc2e3`::finished: {}
Thread-18::DEBUG::2014-11-28 15:29:03,069::task::595::TaskManager.Task::(_updateState) Task=`8a59ffff-ed86-4edf-a3b9-d9f4e8dfc2e3`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:29:03,069::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:29:03,069::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:29:03,069::task::990::TaskManager.Task::(_decref) Task=`8a59ffff-ed86-4edf-a3b9-d9f4e8dfc2e3`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:11,517::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:29:11,517::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:29:11,517::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:29:11,518::task::1185::TaskManager.Task::(prepare) Task=`4f8de92e-3ade-47b7-893b-008898625db1`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:29:11,518::task::595::TaskManager.Task::(_updateState) Task=`4f8de92e-3ade-47b7-893b-008898625db1`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:29:11,518::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:29:11,518::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:29:11,518::task::990::TaskManager.Task::(_decref) Task=`4f8de92e-3ade-47b7-893b-008898625db1`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:18,154::task::595::TaskManager.Task::(_updateState) Task=`cffb9021-6c63-48e1-b324-348dd86baafe`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:29:18,154::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:29:18,154::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:29:18,154::task::1185::TaskManager.Task::(prepare) Task=`cffb9021-6c63-48e1-b324-348dd86baafe`::finished: {}
Thread-22::DEBUG::2014-11-28 15:29:18,154::task::595::TaskManager.Task::(_updateState) Task=`cffb9021-6c63-48e1-b324-348dd86baafe`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:29:18,155::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:29:18,155::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:29:18,155::task::990::TaskManager.Task::(_decref) Task=`cffb9021-6c63-48e1-b324-348dd86baafe`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:33,240::task::595::TaskManager.Task::(_updateState) Task=`439b3ff0-db70-4447-bff2-7ffb090f5fe5`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:29:33,240::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:29:33,240::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:29:33,241::task::1185::TaskManager.Task::(prepare) Task=`439b3ff0-db70-4447-bff2-7ffb090f5fe5`::finished: {}
Thread-22::DEBUG::2014-11-28 15:29:33,241::task::595::TaskManager.Task::(_updateState) Task=`439b3ff0-db70-4447-bff2-7ffb090f5fe5`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:29:33,241::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:29:33,241::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:29:33,241::task::990::TaskManager.Task::(_decref) Task=`439b3ff0-db70-4447-bff2-7ffb090f5fe5`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:48,325::task::595::TaskManager.Task::(_updateState) Task=`42a9d55a-e676-407a-b7cf-65bdde8de48c`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:29:48,325::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:29:48,325::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:29:48,326::task::1185::TaskManager.Task::(prepare) Task=`42a9d55a-e676-407a-b7cf-65bdde8de48c`::finished: {}
Thread-22::DEBUG::2014-11-28 15:29:48,326::task::595::TaskManager.Task::(_updateState) Task=`42a9d55a-e676-407a-b7cf-65bdde8de48c`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:29:48,326::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:29:48,326::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:29:48,326::task::990::TaskManager.Task::(_decref) Task=`42a9d55a-e676-407a-b7cf-65bdde8de48c`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:59,702::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [517c8fc8]
Thread-22::DEBUG::2014-11-28 15:29:59,703::task::595::TaskManager.Task::(_updateState) Task=`7269c851-d30b-42bd-93f9-0f97097a90df`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:29:59,703::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:29:59,709::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-22::ERROR::2014-11-28 15:29:59,732::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::ERROR::2014-11-28 15:29:59,733::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::DEBUG::2014-11-28 15:29:59,733::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:29:59,733::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:29:59,733::task::1185::TaskManager.Task::(prepare) Task=`7269c851-d30b-42bd-93f9-0f97097a90df`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:29:59,733::task::595::TaskManager.Task::(_updateState) Task=`7269c851-d30b-42bd-93f9-0f97097a90df`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:29:59,734::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:29:59,734::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:29:59,734::task::990::TaskManager.Task::(_decref) Task=`7269c851-d30b-42bd-93f9-0f97097a90df`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:29:59,895::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [40b32084]
Thread-22::DEBUG::2014-11-28 15:29:59,895::task::595::TaskManager.Task::(_updateState) Task=`f305b9fc-2df8-4d39-806d-069f80c70f56`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:29:59,896::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:29:59,911::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-18::DEBUG::2014-11-28 15:30:03,409::task::595::TaskManager.Task::(_updateState) Task=`d82a6831-a5d8-4f80-a371-00137839235a`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:30:03,409::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:30:03,409::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:30:03,410::task::1185::TaskManager.Task::(prepare) Task=`d82a6831-a5d8-4f80-a371-00137839235a`::finished: {}
Thread-18::DEBUG::2014-11-28 15:30:03,410::task::595::TaskManager.Task::(_updateState) Task=`d82a6831-a5d8-4f80-a371-00137839235a`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:30:03,410::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:30:03,410::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:30:03,410::task::990::TaskManager.Task::(_decref) Task=`d82a6831-a5d8-4f80-a371-00137839235a`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:30:18,493::task::595::TaskManager.Task::(_updateState) Task=`dcb3141e-27c4-4364-9ef3-bffec9da9979`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:30:18,494::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:30:18,494::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:30:18,494::task::1185::TaskManager.Task::(prepare) Task=`dcb3141e-27c4-4364-9ef3-bffec9da9979`::finished: {}
Thread-18::DEBUG::2014-11-28 15:30:18,494::task::595::TaskManager.Task::(_updateState) Task=`dcb3141e-27c4-4364-9ef3-bffec9da9979`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:30:18,495::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:30:18,495::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:30:18,495::task::990::TaskManager.Task::(_decref) Task=`dcb3141e-27c4-4364-9ef3-bffec9da9979`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:30:33,572::task::595::TaskManager.Task::(_updateState) Task=`fe3307b7-a945-4264-b38e-fbc5dab8ab7d`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:30:33,573::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:30:33,573::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:30:33,573::task::1185::TaskManager.Task::(prepare) Task=`fe3307b7-a945-4264-b38e-fbc5dab8ab7d`::finished: {}
Thread-18::DEBUG::2014-11-28 15:30:33,573::task::595::TaskManager.Task::(_updateState) Task=`fe3307b7-a945-4264-b38e-fbc5dab8ab7d`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:30:33,573::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:30:33,574::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:30:33,574::task::990::TaskManager.Task::(_decref) Task=`fe3307b7-a945-4264-b38e-fbc5dab8ab7d`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:30:48,658::task::595::TaskManager.Task::(_updateState) Task=`a3d9c817-a7d8-4e10-9839-2ecd773d1b77`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:30:48,658::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:30:48,659::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:30:48,659::task::1185::TaskManager.Task::(prepare) Task=`a3d9c817-a7d8-4e10-9839-2ecd773d1b77`::finished: {}
Thread-18::DEBUG::2014-11-28 15:30:48,659::task::595::TaskManager.Task::(_updateState) Task=`a3d9c817-a7d8-4e10-9839-2ecd773d1b77`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:30:48,659::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:30:48,659::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:30:48,660::task::990::TaskManager.Task::(_decref) Task=`a3d9c817-a7d8-4e10-9839-2ecd773d1b77`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:31:03,742::task::595::TaskManager.Task::(_updateState) Task=`f75c2b3f-1128-41a1-91ac-2aed43db11eb`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:31:03,742::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:31:03,742::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:31:03,742::task::1185::TaskManager.Task::(prepare) Task=`f75c2b3f-1128-41a1-91ac-2aed43db11eb`::finished: {}
Thread-18::DEBUG::2014-11-28 15:31:03,743::task::595::TaskManager.Task::(_updateState) Task=`f75c2b3f-1128-41a1-91ac-2aed43db11eb`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:31:03,743::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:31:03,743::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:31:03,743::task::990::TaskManager.Task::(_decref) Task=`f75c2b3f-1128-41a1-91ac-2aed43db11eb`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:31:18,827::task::595::TaskManager.Task::(_updateState) Task=`81502f48-544c-44a0-9305-c2964bfc60c1`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:31:18,827::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:31:18,828::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:31:18,828::task::1185::TaskManager.Task::(prepare) Task=`81502f48-544c-44a0-9305-c2964bfc60c1`::finished: {}
Thread-18::DEBUG::2014-11-28 15:31:18,828::task::595::TaskManager.Task::(_updateState) Task=`81502f48-544c-44a0-9305-c2964bfc60c1`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:31:18,828::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:31:18,828::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:31:18,829::task::990::TaskManager.Task::(_decref) Task=`81502f48-544c-44a0-9305-c2964bfc60c1`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:31:33,924::task::595::TaskManager.Task::(_updateState) Task=`4f8348cc-4423-41a0-b5a5-1f7939eeaff6`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:31:33,924::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:31:33,924::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:31:33,924::task::1185::TaskManager.Task::(prepare) Task=`4f8348cc-4423-41a0-b5a5-1f7939eeaff6`::finished: {}
Thread-18::DEBUG::2014-11-28 15:31:33,925::task::595::TaskManager.Task::(_updateState) Task=`4f8348cc-4423-41a0-b5a5-1f7939eeaff6`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:31:33,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:31:33,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:31:33,925::task::990::TaskManager.Task::(_decref) Task=`4f8348cc-4423-41a0-b5a5-1f7939eeaff6`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:31:49,007::task::595::TaskManager.Task::(_updateState) Task=`47742d83-6ba2-42bf-852a-0c28fadce3cf`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:31:49,007::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:31:49,007::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:31:49,007::task::1185::TaskManager.Task::(prepare) Task=`47742d83-6ba2-42bf-852a-0c28fadce3cf`::finished: {}
Thread-18::DEBUG::2014-11-28 15:31:49,008::task::595::TaskManager.Task::(_updateState) Task=`47742d83-6ba2-42bf-852a-0c28fadce3cf`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:31:49,008::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:31:49,008::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:31:49,008::task::990::TaskManager.Task::(_decref) Task=`47742d83-6ba2-42bf-852a-0c28fadce3cf`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:32:04,091::task::595::TaskManager.Task::(_updateState) Task=`0af8fb14-29d3-4617-b5b8-b8ca2561ed88`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:32:04,091::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:32:04,091::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:32:04,092::task::1185::TaskManager.Task::(prepare) Task=`0af8fb14-29d3-4617-b5b8-b8ca2561ed88`::finished: {}
Thread-18::DEBUG::2014-11-28 15:32:04,092::task::595::TaskManager.Task::(_updateState) Task=`0af8fb14-29d3-4617-b5b8-b8ca2561ed88`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:32:04,092::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:32:04,092::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:32:04,092::task::990::TaskManager.Task::(_decref) Task=`0af8fb14-29d3-4617-b5b8-b8ca2561ed88`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 15:32:11,024::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::ERROR::2014-11-28 15:32:11,025::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::DEBUG::2014-11-28 15:32:11,025::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:32:11,025::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:32:11,025::task::1185::TaskManager.Task::(prepare) Task=`f305b9fc-2df8-4d39-806d-069f80c70f56`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:32:11,026::task::595::TaskManager.Task::(_updateState) Task=`f305b9fc-2df8-4d39-806d-069f80c70f56`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:32:11,026::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:32:11,026::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:32:11,026::task::990::TaskManager.Task::(_decref) Task=`f305b9fc-2df8-4d39-806d-069f80c70f56`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:32:11,165::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [193bc8fa]
Thread-22::DEBUG::2014-11-28 15:32:11,166::task::595::TaskManager.Task::(_updateState) Task=`94b7e0a6-2cf5-4a80-b3fa-9d873171bde9`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:32:11,166::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:32:11,172::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-18::DEBUG::2014-11-28 15:32:19,177::task::595::TaskManager.Task::(_updateState) Task=`e213d5fe-f855-4e6f-9465-b947f182d079`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:32:19,177::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:32:19,178::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:32:19,178::task::1185::TaskManager.Task::(prepare) Task=`e213d5fe-f855-4e6f-9465-b947f182d079`::finished: {}
Thread-18::DEBUG::2014-11-28 15:32:19,178::task::595::TaskManager.Task::(_updateState) Task=`e213d5fe-f855-4e6f-9465-b947f182d079`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:32:19,178::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:32:19,178::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:32:19,179::task::990::TaskManager.Task::(_decref) Task=`e213d5fe-f855-4e6f-9465-b947f182d079`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:32:34,277::task::595::TaskManager.Task::(_updateState) Task=`e84b2eb1-db21-4c78-954a-7ab29ec4f5de`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:32:34,277::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:32:34,277::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:32:34,277::task::1185::TaskManager.Task::(prepare) Task=`e84b2eb1-db21-4c78-954a-7ab29ec4f5de`::finished: {}
Thread-18::DEBUG::2014-11-28 15:32:34,277::task::595::TaskManager.Task::(_updateState) Task=`e84b2eb1-db21-4c78-954a-7ab29ec4f5de`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:32:34,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:32:34,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:32:34,278::task::990::TaskManager.Task::(_decref) Task=`e84b2eb1-db21-4c78-954a-7ab29ec4f5de`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:32:49,358::task::595::TaskManager.Task::(_updateState) Task=`52971a18-172d-47db-b29b-1b6becf0762d`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:32:49,358::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:32:49,359::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:32:49,359::task::1185::TaskManager.Task::(prepare) Task=`52971a18-172d-47db-b29b-1b6becf0762d`::finished: {}
Thread-18::DEBUG::2014-11-28 15:32:49,359::task::595::TaskManager.Task::(_updateState) Task=`52971a18-172d-47db-b29b-1b6becf0762d`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:32:49,359::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:32:49,359::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:32:49,359::task::990::TaskManager.Task::(_decref) Task=`52971a18-172d-47db-b29b-1b6becf0762d`::ref 0 aborting False
Thread-18::DEBUG::2014-11-28 15:33:04,442::task::595::TaskManager.Task::(_updateState) Task=`7f6854c3-f4a3-4ce3-9014-85a627260ced`::moving from state init -> state preparing
Thread-18::INFO::2014-11-28 15:33:04,443::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-18::INFO::2014-11-28 15:33:04,443::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-18::DEBUG::2014-11-28 15:33:04,443::task::1185::TaskManager.Task::(prepare) Task=`7f6854c3-f4a3-4ce3-9014-85a627260ced`::finished: {}
Thread-18::DEBUG::2014-11-28 15:33:04,443::task::595::TaskManager.Task::(_updateState) Task=`7f6854c3-f4a3-4ce3-9014-85a627260ced`::moving from state preparing -> state finished
Thread-18::DEBUG::2014-11-28 15:33:04,444::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-18::DEBUG::2014-11-28 15:33:04,444::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-18::DEBUG::2014-11-28 15:33:04,444::task::990::TaskManager.Task::(_decref) Task=`7f6854c3-f4a3-4ce3-9014-85a627260ced`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:33:11,177::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:33:11,178::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:33:11,178::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:33:11,178::task::1185::TaskManager.Task::(prepare) Task=`94b7e0a6-2cf5-4a80-b3fa-9d873171bde9`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:33:11,178::task::595::TaskManager.Task::(_updateState) Task=`94b7e0a6-2cf5-4a80-b3fa-9d873171bde9`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:33:11,178::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:33:11,179::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:33:11,179::task::990::TaskManager.Task::(_decref) Task=`94b7e0a6-2cf5-4a80-b3fa-9d873171bde9`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:33:11,261::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [4e3fe509]
Thread-22::DEBUG::2014-11-28 15:33:11,262::task::595::TaskManager.Task::(_updateState) Task=`b1a7592e-bd2a-493c-befe-71f87863bc8d`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:33:11,262::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:33:11,268::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-22::DEBUG::2014-11-28 15:33:11,416::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 15:33:11,416::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:33:11,417::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:33:11,417::task::1185::TaskManager.Task::(prepare) Task=`b1a7592e-bd2a-493c-befe-71f87863bc8d`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:33:11,417::task::595::TaskManager.Task::(_updateState) Task=`b1a7592e-bd2a-493c-befe-71f87863bc8d`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:33:11,417::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:33:11,417::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:33:11,418::task::990::TaskManager.Task::(_decref) Task=`b1a7592e-bd2a-493c-befe-71f87863bc8d`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:33:19,552::task::595::TaskManager.Task::(_updateState) Task=`585817c4-cedc-469e-9e51-b30cfe572197`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:33:19,552::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:33:19,552::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:33:19,552::task::1185::TaskManager.Task::(prepare) Task=`585817c4-cedc-469e-9e51-b30cfe572197`::finished: {}
Thread-22::DEBUG::2014-11-28 15:33:19,553::task::595::TaskManager.Task::(_updateState) Task=`585817c4-cedc-469e-9e51-b30cfe572197`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:33:19,553::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:33:19,553::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:33:19,553::task::990::TaskManager.Task::(_decref) Task=`585817c4-cedc-469e-9e51-b30cfe572197`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:33:34,638::task::595::TaskManager.Task::(_updateState) Task=`df48788a-7ff4-4a36-a08b-42290052ca3a`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:33:34,639::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:33:34,639::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:33:34,639::task::1185::TaskManager.Task::(prepare) Task=`df48788a-7ff4-4a36-a08b-42290052ca3a`::finished: {}
Thread-22::DEBUG::2014-11-28 15:33:34,639::task::595::TaskManager.Task::(_updateState) Task=`df48788a-7ff4-4a36-a08b-42290052ca3a`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:33:34,639::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:33:34,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:33:34,640::task::990::TaskManager.Task::(_decref) Task=`df48788a-7ff4-4a36-a08b-42290052ca3a`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:33:49,725::task::595::TaskManager.Task::(_updateState) Task=`4540dba2-f60c-4568-8096-4a101bdd80c8`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:33:49,725::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:33:49,725::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:33:49,725::task::1185::TaskManager.Task::(prepare) Task=`4540dba2-f60c-4568-8096-4a101bdd80c8`::finished: {}
Thread-22::DEBUG::2014-11-28 15:33:49,726::task::595::TaskManager.Task::(_updateState) Task=`4540dba2-f60c-4568-8096-4a101bdd80c8`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:33:49,726::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:33:49,726::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:33:49,726::task::990::TaskManager.Task::(_decref) Task=`4540dba2-f60c-4568-8096-4a101bdd80c8`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:04,802::task::595::TaskManager.Task::(_updateState) Task=`ee955a12-c4ad-4a23-b206-564b06b9c0d7`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:04,802::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:34:04,803::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:34:04,803::task::1185::TaskManager.Task::(prepare) Task=`ee955a12-c4ad-4a23-b206-564b06b9c0d7`::finished: {}
Thread-22::DEBUG::2014-11-28 15:34:04,803::task::595::TaskManager.Task::(_updateState) Task=`ee955a12-c4ad-4a23-b206-564b06b9c0d7`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:04,803::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:04,803::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:04,804::task::990::TaskManager.Task::(_decref) Task=`ee955a12-c4ad-4a23-b206-564b06b9c0d7`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:11,347::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5c415723]
Thread-22::DEBUG::2014-11-28 15:34:11,347::task::595::TaskManager.Task::(_updateState) Task=`8949b000-2964-42a0-8729-18c005d51057`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:11,348::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:34:11,354::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-22::DEBUG::2014-11-28 15:34:11,355::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-22::DEBUG::2014-11-28 15:34:11,356::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:34:11,356::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:34:11,356::task::1185::TaskManager.Task::(prepare) Task=`8949b000-2964-42a0-8729-18c005d51057`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:34:11,356::task::595::TaskManager.Task::(_updateState) Task=`8949b000-2964-42a0-8729-18c005d51057`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:11,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:11,357::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:11,357::task::990::TaskManager.Task::(_decref) Task=`8949b000-2964-42a0-8729-18c005d51057`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:19,883::task::595::TaskManager.Task::(_updateState) Task=`9f7e3e59-1296-4fab-b496-c766a12f1b33`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:19,884::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:34:19,884::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:34:19,884::task::1185::TaskManager.Task::(prepare) Task=`9f7e3e59-1296-4fab-b496-c766a12f1b33`::finished: {}
Thread-22::DEBUG::2014-11-28 15:34:19,884::task::595::TaskManager.Task::(_updateState) Task=`9f7e3e59-1296-4fab-b496-c766a12f1b33`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:19,884::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:19,885::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:19,885::task::990::TaskManager.Task::(_decref) Task=`9f7e3e59-1296-4fab-b496-c766a12f1b33`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:34,966::task::595::TaskManager.Task::(_updateState) Task=`950f3241-1bde-4231-ab7c-516327da9032`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:34,966::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:34:34,966::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:34:34,967::task::1185::TaskManager.Task::(prepare) Task=`950f3241-1bde-4231-ab7c-516327da9032`::finished: {}
Thread-22::DEBUG::2014-11-28 15:34:34,967::task::595::TaskManager.Task::(_updateState) Task=`950f3241-1bde-4231-ab7c-516327da9032`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:34,967::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:34,967::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:34,967::task::990::TaskManager.Task::(_decref) Task=`950f3241-1bde-4231-ab7c-516327da9032`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:50,047::task::595::TaskManager.Task::(_updateState) Task=`d9ea2137-cde1-47d0-83a9-56a24cd49fe8`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:50,047::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:34:50,047::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:34:50,047::task::1185::TaskManager.Task::(prepare) Task=`d9ea2137-cde1-47d0-83a9-56a24cd49fe8`::finished: {}
Thread-22::DEBUG::2014-11-28 15:34:50,047::task::595::TaskManager.Task::(_updateState) Task=`d9ea2137-cde1-47d0-83a9-56a24cd49fe8`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:50,048::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:50,048::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:50,048::task::990::TaskManager.Task::(_decref) Task=`d9ea2137-cde1-47d0-83a9-56a24cd49fe8`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:34:59,689::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [4de47b16]
Thread-22::DEBUG::2014-11-28 15:34:59,689::task::595::TaskManager.Task::(_updateState) Task=`1344c823-78d5-406a-85fc-d5ee2409d21a`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:34:59,690::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:34:59,696::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-22::ERROR::2014-11-28 15:34:59,714::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::ERROR::2014-11-28 15:34:59,715::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::DEBUG::2014-11-28 15:34:59,715::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:34:59,715::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:34:59,715::task::1185::TaskManager.Task::(prepare) Task=`1344c823-78d5-406a-85fc-d5ee2409d21a`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:34:59,715::task::595::TaskManager.Task::(_updateState) Task=`1344c823-78d5-406a-85fc-d5ee2409d21a`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:34:59,716::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:34:59,716::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:34:59,716::task::990::TaskManager.Task::(_decref) Task=`1344c823-78d5-406a-85fc-d5ee2409d21a`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:35:05,127::task::595::TaskManager.Task::(_updateState) Task=`5ecbaf9d-9b2b-41d1-a165-b9410971c919`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:35:05,127::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:35:05,127::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:35:05,128::task::1185::TaskManager.Task::(prepare) Task=`5ecbaf9d-9b2b-41d1-a165-b9410971c919`::finished: {}
Thread-22::DEBUG::2014-11-28 15:35:05,128::task::595::TaskManager.Task::(_updateState) Task=`5ecbaf9d-9b2b-41d1-a165-b9410971c919`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:35:05,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:35:05,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:35:05,128::task::990::TaskManager.Task::(_decref) Task=`5ecbaf9d-9b2b-41d1-a165-b9410971c919`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:35:11,340::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [272df875]
Thread-22::DEBUG::2014-11-28 15:35:11,340::task::595::TaskManager.Task::(_updateState) Task=`0e1607ed-09dc-400c-98ef-f3793ad3037a`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:35:11,341::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:35:11,347::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-27::DEBUG::2014-11-28 15:35:20,272::task::595::TaskManager.Task::(_updateState) Task=`1efbe7ed-861a-4d32-b33d-534890124607`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:35:20,273::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:35:20,273::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:35:20,273::task::1185::TaskManager.Task::(prepare) Task=`1efbe7ed-861a-4d32-b33d-534890124607`::finished: {}
Thread-27::DEBUG::2014-11-28 15:35:20,273::task::595::TaskManager.Task::(_updateState) Task=`1efbe7ed-861a-4d32-b33d-534890124607`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:35:20,273::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:35:20,274::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:35:20,274::task::990::TaskManager.Task::(_decref) Task=`1efbe7ed-861a-4d32-b33d-534890124607`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:35:35,355::task::595::TaskManager.Task::(_updateState) Task=`6b7ac1ee-0bed-41d3-bf25-caea48c04fcc`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:35:35,355::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:35:35,356::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:35:35,356::task::1185::TaskManager.Task::(prepare) Task=`6b7ac1ee-0bed-41d3-bf25-caea48c04fcc`::finished: {}
Thread-27::DEBUG::2014-11-28 15:35:35,356::task::595::TaskManager.Task::(_updateState) Task=`6b7ac1ee-0bed-41d3-bf25-caea48c04fcc`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:35:35,356::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:35:35,356::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:35:35,357::task::990::TaskManager.Task::(_decref) Task=`6b7ac1ee-0bed-41d3-bf25-caea48c04fcc`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:35:50,439::task::595::TaskManager.Task::(_updateState) Task=`45bcec97-d91c-4855-80dc-39a773350680`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:35:50,439::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:35:50,439::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:35:50,439::task::1185::TaskManager.Task::(prepare) Task=`45bcec97-d91c-4855-80dc-39a773350680`::finished: {}
Thread-27::DEBUG::2014-11-28 15:35:50,440::task::595::TaskManager.Task::(_updateState) Task=`45bcec97-d91c-4855-80dc-39a773350680`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:35:50,440::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:35:50,440::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:35:50,440::task::990::TaskManager.Task::(_decref) Task=`45bcec97-d91c-4855-80dc-39a773350680`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:36:05,529::task::595::TaskManager.Task::(_updateState) Task=`15b9a250-a80e-4191-837c-f77060ae99b2`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:36:05,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:36:05,530::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:36:05,530::task::1185::TaskManager.Task::(prepare) Task=`15b9a250-a80e-4191-837c-f77060ae99b2`::finished: {}
Thread-27::DEBUG::2014-11-28 15:36:05,530::task::595::TaskManager.Task::(_updateState) Task=`15b9a250-a80e-4191-837c-f77060ae99b2`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:36:05,530::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:36:05,530::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:36:05,531::task::990::TaskManager.Task::(_decref) Task=`15b9a250-a80e-4191-837c-f77060ae99b2`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:36:20,646::task::595::TaskManager.Task::(_updateState) Task=`9f5d2b84-88a5-4bd0-bf43-d06de12fed57`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:36:20,646::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:36:20,646::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:36:20,646::task::1185::TaskManager.Task::(prepare) Task=`9f5d2b84-88a5-4bd0-bf43-d06de12fed57`::finished: {}
Thread-27::DEBUG::2014-11-28 15:36:20,646::task::595::TaskManager.Task::(_updateState) Task=`9f5d2b84-88a5-4bd0-bf43-d06de12fed57`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:36:20,647::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:36:20,647::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:36:20,647::task::990::TaskManager.Task::(_decref) Task=`9f5d2b84-88a5-4bd0-bf43-d06de12fed57`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:36:35,749::task::595::TaskManager.Task::(_updateState) Task=`1f14bb38-f0b6-4a3f-b909-ffa094d8893c`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:36:35,749::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:36:35,749::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:36:35,749::task::1185::TaskManager.Task::(prepare) Task=`1f14bb38-f0b6-4a3f-b909-ffa094d8893c`::finished: {}
Thread-27::DEBUG::2014-11-28 15:36:35,750::task::595::TaskManager.Task::(_updateState) Task=`1f14bb38-f0b6-4a3f-b909-ffa094d8893c`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:36:35,750::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:36:35,750::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:36:35,750::task::990::TaskManager.Task::(_decref) Task=`1f14bb38-f0b6-4a3f-b909-ffa094d8893c`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:36:50,843::task::595::TaskManager.Task::(_updateState) Task=`0e9f2a64-28e2-46fc-b2e4-ce9a78c98997`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:36:50,843::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:36:50,843::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:36:50,843::task::1185::TaskManager.Task::(prepare) Task=`0e9f2a64-28e2-46fc-b2e4-ce9a78c98997`::finished: {}
Thread-27::DEBUG::2014-11-28 15:36:50,844::task::595::TaskManager.Task::(_updateState) Task=`0e9f2a64-28e2-46fc-b2e4-ce9a78c98997`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:36:50,844::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:36:50,844::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:36:50,844::task::990::TaskManager.Task::(_decref) Task=`0e9f2a64-28e2-46fc-b2e4-ce9a78c98997`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:37:05,945::task::595::TaskManager.Task::(_updateState) Task=`1ef4a8db-df63-4543-999e-3f99348f6805`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:37:05,945::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:37:05,946::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:37:05,946::task::1185::TaskManager.Task::(prepare) Task=`1ef4a8db-df63-4543-999e-3f99348f6805`::finished: {}
Thread-27::DEBUG::2014-11-28 15:37:05,946::task::595::TaskManager.Task::(_updateState) Task=`1ef4a8db-df63-4543-999e-3f99348f6805`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:37:05,946::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:37:05,946::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:37:05,946::task::990::TaskManager.Task::(_decref) Task=`1ef4a8db-df63-4543-999e-3f99348f6805`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:37:21,043::task::595::TaskManager.Task::(_updateState) Task=`6f6f1e17-bd62-4677-b7ae-d85c728f86f3`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:37:21,044::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:37:21,044::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:37:21,044::task::1185::TaskManager.Task::(prepare) Task=`6f6f1e17-bd62-4677-b7ae-d85c728f86f3`::finished: {}
Thread-27::DEBUG::2014-11-28 15:37:21,044::task::595::TaskManager.Task::(_updateState) Task=`6f6f1e17-bd62-4677-b7ae-d85c728f86f3`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:37:21,044::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:37:21,044::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:37:21,045::task::990::TaskManager.Task::(_decref) Task=`6f6f1e17-bd62-4677-b7ae-d85c728f86f3`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 15:37:22,460::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::ERROR::2014-11-28 15:37:22,461::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::DEBUG::2014-11-28 15:37:22,461::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:37:22,461::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:37:22,461::task::1185::TaskManager.Task::(prepare) Task=`0e1607ed-09dc-400c-98ef-f3793ad3037a`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:37:22,461::task::595::TaskManager.Task::(_updateState) Task=`0e1607ed-09dc-400c-98ef-f3793ad3037a`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:37:22,462::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:37:22,462::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:37:22,462::task::990::TaskManager.Task::(_decref) Task=`0e1607ed-09dc-400c-98ef-f3793ad3037a`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:37:22,648::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [c403bcc]
Thread-22::DEBUG::2014-11-28 15:37:22,648::task::595::TaskManager.Task::(_updateState) Task=`2837f849-83ac-462a-a2c8-0644bf7c6938`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:37:22,648::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:37:22,654::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:37:36,135::task::595::TaskManager.Task::(_updateState) Task=`d5d62661-2832-43c7-a45f-0ecf54219983`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:37:36,135::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:37:36,135::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:37:36,135::task::1185::TaskManager.Task::(prepare) Task=`d5d62661-2832-43c7-a45f-0ecf54219983`::finished: {}
Thread-27::DEBUG::2014-11-28 15:37:36,136::task::595::TaskManager.Task::(_updateState) Task=`d5d62661-2832-43c7-a45f-0ecf54219983`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:37:36,136::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:37:36,136::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:37:36,136::task::990::TaskManager.Task::(_decref) Task=`d5d62661-2832-43c7-a45f-0ecf54219983`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:37:51,225::task::595::TaskManager.Task::(_updateState) Task=`d5da99dd-783c-41ec-bec1-5e577afd12c9`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:37:51,225::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:37:51,226::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:37:51,226::task::1185::TaskManager.Task::(prepare) Task=`d5da99dd-783c-41ec-bec1-5e577afd12c9`::finished: {}
Thread-27::DEBUG::2014-11-28 15:37:51,226::task::595::TaskManager.Task::(_updateState) Task=`d5da99dd-783c-41ec-bec1-5e577afd12c9`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:37:51,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:37:51,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:37:51,227::task::990::TaskManager.Task::(_decref) Task=`d5da99dd-783c-41ec-bec1-5e577afd12c9`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:38:06,314::task::595::TaskManager.Task::(_updateState) Task=`8d0c38dc-4faa-4b96-b781-b59394b47a16`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:38:06,315::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:38:06,315::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:38:06,315::task::1185::TaskManager.Task::(prepare) Task=`8d0c38dc-4faa-4b96-b781-b59394b47a16`::finished: {}
Thread-27::DEBUG::2014-11-28 15:38:06,315::task::595::TaskManager.Task::(_updateState) Task=`8d0c38dc-4faa-4b96-b781-b59394b47a16`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:38:06,315::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:38:06,316::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:38:06,316::task::990::TaskManager.Task::(_decref) Task=`8d0c38dc-4faa-4b96-b781-b59394b47a16`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:38:12,706::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:38:12,707::task::595::TaskManager.Task::(_updateState) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:38:12,707::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-27::ERROR::2014-11-28 15:38:12,707::task::866::TaskManager.Task::(_setError) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-27::DEBUG::2014-11-28 15:38:12,707::task::885::TaskManager.Task::(_run) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::Task._run: 09fd1df8-3606-41de-8b7f-f3aba3fa608a ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:38:12,708::task::1211::TaskManager.Task::(stop) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:38:12,708::task::990::TaskManager.Task::(_decref) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:38:12,708::task::1168::TaskManager.Task::(prepare) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-27::DEBUG::2014-11-28 15:38:12,708::task::1173::TaskManager.Task::(prepare) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::Prepare: aborted: Unknown pool id, pool not connected
Thread-27::DEBUG::2014-11-28 15:38:12,708::task::990::TaskManager.Task::(_decref) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:38:12,709::task::925::TaskManager.Task::(_doAbort) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:38:12,709::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:38:12,709::task::595::TaskManager.Task::(_updateState) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:38:12,709::task::550::TaskManager.Task::(__state_aborting) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:38:12,709::task::595::TaskManager.Task::(_updateState) Task=`09fd1df8-3606-41de-8b7f-f3aba3fa608a`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:38:12,710::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:38:12,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:38:12,710::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-27::DEBUG::2014-11-28 15:38:12,724::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:38:12,724::task::595::TaskManager.Task::(_updateState) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:38:12,725::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24238, domainsMap=None, options=None)
Thread-27::DEBUG::2014-11-28 15:38:12,725::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dd7c1feb-22e3-4ad0-b710-d246e818831b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:38:12,725::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:38:12,726::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:38:12,726::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`dd7c1feb-22e3-4ad0-b710-d246e818831b`::Granted request
Thread-27::DEBUG::2014-11-28 15:38:12,726::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`33fa8c29-5b37-41fe-b614-82835457e805`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:38:12,727::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:38:12,727::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:38:12,727::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`33fa8c29-5b37-41fe-b614-82835457e805`::Granted request
Thread-27::INFO::2014-11-28 15:38:12,728::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x1accb48
Thread-27::INFO::2014-11-28 15:38:12,728::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24238)
Thread-27::DEBUG::2014-11-28 15:38:12,728::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,728::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,728::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,729::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,729::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,729::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,729::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-27::DEBUG::2014-11-28 15:38:12,730::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:38:12,730::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-27::DEBUG::2014-11-28 15:38:12,730::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:38:12,730::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-27::DEBUG::2014-11-28 15:38:12,731::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-27::DEBUG::2014-11-28 15:38:12,742::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::DEBUG::2014-11-28 15:38:12,744::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-27::DEBUG::2014-11-28 15:38:12,778::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-27::DEBUG::2014-11-28 15:38:12,778::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,778::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,779::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,779::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,779::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,779::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:38:12,780::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::DEBUG::2014-11-28 15:38:12,938::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-27::DEBUG::2014-11-28 15:38:13,085::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-27::DEBUG::2014-11-28 15:38:13,088::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-22::DEBUG::2014-11-28 15:38:22,655::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:38:22,655::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:38:22,656::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:38:22,656::task::1185::TaskManager.Task::(prepare) Task=`2837f849-83ac-462a-a2c8-0644bf7c6938`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:38:22,656::task::595::TaskManager.Task::(_updateState) Task=`2837f849-83ac-462a-a2c8-0644bf7c6938`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:38:22,656::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:38:22,656::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:38:22,657::task::990::TaskManager.Task::(_decref) Task=`2837f849-83ac-462a-a2c8-0644bf7c6938`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:38:22,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [158fbf86]
Thread-22::DEBUG::2014-11-28 15:38:22,800::task::595::TaskManager.Task::(_updateState) Task=`fb590b83-4377-4fd9-96ad-c28136686c77`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:38:22,801::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:38:22,807::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-30::WARNING::2014-11-28 15:39:12,793::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-30::DEBUG::2014-11-28 15:39:12,796::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-27::DEBUG::2014-11-28 15:39:13,089::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-27::DEBUG::2014-11-28 15:39:13,089::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-27::DEBUG::2014-11-28 15:39:13,090::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:39:13,090::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-27::DEBUG::2014-11-28 15:39:13,090::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-27::DEBUG::2014-11-28 15:39:13,090::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-27::DEBUG::2014-11-28 15:39:13,091::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:39:13,091::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-27::ERROR::2014-11-28 15:39:13,091::task::866::TaskManager.Task::(_setError) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-27::DEBUG::2014-11-28 15:39:13,092::task::885::TaskManager.Task::(_run) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::Task._run: bd062034-12ba-4a3f-9508-0e2225da25b0 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24238, None) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:39:13,092::task::1211::TaskManager.Task::(stop) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:39:13,092::task::990::TaskManager.Task::(_decref) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::ref 1 aborting True
Thread-27::DEBUG::2014-11-28 15:39:13,092::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-27::INFO::2014-11-28 15:39:13,093::task::1168::TaskManager.Task::(prepare) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::aborting: Task is aborted: u'' - code 100
Thread-27::DEBUG::2014-11-28 15:39:13,093::task::1173::TaskManager.Task::(prepare) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::Prepare: aborted: 
Thread-27::DEBUG::2014-11-28 15:39:13,093::task::990::TaskManager.Task::(_decref) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:39:13,093::task::925::TaskManager.Task::(_doAbort) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:39:13,093::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:39:13,094::task::595::TaskManager.Task::(_updateState) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:39:13,094::task::550::TaskManager.Task::(__state_aborting) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:39:13,094::task::595::TaskManager.Task::(_updateState) Task=`bd062034-12ba-4a3f-9508-0e2225da25b0`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:39:13,094::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:39:13,094::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:39:13,094::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-27::DEBUG::2014-11-28 15:39:13,116::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:39:13,116::task::595::TaskManager.Task::(_updateState) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:39:13,116::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-27::ERROR::2014-11-28 15:39:13,117::task::866::TaskManager.Task::(_setError) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-27::DEBUG::2014-11-28 15:39:13,117::task::885::TaskManager.Task::(_run) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::Task._run: 4eefaba6-f28b-4c69-980a-51e1335ca22b ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:39:13,117::task::1211::TaskManager.Task::(stop) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:39:13,117::task::990::TaskManager.Task::(_decref) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:39:13,118::task::1168::TaskManager.Task::(prepare) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-27::DEBUG::2014-11-28 15:39:13,118::task::1173::TaskManager.Task::(prepare) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::Prepare: aborted: Unknown pool id, pool not connected
Thread-27::DEBUG::2014-11-28 15:39:13,118::task::990::TaskManager.Task::(_decref) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:39:13,118::task::925::TaskManager.Task::(_doAbort) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:39:13,118::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:39:13,118::task::595::TaskManager.Task::(_updateState) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:39:13,119::task::550::TaskManager.Task::(__state_aborting) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:39:13,119::task::595::TaskManager.Task::(_updateState) Task=`4eefaba6-f28b-4c69-980a-51e1335ca22b`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:39:13,119::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:39:13,119::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:39:13,119::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-27::DEBUG::2014-11-28 15:39:16,112::task::595::TaskManager.Task::(_updateState) Task=`f22e44f2-a015-4896-a722-4cfb21961fc2`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:39:16,112::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:39:16,112::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:39:16,113::task::1185::TaskManager.Task::(prepare) Task=`f22e44f2-a015-4896-a722-4cfb21961fc2`::finished: {}
Thread-27::DEBUG::2014-11-28 15:39:16,113::task::595::TaskManager.Task::(_updateState) Task=`f22e44f2-a015-4896-a722-4cfb21961fc2`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:39:16,113::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:39:16,113::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:39:16,113::task::990::TaskManager.Task::(_decref) Task=`f22e44f2-a015-4896-a722-4cfb21961fc2`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:22,811::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:39:22,812::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:39:22,812::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:39:22,812::task::1185::TaskManager.Task::(prepare) Task=`fb590b83-4377-4fd9-96ad-c28136686c77`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:39:22,813::task::595::TaskManager.Task::(_updateState) Task=`fb590b83-4377-4fd9-96ad-c28136686c77`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:39:22,813::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:39:22,813::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:39:22,813::task::990::TaskManager.Task::(_decref) Task=`fb590b83-4377-4fd9-96ad-c28136686c77`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:22,864::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [295d1612]
Thread-22::DEBUG::2014-11-28 15:39:22,865::task::595::TaskManager.Task::(_updateState) Task=`c4a719fd-3030-4b70-9e35-e8f8164d8600`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:39:22,865::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:39:22,871::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-22::DEBUG::2014-11-28 15:39:23,019::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 15:39:23,019::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:39:23,019::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:39:23,019::task::1185::TaskManager.Task::(prepare) Task=`c4a719fd-3030-4b70-9e35-e8f8164d8600`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:39:23,020::task::595::TaskManager.Task::(_updateState) Task=`c4a719fd-3030-4b70-9e35-e8f8164d8600`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:39:23,020::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:39:23,020::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:39:23,020::task::990::TaskManager.Task::(_decref) Task=`c4a719fd-3030-4b70-9e35-e8f8164d8600`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:31,199::task::595::TaskManager.Task::(_updateState) Task=`d6d8b520-79d5-4a09-82be-8c01a0cd7707`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:39:31,200::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:39:31,200::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:39:31,200::task::1185::TaskManager.Task::(prepare) Task=`d6d8b520-79d5-4a09-82be-8c01a0cd7707`::finished: {}
Thread-22::DEBUG::2014-11-28 15:39:31,200::task::595::TaskManager.Task::(_updateState) Task=`d6d8b520-79d5-4a09-82be-8c01a0cd7707`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:39:31,200::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:39:31,201::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:39:31,201::task::990::TaskManager.Task::(_decref) Task=`d6d8b520-79d5-4a09-82be-8c01a0cd7707`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:46,284::task::595::TaskManager.Task::(_updateState) Task=`de40528c-8f84-4e33-b5db-55f2046ef6e5`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:39:46,285::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:39:46,285::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:39:46,285::task::1185::TaskManager.Task::(prepare) Task=`de40528c-8f84-4e33-b5db-55f2046ef6e5`::finished: {}
Thread-22::DEBUG::2014-11-28 15:39:46,285::task::595::TaskManager.Task::(_updateState) Task=`de40528c-8f84-4e33-b5db-55f2046ef6e5`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:39:46,285::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:39:46,286::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:39:46,286::task::990::TaskManager.Task::(_decref) Task=`de40528c-8f84-4e33-b5db-55f2046ef6e5`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:59,696::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1b9dda4d]
Thread-22::DEBUG::2014-11-28 15:39:59,697::task::595::TaskManager.Task::(_updateState) Task=`49f45f64-af31-4916-907e-a0f35c258701`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:39:59,697::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:39:59,704::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-22::ERROR::2014-11-28 15:39:59,733::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::ERROR::2014-11-28 15:39:59,734::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::DEBUG::2014-11-28 15:39:59,734::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:39:59,734::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:39:59,734::task::1185::TaskManager.Task::(prepare) Task=`49f45f64-af31-4916-907e-a0f35c258701`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:39:59,735::task::595::TaskManager.Task::(_updateState) Task=`49f45f64-af31-4916-907e-a0f35c258701`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:39:59,735::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:39:59,735::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:39:59,735::task::990::TaskManager.Task::(_decref) Task=`49f45f64-af31-4916-907e-a0f35c258701`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:39:59,877::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [69278e4f]
Thread-22::DEBUG::2014-11-28 15:39:59,877::task::595::TaskManager.Task::(_updateState) Task=`83bc598d-4ed3-43e1-85ea-be3d404e6b5a`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:39:59,878::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:39:59,902::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-27::DEBUG::2014-11-28 15:40:01,370::task::595::TaskManager.Task::(_updateState) Task=`101ef85a-bbe6-46a9-ba99-d468afef8a73`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:40:01,370::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:40:01,370::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:40:01,371::task::1185::TaskManager.Task::(prepare) Task=`101ef85a-bbe6-46a9-ba99-d468afef8a73`::finished: {}
Thread-27::DEBUG::2014-11-28 15:40:01,371::task::595::TaskManager.Task::(_updateState) Task=`101ef85a-bbe6-46a9-ba99-d468afef8a73`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:40:01,371::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:40:01,371::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:40:01,371::task::990::TaskManager.Task::(_decref) Task=`101ef85a-bbe6-46a9-ba99-d468afef8a73`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:40:16,459::task::595::TaskManager.Task::(_updateState) Task=`a2141fbe-c2ad-4540-acb6-9445763c0b9a`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:40:16,460::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:40:16,460::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:40:16,460::task::1185::TaskManager.Task::(prepare) Task=`a2141fbe-c2ad-4540-acb6-9445763c0b9a`::finished: {}
Thread-27::DEBUG::2014-11-28 15:40:16,460::task::595::TaskManager.Task::(_updateState) Task=`a2141fbe-c2ad-4540-acb6-9445763c0b9a`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:40:16,460::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:40:16,461::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:40:16,461::task::990::TaskManager.Task::(_decref) Task=`a2141fbe-c2ad-4540-acb6-9445763c0b9a`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:40:31,566::task::595::TaskManager.Task::(_updateState) Task=`bc5756fb-6f0b-4811-8fe2-f66779115d08`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:40:31,567::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:40:31,567::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:40:31,567::task::1185::TaskManager.Task::(prepare) Task=`bc5756fb-6f0b-4811-8fe2-f66779115d08`::finished: {}
Thread-27::DEBUG::2014-11-28 15:40:31,567::task::595::TaskManager.Task::(_updateState) Task=`bc5756fb-6f0b-4811-8fe2-f66779115d08`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:40:31,567::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:40:31,568::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:40:31,568::task::990::TaskManager.Task::(_decref) Task=`bc5756fb-6f0b-4811-8fe2-f66779115d08`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:40:46,672::task::595::TaskManager.Task::(_updateState) Task=`c26049c9-5bbc-4dd0-9c9c-6b574a4510fd`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:40:46,673::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:40:46,673::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:40:46,673::task::1185::TaskManager.Task::(prepare) Task=`c26049c9-5bbc-4dd0-9c9c-6b574a4510fd`::finished: {}
Thread-27::DEBUG::2014-11-28 15:40:46,673::task::595::TaskManager.Task::(_updateState) Task=`c26049c9-5bbc-4dd0-9c9c-6b574a4510fd`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:40:46,673::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:40:46,674::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:40:46,674::task::990::TaskManager.Task::(_decref) Task=`c26049c9-5bbc-4dd0-9c9c-6b574a4510fd`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:41:01,784::task::595::TaskManager.Task::(_updateState) Task=`3150cf76-636c-4abd-8d0e-30aa3d8db501`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:41:01,785::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:41:01,785::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:41:01,785::task::1185::TaskManager.Task::(prepare) Task=`3150cf76-636c-4abd-8d0e-30aa3d8db501`::finished: {}
Thread-27::DEBUG::2014-11-28 15:41:01,785::task::595::TaskManager.Task::(_updateState) Task=`3150cf76-636c-4abd-8d0e-30aa3d8db501`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:41:01,785::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:41:01,785::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:41:01,786::task::990::TaskManager.Task::(_decref) Task=`3150cf76-636c-4abd-8d0e-30aa3d8db501`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:41:16,874::task::595::TaskManager.Task::(_updateState) Task=`6ded1b37-a40b-4fd2-8914-f651280ce498`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:41:16,874::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:41:16,875::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:41:16,875::task::1185::TaskManager.Task::(prepare) Task=`6ded1b37-a40b-4fd2-8914-f651280ce498`::finished: {}
Thread-27::DEBUG::2014-11-28 15:41:16,875::task::595::TaskManager.Task::(_updateState) Task=`6ded1b37-a40b-4fd2-8914-f651280ce498`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:41:16,875::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:41:16,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:41:16,876::task::990::TaskManager.Task::(_decref) Task=`6ded1b37-a40b-4fd2-8914-f651280ce498`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:41:31,985::task::595::TaskManager.Task::(_updateState) Task=`66c7a67d-946a-4ad2-bae7-ff867f508dd1`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:41:31,985::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:41:31,985::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:41:31,986::task::1185::TaskManager.Task::(prepare) Task=`66c7a67d-946a-4ad2-bae7-ff867f508dd1`::finished: {}
Thread-27::DEBUG::2014-11-28 15:41:31,986::task::595::TaskManager.Task::(_updateState) Task=`66c7a67d-946a-4ad2-bae7-ff867f508dd1`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:41:31,986::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:41:31,986::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:41:31,986::task::990::TaskManager.Task::(_decref) Task=`66c7a67d-946a-4ad2-bae7-ff867f508dd1`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:41:47,097::task::595::TaskManager.Task::(_updateState) Task=`03ae9fdf-d049-464a-ac0e-6ccd6c1f1ebc`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:41:47,097::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:41:47,097::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:41:47,098::task::1185::TaskManager.Task::(prepare) Task=`03ae9fdf-d049-464a-ac0e-6ccd6c1f1ebc`::finished: {}
Thread-27::DEBUG::2014-11-28 15:41:47,098::task::595::TaskManager.Task::(_updateState) Task=`03ae9fdf-d049-464a-ac0e-6ccd6c1f1ebc`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:41:47,098::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:41:47,098::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:41:47,098::task::990::TaskManager.Task::(_decref) Task=`03ae9fdf-d049-464a-ac0e-6ccd6c1f1ebc`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:42:02,184::task::595::TaskManager.Task::(_updateState) Task=`00aa6da2-3e1e-47ca-80a1-3188f1624296`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:42:02,184::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:42:02,185::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:42:02,185::task::1185::TaskManager.Task::(prepare) Task=`00aa6da2-3e1e-47ca-80a1-3188f1624296`::finished: {}
Thread-27::DEBUG::2014-11-28 15:42:02,185::task::595::TaskManager.Task::(_updateState) Task=`00aa6da2-3e1e-47ca-80a1-3188f1624296`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:42:02,185::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:42:02,185::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:42:02,186::task::990::TaskManager.Task::(_decref) Task=`00aa6da2-3e1e-47ca-80a1-3188f1624296`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 15:42:11,012::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::ERROR::2014-11-28 15:42:11,013::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::DEBUG::2014-11-28 15:42:11,013::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:42:11,013::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:42:11,013::task::1185::TaskManager.Task::(prepare) Task=`83bc598d-4ed3-43e1-85ea-be3d404e6b5a`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:42:11,014::task::595::TaskManager.Task::(_updateState) Task=`83bc598d-4ed3-43e1-85ea-be3d404e6b5a`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:42:11,014::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:42:11,014::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:42:11,014::task::990::TaskManager.Task::(_decref) Task=`83bc598d-4ed3-43e1-85ea-be3d404e6b5a`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:42:11,166::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2968cd34]
Thread-22::DEBUG::2014-11-28 15:42:11,167::task::595::TaskManager.Task::(_updateState) Task=`2613443e-dc82-46f1-a61d-b1c2576b53fd`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:42:11,167::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:42:11,173::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:42:17,266::task::595::TaskManager.Task::(_updateState) Task=`ff539f88-d81d-43d4-bb2b-19059627101b`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:42:17,266::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:42:17,266::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:42:17,266::task::1185::TaskManager.Task::(prepare) Task=`ff539f88-d81d-43d4-bb2b-19059627101b`::finished: {}
Thread-27::DEBUG::2014-11-28 15:42:17,266::task::595::TaskManager.Task::(_updateState) Task=`ff539f88-d81d-43d4-bb2b-19059627101b`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:42:17,267::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:42:17,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:42:17,267::task::990::TaskManager.Task::(_decref) Task=`ff539f88-d81d-43d4-bb2b-19059627101b`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:42:32,366::task::595::TaskManager.Task::(_updateState) Task=`c98865d0-8fea-4d15-8fc0-ba34efa8c1ec`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:42:32,366::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:42:32,366::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:42:32,367::task::1185::TaskManager.Task::(prepare) Task=`c98865d0-8fea-4d15-8fc0-ba34efa8c1ec`::finished: {}
Thread-27::DEBUG::2014-11-28 15:42:32,367::task::595::TaskManager.Task::(_updateState) Task=`c98865d0-8fea-4d15-8fc0-ba34efa8c1ec`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:42:32,367::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:42:32,367::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:42:32,368::task::990::TaskManager.Task::(_decref) Task=`c98865d0-8fea-4d15-8fc0-ba34efa8c1ec`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:42:47,448::task::595::TaskManager.Task::(_updateState) Task=`6243eb19-83c6-48ee-be74-657ff5b3f6b6`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:42:47,449::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:42:47,449::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:42:47,449::task::1185::TaskManager.Task::(prepare) Task=`6243eb19-83c6-48ee-be74-657ff5b3f6b6`::finished: {}
Thread-27::DEBUG::2014-11-28 15:42:47,449::task::595::TaskManager.Task::(_updateState) Task=`6243eb19-83c6-48ee-be74-657ff5b3f6b6`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:42:47,449::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:42:47,450::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:42:47,450::task::990::TaskManager.Task::(_decref) Task=`6243eb19-83c6-48ee-be74-657ff5b3f6b6`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:43:02,529::task::595::TaskManager.Task::(_updateState) Task=`288f9750-63df-48a8-93c3-17180beb71d1`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:43:02,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:43:02,530::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:43:02,530::task::1185::TaskManager.Task::(prepare) Task=`288f9750-63df-48a8-93c3-17180beb71d1`::finished: {}
Thread-27::DEBUG::2014-11-28 15:43:02,530::task::595::TaskManager.Task::(_updateState) Task=`288f9750-63df-48a8-93c3-17180beb71d1`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:43:02,530::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:43:02,531::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:43:02,531::task::990::TaskManager.Task::(_decref) Task=`288f9750-63df-48a8-93c3-17180beb71d1`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:43:11,174::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:43:11,174::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:43:11,175::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:43:11,175::task::1185::TaskManager.Task::(prepare) Task=`2613443e-dc82-46f1-a61d-b1c2576b53fd`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:43:11,175::task::595::TaskManager.Task::(_updateState) Task=`2613443e-dc82-46f1-a61d-b1c2576b53fd`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:43:11,175::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:43:11,175::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:43:11,176::task::990::TaskManager.Task::(_decref) Task=`2613443e-dc82-46f1-a61d-b1c2576b53fd`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:43:11,283::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5de22e3d]
Thread-22::DEBUG::2014-11-28 15:43:11,283::task::595::TaskManager.Task::(_updateState) Task=`6480abb4-21b9-4ae2-9897-6b3163f86d8f`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:43:11,283::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:43:11,290::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:43:17,615::task::595::TaskManager.Task::(_updateState) Task=`afe581c0-08f4-43de-baeb-8be6e333b7d7`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:43:17,615::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:43:17,615::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:43:17,616::task::1185::TaskManager.Task::(prepare) Task=`afe581c0-08f4-43de-baeb-8be6e333b7d7`::finished: {}
Thread-27::DEBUG::2014-11-28 15:43:17,616::task::595::TaskManager.Task::(_updateState) Task=`afe581c0-08f4-43de-baeb-8be6e333b7d7`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:43:17,616::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:43:17,616::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:43:17,616::task::990::TaskManager.Task::(_decref) Task=`afe581c0-08f4-43de-baeb-8be6e333b7d7`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:43:32,702::task::595::TaskManager.Task::(_updateState) Task=`9aa06194-9b9b-49bf-84b6-1179de8ea916`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:43:32,703::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:43:32,703::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:43:32,703::task::1185::TaskManager.Task::(prepare) Task=`9aa06194-9b9b-49bf-84b6-1179de8ea916`::finished: {}
Thread-27::DEBUG::2014-11-28 15:43:32,703::task::595::TaskManager.Task::(_updateState) Task=`9aa06194-9b9b-49bf-84b6-1179de8ea916`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:43:32,703::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:43:32,704::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:43:32,704::task::990::TaskManager.Task::(_decref) Task=`9aa06194-9b9b-49bf-84b6-1179de8ea916`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:43:47,803::task::595::TaskManager.Task::(_updateState) Task=`51943da9-42e9-4220-8272-33b7d26e331e`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:43:47,804::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:43:47,804::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:43:47,804::task::1185::TaskManager.Task::(prepare) Task=`51943da9-42e9-4220-8272-33b7d26e331e`::finished: {}
Thread-27::DEBUG::2014-11-28 15:43:47,804::task::595::TaskManager.Task::(_updateState) Task=`51943da9-42e9-4220-8272-33b7d26e331e`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:43:47,805::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:43:47,805::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:43:47,805::task::990::TaskManager.Task::(_decref) Task=`51943da9-42e9-4220-8272-33b7d26e331e`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:44:02,887::task::595::TaskManager.Task::(_updateState) Task=`b9a3c23b-a206-46e0-80b0-547f884c08de`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:44:02,888::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:44:02,888::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:44:02,888::task::1185::TaskManager.Task::(prepare) Task=`b9a3c23b-a206-46e0-80b0-547f884c08de`::finished: {}
Thread-27::DEBUG::2014-11-28 15:44:02,888::task::595::TaskManager.Task::(_updateState) Task=`b9a3c23b-a206-46e0-80b0-547f884c08de`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:44:02,889::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:44:02,889::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:44:02,889::task::990::TaskManager.Task::(_decref) Task=`b9a3c23b-a206-46e0-80b0-547f884c08de`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:11,294::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:44:11,295::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:44:11,295::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:44:11,295::task::1185::TaskManager.Task::(prepare) Task=`6480abb4-21b9-4ae2-9897-6b3163f86d8f`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:44:11,295::task::595::TaskManager.Task::(_updateState) Task=`6480abb4-21b9-4ae2-9897-6b3163f86d8f`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:11,295::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:11,296::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:11,296::task::990::TaskManager.Task::(_decref) Task=`6480abb4-21b9-4ae2-9897-6b3163f86d8f`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:11,370::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7b7099fd]
Thread-22::DEBUG::2014-11-28 15:44:11,371::task::595::TaskManager.Task::(_updateState) Task=`d5005da5-b33e-43d4-9c39-df4d554b428b`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:11,371::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:44:11,377::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-22::DEBUG::2014-11-28 15:44:11,525::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 15:44:11,525::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:44:11,525::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:44:11,525::task::1185::TaskManager.Task::(prepare) Task=`d5005da5-b33e-43d4-9c39-df4d554b428b`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:44:11,526::task::595::TaskManager.Task::(_updateState) Task=`d5005da5-b33e-43d4-9c39-df4d554b428b`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:11,526::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:11,526::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:11,526::task::990::TaskManager.Task::(_decref) Task=`d5005da5-b33e-43d4-9c39-df4d554b428b`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:17,977::task::595::TaskManager.Task::(_updateState) Task=`05fd566f-7be8-4a92-bc76-03e0a9f93d30`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:17,977::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:44:17,977::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:44:17,977::task::1185::TaskManager.Task::(prepare) Task=`05fd566f-7be8-4a92-bc76-03e0a9f93d30`::finished: {}
Thread-22::DEBUG::2014-11-28 15:44:17,977::task::595::TaskManager.Task::(_updateState) Task=`05fd566f-7be8-4a92-bc76-03e0a9f93d30`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:17,978::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:17,978::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:17,978::task::990::TaskManager.Task::(_decref) Task=`05fd566f-7be8-4a92-bc76-03e0a9f93d30`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:33,075::task::595::TaskManager.Task::(_updateState) Task=`13e4b98c-28ba-4537-8e7b-33f9f6a72ce7`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:33,076::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:44:33,076::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:44:33,076::task::1185::TaskManager.Task::(prepare) Task=`13e4b98c-28ba-4537-8e7b-33f9f6a72ce7`::finished: {}
Thread-22::DEBUG::2014-11-28 15:44:33,076::task::595::TaskManager.Task::(_updateState) Task=`13e4b98c-28ba-4537-8e7b-33f9f6a72ce7`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:33,076::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:33,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:33,077::task::990::TaskManager.Task::(_decref) Task=`13e4b98c-28ba-4537-8e7b-33f9f6a72ce7`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:48,160::task::595::TaskManager.Task::(_updateState) Task=`ed2f042e-6502-49a8-9c73-d2ab22a1e78e`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:48,161::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:44:48,161::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:44:48,161::task::1185::TaskManager.Task::(prepare) Task=`ed2f042e-6502-49a8-9c73-d2ab22a1e78e`::finished: {}
Thread-22::DEBUG::2014-11-28 15:44:48,161::task::595::TaskManager.Task::(_updateState) Task=`ed2f042e-6502-49a8-9c73-d2ab22a1e78e`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:48,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:48,162::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:48,162::task::990::TaskManager.Task::(_decref) Task=`ed2f042e-6502-49a8-9c73-d2ab22a1e78e`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:59,680::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7d038482]
Thread-22::DEBUG::2014-11-28 15:44:59,681::task::595::TaskManager.Task::(_updateState) Task=`54bad5d4-3a46-4071-8b71-a24e6a7ebb7c`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:59,681::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:44:59,688::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-22::ERROR::2014-11-28 15:44:59,716::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::ERROR::2014-11-28 15:44:59,717::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::DEBUG::2014-11-28 15:44:59,717::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:44:59,718::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:44:59,718::task::1185::TaskManager.Task::(prepare) Task=`54bad5d4-3a46-4071-8b71-a24e6a7ebb7c`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:44:59,718::task::595::TaskManager.Task::(_updateState) Task=`54bad5d4-3a46-4071-8b71-a24e6a7ebb7c`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:44:59,718::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:44:59,718::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:44:59,719::task::990::TaskManager.Task::(_decref) Task=`54bad5d4-3a46-4071-8b71-a24e6a7ebb7c`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:44:59,867::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [36cc420c]
Thread-22::DEBUG::2014-11-28 15:44:59,868::task::595::TaskManager.Task::(_updateState) Task=`b053fa62-e0b8-4d25-8cdc-b94740a36dea`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:44:59,868::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:44:59,894::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-27::DEBUG::2014-11-28 15:45:03,248::task::595::TaskManager.Task::(_updateState) Task=`d5026206-800d-4788-a5b1-ce8c928a504c`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:45:03,248::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:45:03,248::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:45:03,248::task::1185::TaskManager.Task::(prepare) Task=`d5026206-800d-4788-a5b1-ce8c928a504c`::finished: {}
Thread-27::DEBUG::2014-11-28 15:45:03,248::task::595::TaskManager.Task::(_updateState) Task=`d5026206-800d-4788-a5b1-ce8c928a504c`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:45:03,249::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:45:03,249::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:45:03,249::task::990::TaskManager.Task::(_decref) Task=`d5026206-800d-4788-a5b1-ce8c928a504c`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:45:18,335::task::595::TaskManager.Task::(_updateState) Task=`e109541e-8666-460e-aa26-01b38cd023d3`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:45:18,336::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:45:18,336::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:45:18,336::task::1185::TaskManager.Task::(prepare) Task=`e109541e-8666-460e-aa26-01b38cd023d3`::finished: {}
Thread-27::DEBUG::2014-11-28 15:45:18,336::task::595::TaskManager.Task::(_updateState) Task=`e109541e-8666-460e-aa26-01b38cd023d3`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:45:18,337::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:45:18,337::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:45:18,337::task::990::TaskManager.Task::(_decref) Task=`e109541e-8666-460e-aa26-01b38cd023d3`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:45:33,419::task::595::TaskManager.Task::(_updateState) Task=`49e97eaf-90e8-4b60-a2e7-bb034d30a200`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:45:33,419::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:45:33,419::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:45:33,420::task::1185::TaskManager.Task::(prepare) Task=`49e97eaf-90e8-4b60-a2e7-bb034d30a200`::finished: {}
Thread-27::DEBUG::2014-11-28 15:45:33,420::task::595::TaskManager.Task::(_updateState) Task=`49e97eaf-90e8-4b60-a2e7-bb034d30a200`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:45:33,420::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:45:33,420::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:45:33,420::task::990::TaskManager.Task::(_decref) Task=`49e97eaf-90e8-4b60-a2e7-bb034d30a200`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:45:39,286::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:45:39,287::task::595::TaskManager.Task::(_updateState) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:45:39,287::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-27::ERROR::2014-11-28 15:45:39,287::task::866::TaskManager.Task::(_setError) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-27::DEBUG::2014-11-28 15:45:39,288::task::885::TaskManager.Task::(_run) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::Task._run: 111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:45:39,288::task::1211::TaskManager.Task::(stop) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:45:39,288::task::990::TaskManager.Task::(_decref) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:45:39,288::task::1168::TaskManager.Task::(prepare) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-27::DEBUG::2014-11-28 15:45:39,288::task::1173::TaskManager.Task::(prepare) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::Prepare: aborted: Unknown pool id, pool not connected
Thread-27::DEBUG::2014-11-28 15:45:39,289::task::990::TaskManager.Task::(_decref) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:45:39,289::task::925::TaskManager.Task::(_doAbort) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:45:39,289::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:45:39,289::task::595::TaskManager.Task::(_updateState) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:45:39,289::task::550::TaskManager.Task::(__state_aborting) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:45:39,290::task::595::TaskManager.Task::(_updateState) Task=`111b76e8-5cb2-4a15-9f2c-fa7f8d45d3f2`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:45:39,290::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:45:39,290::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:45:39,290::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-27::DEBUG::2014-11-28 15:45:39,303::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:45:39,303::task::595::TaskManager.Task::(_updateState) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:45:39,303::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24238, domainsMap=None, options=None)
Thread-27::DEBUG::2014-11-28 15:45:39,304::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f83069f9-e563-4538-9841-44a2eb16865a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:45:39,304::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:45:39,304::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:45:39,304::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f83069f9-e563-4538-9841-44a2eb16865a`::Granted request
Thread-27::DEBUG::2014-11-28 15:45:39,305::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`4d9d42ff-4f6a-4d1a-80b0-c40984085011`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:45:39,305::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:45:39,305::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:45:39,305::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`4d9d42ff-4f6a-4d1a-80b0-c40984085011`::Granted request
Thread-27::INFO::2014-11-28 15:45:39,306::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x7fc2380c9950
Thread-27::INFO::2014-11-28 15:45:39,306::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24238)
Thread-27::DEBUG::2014-11-28 15:45:39,306::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,307::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,307::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,307::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,307::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,308::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,308::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-27::DEBUG::2014-11-28 15:45:39,308::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:45:39,308::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-27::DEBUG::2014-11-28 15:45:39,308::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:45:39,309::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-27::DEBUG::2014-11-28 15:45:39,309::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-27::DEBUG::2014-11-28 15:45:39,322::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::DEBUG::2014-11-28 15:45:39,324::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-27::DEBUG::2014-11-28 15:45:39,406::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-27::DEBUG::2014-11-28 15:45:39,407::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,407::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,407::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,408::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,408::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,408::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:45:39,408::misc::748::SamplingMethod::(__call__) Returning last result
Thread-32::WARNING::2014-11-28 15:46:39,417::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-33::WARNING::2014-11-28 15:46:39,418::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-33::DEBUG::2014-11-28 15:46:39,419::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-27::ERROR::2014-11-28 15:46:39,420::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-27::DEBUG::2014-11-28 15:46:39,420::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-27::DEBUG::2014-11-28 15:46:39,420::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-27::DEBUG::2014-11-28 15:46:39,421::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:46:39,421::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-27::DEBUG::2014-11-28 15:46:39,421::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-27::DEBUG::2014-11-28 15:46:39,421::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-27::DEBUG::2014-11-28 15:46:39,422::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:46:39,422::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-27::ERROR::2014-11-28 15:46:39,422::task::866::TaskManager.Task::(_setError) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-27::DEBUG::2014-11-28 15:46:39,422::task::885::TaskManager.Task::(_run) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::Task._run: 6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24238, None) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:46:39,423::task::1211::TaskManager.Task::(stop) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:46:39,423::task::990::TaskManager.Task::(_decref) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:46:39,423::task::1168::TaskManager.Task::(prepare) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-27::DEBUG::2014-11-28 15:46:39,423::task::1173::TaskManager.Task::(prepare) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::Prepare: aborted: Cannot find master domain
Thread-27::DEBUG::2014-11-28 15:46:39,423::task::990::TaskManager.Task::(_decref) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:46:39,424::task::925::TaskManager.Task::(_doAbort) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:46:39,424::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:46:39,424::task::595::TaskManager.Task::(_updateState) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:46:39,424::task::550::TaskManager.Task::(__state_aborting) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:46:39,424::task::595::TaskManager.Task::(_updateState) Task=`6bc3d2cc-3c9b-4cbe-a8c0-af445fbe18b1`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:46:39,425::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:46:39,425::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:46:39,425::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-27::DEBUG::2014-11-28 15:46:39,453::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:46:39,453::task::595::TaskManager.Task::(_updateState) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:46:39,454::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-27::ERROR::2014-11-28 15:46:39,454::task::866::TaskManager.Task::(_setError) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-27::DEBUG::2014-11-28 15:46:39,454::task::885::TaskManager.Task::(_run) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::Task._run: cf0988d0-2cc5-43fb-8066-83680423644d ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:46:39,454::task::1211::TaskManager.Task::(stop) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:46:39,455::task::990::TaskManager.Task::(_decref) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:46:39,455::task::1168::TaskManager.Task::(prepare) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-27::DEBUG::2014-11-28 15:46:39,455::task::1173::TaskManager.Task::(prepare) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::Prepare: aborted: Unknown pool id, pool not connected
Thread-27::DEBUG::2014-11-28 15:46:39,455::task::990::TaskManager.Task::(_decref) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:46:39,455::task::925::TaskManager.Task::(_doAbort) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:46:39,455::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:46:39,456::task::595::TaskManager.Task::(_updateState) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:46:39,456::task::550::TaskManager.Task::(__state_aborting) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:46:39,456::task::595::TaskManager.Task::(_updateState) Task=`cf0988d0-2cc5-43fb-8066-83680423644d`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:46:39,456::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:46:39,456::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:46:39,457::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-27::DEBUG::2014-11-28 15:46:48,467::task::595::TaskManager.Task::(_updateState) Task=`bb6ee29c-6111-4bc0-a238-e318a341f423`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:46:48,468::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:46:48,468::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:46:48,468::task::1185::TaskManager.Task::(prepare) Task=`bb6ee29c-6111-4bc0-a238-e318a341f423`::finished: {}
Thread-27::DEBUG::2014-11-28 15:46:48,468::task::595::TaskManager.Task::(_updateState) Task=`bb6ee29c-6111-4bc0-a238-e318a341f423`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:46:48,468::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:46:48,468::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:46:48,469::task::990::TaskManager.Task::(_decref) Task=`bb6ee29c-6111-4bc0-a238-e318a341f423`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:47:03,549::task::595::TaskManager.Task::(_updateState) Task=`c0563989-e355-4200-a694-dada165b3034`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:47:03,550::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:47:03,550::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:47:03,550::task::1185::TaskManager.Task::(prepare) Task=`c0563989-e355-4200-a694-dada165b3034`::finished: {}
Thread-27::DEBUG::2014-11-28 15:47:03,550::task::595::TaskManager.Task::(_updateState) Task=`c0563989-e355-4200-a694-dada165b3034`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:47:03,550::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:47:03,550::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:47:03,551::task::990::TaskManager.Task::(_decref) Task=`c0563989-e355-4200-a694-dada165b3034`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 15:47:11,004::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::ERROR::2014-11-28 15:47:11,005::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::DEBUG::2014-11-28 15:47:11,005::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:47:11,005::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:47:11,005::task::1185::TaskManager.Task::(prepare) Task=`b053fa62-e0b8-4d25-8cdc-b94740a36dea`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:47:11,006::task::595::TaskManager.Task::(_updateState) Task=`b053fa62-e0b8-4d25-8cdc-b94740a36dea`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:47:11,006::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:47:11,006::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:47:11,006::task::990::TaskManager.Task::(_decref) Task=`b053fa62-e0b8-4d25-8cdc-b94740a36dea`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:47:11,119::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5e7e7847]
Thread-22::DEBUG::2014-11-28 15:47:11,119::task::595::TaskManager.Task::(_updateState) Task=`6dfe8df0-e2ed-4478-a2aa-1d5cbd1ac46b`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:47:11,120::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:47:11,126::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:47:18,632::task::595::TaskManager.Task::(_updateState) Task=`54517490-a6c2-4b3e-98af-735818977eb2`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:47:18,632::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:47:18,633::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:47:18,633::task::1185::TaskManager.Task::(prepare) Task=`54517490-a6c2-4b3e-98af-735818977eb2`::finished: {}
Thread-27::DEBUG::2014-11-28 15:47:18,633::task::595::TaskManager.Task::(_updateState) Task=`54517490-a6c2-4b3e-98af-735818977eb2`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:47:18,633::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:47:18,633::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:47:18,633::task::990::TaskManager.Task::(_decref) Task=`54517490-a6c2-4b3e-98af-735818977eb2`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:47:33,715::task::595::TaskManager.Task::(_updateState) Task=`99f998e9-9871-4526-9b3b-4d5ad07960ae`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:47:33,715::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:47:33,715::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:47:33,716::task::1185::TaskManager.Task::(prepare) Task=`99f998e9-9871-4526-9b3b-4d5ad07960ae`::finished: {}
Thread-27::DEBUG::2014-11-28 15:47:33,716::task::595::TaskManager.Task::(_updateState) Task=`99f998e9-9871-4526-9b3b-4d5ad07960ae`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:47:33,716::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:47:33,716::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:47:33,716::task::990::TaskManager.Task::(_decref) Task=`99f998e9-9871-4526-9b3b-4d5ad07960ae`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:47:48,798::task::595::TaskManager.Task::(_updateState) Task=`450688b6-a3f9-49fe-8383-beab25fb3bac`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:47:48,799::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:47:48,799::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:47:48,799::task::1185::TaskManager.Task::(prepare) Task=`450688b6-a3f9-49fe-8383-beab25fb3bac`::finished: {}
Thread-27::DEBUG::2014-11-28 15:47:48,799::task::595::TaskManager.Task::(_updateState) Task=`450688b6-a3f9-49fe-8383-beab25fb3bac`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:47:48,799::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:47:48,799::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:47:48,800::task::990::TaskManager.Task::(_decref) Task=`450688b6-a3f9-49fe-8383-beab25fb3bac`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:48:03,881::task::595::TaskManager.Task::(_updateState) Task=`a27c696b-3fd9-4a6b-81af-da60344c058f`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:48:03,881::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:48:03,881::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:48:03,882::task::1185::TaskManager.Task::(prepare) Task=`a27c696b-3fd9-4a6b-81af-da60344c058f`::finished: {}
Thread-27::DEBUG::2014-11-28 15:48:03,882::task::595::TaskManager.Task::(_updateState) Task=`a27c696b-3fd9-4a6b-81af-da60344c058f`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:48:03,882::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:48:03,882::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:48:03,882::task::990::TaskManager.Task::(_decref) Task=`a27c696b-3fd9-4a6b-81af-da60344c058f`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:48:11,126::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:48:11,127::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:48:11,127::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:48:11,127::task::1185::TaskManager.Task::(prepare) Task=`6dfe8df0-e2ed-4478-a2aa-1d5cbd1ac46b`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:48:11,127::task::595::TaskManager.Task::(_updateState) Task=`6dfe8df0-e2ed-4478-a2aa-1d5cbd1ac46b`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:48:11,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:48:11,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:48:11,128::task::990::TaskManager.Task::(_decref) Task=`6dfe8df0-e2ed-4478-a2aa-1d5cbd1ac46b`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:48:11,240::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1e2c5053]
Thread-22::DEBUG::2014-11-28 15:48:11,240::task::595::TaskManager.Task::(_updateState) Task=`3a893582-46ce-412d-bcec-cb748acb1c35`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:48:11,240::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:48:11,247::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:48:18,967::task::595::TaskManager.Task::(_updateState) Task=`bfb326f8-82a5-4c38-a589-f9a90207b841`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:48:18,968::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:48:18,968::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:48:18,968::task::1185::TaskManager.Task::(prepare) Task=`bfb326f8-82a5-4c38-a589-f9a90207b841`::finished: {}
Thread-27::DEBUG::2014-11-28 15:48:18,968::task::595::TaskManager.Task::(_updateState) Task=`bfb326f8-82a5-4c38-a589-f9a90207b841`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:48:18,968::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:48:18,969::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:48:18,969::task::990::TaskManager.Task::(_decref) Task=`bfb326f8-82a5-4c38-a589-f9a90207b841`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:48:34,060::task::595::TaskManager.Task::(_updateState) Task=`1b75b156-1dd7-4be3-b803-0715fcb8aa4e`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:48:34,060::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:48:34,060::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:48:34,060::task::1185::TaskManager.Task::(prepare) Task=`1b75b156-1dd7-4be3-b803-0715fcb8aa4e`::finished: {}
Thread-27::DEBUG::2014-11-28 15:48:34,061::task::595::TaskManager.Task::(_updateState) Task=`1b75b156-1dd7-4be3-b803-0715fcb8aa4e`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:48:34,061::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:48:34,061::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:48:34,061::task::990::TaskManager.Task::(_decref) Task=`1b75b156-1dd7-4be3-b803-0715fcb8aa4e`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:48:49,150::task::595::TaskManager.Task::(_updateState) Task=`bf43ed22-9fa4-428d-b57c-78118eac5c58`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:48:49,150::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:48:49,151::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:48:49,151::task::1185::TaskManager.Task::(prepare) Task=`bf43ed22-9fa4-428d-b57c-78118eac5c58`::finished: {}
Thread-27::DEBUG::2014-11-28 15:48:49,151::task::595::TaskManager.Task::(_updateState) Task=`bf43ed22-9fa4-428d-b57c-78118eac5c58`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:48:49,151::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:48:49,151::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:48:49,152::task::990::TaskManager.Task::(_decref) Task=`bf43ed22-9fa4-428d-b57c-78118eac5c58`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:49:04,241::task::595::TaskManager.Task::(_updateState) Task=`902a4fbd-ef86-4312-b1c6-65de55416e2c`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:49:04,241::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:49:04,241::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:49:04,242::task::1185::TaskManager.Task::(prepare) Task=`902a4fbd-ef86-4312-b1c6-65de55416e2c`::finished: {}
Thread-27::DEBUG::2014-11-28 15:49:04,242::task::595::TaskManager.Task::(_updateState) Task=`902a4fbd-ef86-4312-b1c6-65de55416e2c`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:49:04,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:49:04,242::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:49:04,242::task::990::TaskManager.Task::(_decref) Task=`902a4fbd-ef86-4312-b1c6-65de55416e2c`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:11,251::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:49:11,252::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:49:11,252::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:49:11,252::task::1185::TaskManager.Task::(prepare) Task=`3a893582-46ce-412d-bcec-cb748acb1c35`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:49:11,252::task::595::TaskManager.Task::(_updateState) Task=`3a893582-46ce-412d-bcec-cb748acb1c35`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:11,253::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:11,253::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:11,253::task::990::TaskManager.Task::(_decref) Task=`3a893582-46ce-412d-bcec-cb748acb1c35`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:19,329::task::595::TaskManager.Task::(_updateState) Task=`7e38a5b6-40a9-4b00-8f70-4924f499b50f`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:19,329::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:49:19,329::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:49:19,329::task::1185::TaskManager.Task::(prepare) Task=`7e38a5b6-40a9-4b00-8f70-4924f499b50f`::finished: {}
Thread-22::DEBUG::2014-11-28 15:49:19,330::task::595::TaskManager.Task::(_updateState) Task=`7e38a5b6-40a9-4b00-8f70-4924f499b50f`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:19,330::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:19,330::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:19,330::task::990::TaskManager.Task::(_decref) Task=`7e38a5b6-40a9-4b00-8f70-4924f499b50f`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:34,414::task::595::TaskManager.Task::(_updateState) Task=`f6506183-d6db-4fa5-bd4d-8f50e5948a78`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:34,415::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:49:34,415::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:49:34,416::task::1185::TaskManager.Task::(prepare) Task=`f6506183-d6db-4fa5-bd4d-8f50e5948a78`::finished: {}
Thread-22::DEBUG::2014-11-28 15:49:34,416::task::595::TaskManager.Task::(_updateState) Task=`f6506183-d6db-4fa5-bd4d-8f50e5948a78`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:34,416::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:34,416::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:34,416::task::990::TaskManager.Task::(_decref) Task=`f6506183-d6db-4fa5-bd4d-8f50e5948a78`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:40,123::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3f33226d]
Thread-22::DEBUG::2014-11-28 15:49:40,123::task::595::TaskManager.Task::(_updateState) Task=`6af72ffc-4fb6-41e8-8496-7e28611f25c8`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:40,124::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:49:40,130::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-22::DEBUG::2014-11-28 15:49:40,277::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 15:49:40,277::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:49:40,278::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:49:40,278::task::1185::TaskManager.Task::(prepare) Task=`6af72ffc-4fb6-41e8-8496-7e28611f25c8`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:49:40,278::task::595::TaskManager.Task::(_updateState) Task=`6af72ffc-4fb6-41e8-8496-7e28611f25c8`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:40,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:40,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:40,279::task::990::TaskManager.Task::(_decref) Task=`6af72ffc-4fb6-41e8-8496-7e28611f25c8`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:49,502::task::595::TaskManager.Task::(_updateState) Task=`780bd099-3bc2-46eb-88d8-b8123774b5b6`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:49,502::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:49:49,502::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:49:49,502::task::1185::TaskManager.Task::(prepare) Task=`780bd099-3bc2-46eb-88d8-b8123774b5b6`::finished: {}
Thread-22::DEBUG::2014-11-28 15:49:49,502::task::595::TaskManager.Task::(_updateState) Task=`780bd099-3bc2-46eb-88d8-b8123774b5b6`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:49,503::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:49,503::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:49,503::task::990::TaskManager.Task::(_decref) Task=`780bd099-3bc2-46eb-88d8-b8123774b5b6`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:59,690::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [76000424]
Thread-22::DEBUG::2014-11-28 15:49:59,691::task::595::TaskManager.Task::(_updateState) Task=`326066ef-73cb-482e-8a93-f14390884685`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:59,691::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:49:59,698::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-22::ERROR::2014-11-28 15:49:59,723::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::ERROR::2014-11-28 15:49:59,723::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-22::DEBUG::2014-11-28 15:49:59,724::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:49:59,724::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:49:59,724::task::1185::TaskManager.Task::(prepare) Task=`326066ef-73cb-482e-8a93-f14390884685`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-22::DEBUG::2014-11-28 15:49:59,724::task::595::TaskManager.Task::(_updateState) Task=`326066ef-73cb-482e-8a93-f14390884685`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:49:59,724::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:49:59,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:49:59,725::task::990::TaskManager.Task::(_decref) Task=`326066ef-73cb-482e-8a93-f14390884685`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:49:59,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [f72dfff]
Thread-22::DEBUG::2014-11-28 15:49:59,850::task::595::TaskManager.Task::(_updateState) Task=`65786a88-e492-44e6-a763-624aa581df46`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:49:59,851::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:49:59,885::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-27::DEBUG::2014-11-28 15:50:04,587::task::595::TaskManager.Task::(_updateState) Task=`3bdf6429-f55e-44c1-be3b-14f1fc6d7fb8`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:50:04,588::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:50:04,588::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:50:04,588::task::1185::TaskManager.Task::(prepare) Task=`3bdf6429-f55e-44c1-be3b-14f1fc6d7fb8`::finished: {}
Thread-27::DEBUG::2014-11-28 15:50:04,588::task::595::TaskManager.Task::(_updateState) Task=`3bdf6429-f55e-44c1-be3b-14f1fc6d7fb8`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:50:04,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:50:04,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:50:04,589::task::990::TaskManager.Task::(_decref) Task=`3bdf6429-f55e-44c1-be3b-14f1fc6d7fb8`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:50:19,694::task::595::TaskManager.Task::(_updateState) Task=`1b2e3ac8-561e-49aa-a83a-52d37a986f6a`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:50:19,694::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:50:19,694::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:50:19,694::task::1185::TaskManager.Task::(prepare) Task=`1b2e3ac8-561e-49aa-a83a-52d37a986f6a`::finished: {}
Thread-27::DEBUG::2014-11-28 15:50:19,695::task::595::TaskManager.Task::(_updateState) Task=`1b2e3ac8-561e-49aa-a83a-52d37a986f6a`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:50:19,695::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:50:19,695::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:50:19,695::task::990::TaskManager.Task::(_decref) Task=`1b2e3ac8-561e-49aa-a83a-52d37a986f6a`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:50:34,776::task::595::TaskManager.Task::(_updateState) Task=`2a2da52d-33f6-41fd-9462-af3cc703c454`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:50:34,776::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:50:34,776::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:50:34,777::task::1185::TaskManager.Task::(prepare) Task=`2a2da52d-33f6-41fd-9462-af3cc703c454`::finished: {}
Thread-27::DEBUG::2014-11-28 15:50:34,777::task::595::TaskManager.Task::(_updateState) Task=`2a2da52d-33f6-41fd-9462-af3cc703c454`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:50:34,777::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:50:34,777::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:50:34,777::task::990::TaskManager.Task::(_decref) Task=`2a2da52d-33f6-41fd-9462-af3cc703c454`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:50:49,866::task::595::TaskManager.Task::(_updateState) Task=`dedc0d6e-094d-4dc8-9575-61f435f7d500`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:50:49,866::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:50:49,866::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:50:49,866::task::1185::TaskManager.Task::(prepare) Task=`dedc0d6e-094d-4dc8-9575-61f435f7d500`::finished: {}
Thread-27::DEBUG::2014-11-28 15:50:49,866::task::595::TaskManager.Task::(_updateState) Task=`dedc0d6e-094d-4dc8-9575-61f435f7d500`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:50:49,867::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:50:49,867::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:50:49,867::task::990::TaskManager.Task::(_decref) Task=`dedc0d6e-094d-4dc8-9575-61f435f7d500`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:51:04,949::task::595::TaskManager.Task::(_updateState) Task=`a8e0b940-14b6-47f3-a185-79b9345383a1`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:51:04,949::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:51:04,950::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:51:04,950::task::1185::TaskManager.Task::(prepare) Task=`a8e0b940-14b6-47f3-a185-79b9345383a1`::finished: {}
Thread-27::DEBUG::2014-11-28 15:51:04,950::task::595::TaskManager.Task::(_updateState) Task=`a8e0b940-14b6-47f3-a185-79b9345383a1`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:51:04,950::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:51:04,950::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:51:04,950::task::990::TaskManager.Task::(_decref) Task=`a8e0b940-14b6-47f3-a185-79b9345383a1`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:51:20,033::task::595::TaskManager.Task::(_updateState) Task=`fb20311c-c735-411d-97c0-1fef85774b01`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:51:20,033::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:51:20,034::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:51:20,034::task::1185::TaskManager.Task::(prepare) Task=`fb20311c-c735-411d-97c0-1fef85774b01`::finished: {}
Thread-27::DEBUG::2014-11-28 15:51:20,034::task::595::TaskManager.Task::(_updateState) Task=`fb20311c-c735-411d-97c0-1fef85774b01`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:51:20,034::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:51:20,034::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:51:20,034::task::990::TaskManager.Task::(_decref) Task=`fb20311c-c735-411d-97c0-1fef85774b01`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:51:35,134::task::595::TaskManager.Task::(_updateState) Task=`800ad2d1-2fbe-457b-bf8b-8ccbb803c812`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:51:35,135::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:51:35,135::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:51:35,135::task::1185::TaskManager.Task::(prepare) Task=`800ad2d1-2fbe-457b-bf8b-8ccbb803c812`::finished: {}
Thread-27::DEBUG::2014-11-28 15:51:35,135::task::595::TaskManager.Task::(_updateState) Task=`800ad2d1-2fbe-457b-bf8b-8ccbb803c812`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:51:35,135::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:51:35,136::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:51:35,136::task::990::TaskManager.Task::(_decref) Task=`800ad2d1-2fbe-457b-bf8b-8ccbb803c812`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:51:50,219::task::595::TaskManager.Task::(_updateState) Task=`a364d4c3-a3a6-46d6-a16c-a447723477cf`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:51:50,219::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:51:50,219::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:51:50,219::task::1185::TaskManager.Task::(prepare) Task=`a364d4c3-a3a6-46d6-a16c-a447723477cf`::finished: {}
Thread-27::DEBUG::2014-11-28 15:51:50,220::task::595::TaskManager.Task::(_updateState) Task=`a364d4c3-a3a6-46d6-a16c-a447723477cf`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:51:50,220::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:51:50,220::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:51:50,220::task::990::TaskManager.Task::(_decref) Task=`a364d4c3-a3a6-46d6-a16c-a447723477cf`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:52:05,302::task::595::TaskManager.Task::(_updateState) Task=`b59289d8-1fd3-43c4-b2f4-8d892247233b`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:52:05,302::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:52:05,302::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:52:05,303::task::1185::TaskManager.Task::(prepare) Task=`b59289d8-1fd3-43c4-b2f4-8d892247233b`::finished: {}
Thread-27::DEBUG::2014-11-28 15:52:05,303::task::595::TaskManager.Task::(_updateState) Task=`b59289d8-1fd3-43c4-b2f4-8d892247233b`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:52:05,303::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:52:05,303::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:52:05,303::task::990::TaskManager.Task::(_decref) Task=`b59289d8-1fd3-43c4-b2f4-8d892247233b`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 15:52:10,996::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::ERROR::2014-11-28 15:52:10,997::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-22::DEBUG::2014-11-28 15:52:10,997::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:52:10,997::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:52:10,997::task::1185::TaskManager.Task::(prepare) Task=`65786a88-e492-44e6-a763-624aa581df46`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-22::DEBUG::2014-11-28 15:52:10,998::task::595::TaskManager.Task::(_updateState) Task=`65786a88-e492-44e6-a763-624aa581df46`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:52:10,998::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:52:10,998::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:52:10,998::task::990::TaskManager.Task::(_decref) Task=`65786a88-e492-44e6-a763-624aa581df46`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:52:11,114::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [53d073b6]
Thread-22::DEBUG::2014-11-28 15:52:11,115::task::595::TaskManager.Task::(_updateState) Task=`66fd7d6d-c5ba-48ff-a03e-44659567eea3`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:52:11,115::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:52:11,121::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:52:20,385::task::595::TaskManager.Task::(_updateState) Task=`3dfb88f8-63f9-47c9-8e0d-2a8778b98d28`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:52:20,385::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:52:20,385::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:52:20,386::task::1185::TaskManager.Task::(prepare) Task=`3dfb88f8-63f9-47c9-8e0d-2a8778b98d28`::finished: {}
Thread-27::DEBUG::2014-11-28 15:52:20,386::task::595::TaskManager.Task::(_updateState) Task=`3dfb88f8-63f9-47c9-8e0d-2a8778b98d28`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:52:20,386::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:52:20,386::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:52:20,386::task::990::TaskManager.Task::(_decref) Task=`3dfb88f8-63f9-47c9-8e0d-2a8778b98d28`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:52:35,470::task::595::TaskManager.Task::(_updateState) Task=`17bdf6d8-da6d-4e86-8da9-f930aef3e164`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:52:35,470::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:52:35,470::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:52:35,471::task::1185::TaskManager.Task::(prepare) Task=`17bdf6d8-da6d-4e86-8da9-f930aef3e164`::finished: {}
Thread-27::DEBUG::2014-11-28 15:52:35,471::task::595::TaskManager.Task::(_updateState) Task=`17bdf6d8-da6d-4e86-8da9-f930aef3e164`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:52:35,471::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:52:35,471::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:52:35,471::task::990::TaskManager.Task::(_decref) Task=`17bdf6d8-da6d-4e86-8da9-f930aef3e164`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:52:50,554::task::595::TaskManager.Task::(_updateState) Task=`34aff9da-08e0-410f-9016-7e121e3d0fb8`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:52:50,554::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:52:50,554::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:52:50,555::task::1185::TaskManager.Task::(prepare) Task=`34aff9da-08e0-410f-9016-7e121e3d0fb8`::finished: {}
Thread-27::DEBUG::2014-11-28 15:52:50,555::task::595::TaskManager.Task::(_updateState) Task=`34aff9da-08e0-410f-9016-7e121e3d0fb8`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:52:50,555::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:52:50,555::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:52:50,555::task::990::TaskManager.Task::(_decref) Task=`34aff9da-08e0-410f-9016-7e121e3d0fb8`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:53:05,642::task::595::TaskManager.Task::(_updateState) Task=`7f41b660-d835-4c91-a4ce-6ba7ab7e49ff`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:53:05,642::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:53:05,642::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:53:05,642::task::1185::TaskManager.Task::(prepare) Task=`7f41b660-d835-4c91-a4ce-6ba7ab7e49ff`::finished: {}
Thread-27::DEBUG::2014-11-28 15:53:05,643::task::595::TaskManager.Task::(_updateState) Task=`7f41b660-d835-4c91-a4ce-6ba7ab7e49ff`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:53:05,643::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:53:05,643::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:53:05,643::task::990::TaskManager.Task::(_decref) Task=`7f41b660-d835-4c91-a4ce-6ba7ab7e49ff`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:53:11,122::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 15:53:11,122::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:53:11,122::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:53:11,123::task::1185::TaskManager.Task::(prepare) Task=`66fd7d6d-c5ba-48ff-a03e-44659567eea3`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-22::DEBUG::2014-11-28 15:53:11,123::task::595::TaskManager.Task::(_updateState) Task=`66fd7d6d-c5ba-48ff-a03e-44659567eea3`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:53:11,123::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:53:11,123::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:53:11,123::task::990::TaskManager.Task::(_decref) Task=`66fd7d6d-c5ba-48ff-a03e-44659567eea3`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:53:11,226::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [782e2031]
Thread-22::DEBUG::2014-11-28 15:53:11,226::task::595::TaskManager.Task::(_updateState) Task=`e049b55f-b50c-40b1-a7bb-f5a27a270e3d`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:53:11,226::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:53:11,232::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-22::DEBUG::2014-11-28 15:53:11,380::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 15:53:11,381::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:53:11,381::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:53:11,381::task::1185::TaskManager.Task::(prepare) Task=`e049b55f-b50c-40b1-a7bb-f5a27a270e3d`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-22::DEBUG::2014-11-28 15:53:11,381::task::595::TaskManager.Task::(_updateState) Task=`e049b55f-b50c-40b1-a7bb-f5a27a270e3d`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:53:11,381::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:53:11,382::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:53:11,382::task::990::TaskManager.Task::(_decref) Task=`e049b55f-b50c-40b1-a7bb-f5a27a270e3d`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:53:11,452::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2a4b8b4f]
Thread-22::DEBUG::2014-11-28 15:53:11,453::task::595::TaskManager.Task::(_updateState) Task=`77d99d29-54c3-40ae-9fcb-d7cc8af16897`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:53:11,453::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-22::DEBUG::2014-11-28 15:53:11,459::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:53:20,727::task::595::TaskManager.Task::(_updateState) Task=`ea016bda-b224-455f-9ee0-fd957cd19bc5`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:53:20,728::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:53:20,728::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:53:20,728::task::1185::TaskManager.Task::(prepare) Task=`ea016bda-b224-455f-9ee0-fd957cd19bc5`::finished: {}
Thread-27::DEBUG::2014-11-28 15:53:20,728::task::595::TaskManager.Task::(_updateState) Task=`ea016bda-b224-455f-9ee0-fd957cd19bc5`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:53:20,728::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:53:20,728::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:53:20,729::task::990::TaskManager.Task::(_decref) Task=`ea016bda-b224-455f-9ee0-fd957cd19bc5`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:53:35,813::task::595::TaskManager.Task::(_updateState) Task=`964dfaaf-7cb2-4dcb-9f00-316d5012a50e`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:53:35,813::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:53:35,813::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:53:35,813::task::1185::TaskManager.Task::(prepare) Task=`964dfaaf-7cb2-4dcb-9f00-316d5012a50e`::finished: {}
Thread-27::DEBUG::2014-11-28 15:53:35,814::task::595::TaskManager.Task::(_updateState) Task=`964dfaaf-7cb2-4dcb-9f00-316d5012a50e`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:53:35,814::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:53:35,814::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:53:35,814::task::990::TaskManager.Task::(_decref) Task=`964dfaaf-7cb2-4dcb-9f00-316d5012a50e`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:53:50,459::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:53:50,459::task::595::TaskManager.Task::(_updateState) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:53:50,460::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-27::ERROR::2014-11-28 15:53:50,460::task::866::TaskManager.Task::(_setError) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-27::DEBUG::2014-11-28 15:53:50,460::task::885::TaskManager.Task::(_run) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::Task._run: 1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:53:50,461::task::1211::TaskManager.Task::(stop) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:53:50,461::task::990::TaskManager.Task::(_decref) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::ref 1 aborting True
Thread-27::INFO::2014-11-28 15:53:50,461::task::1168::TaskManager.Task::(prepare) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-27::DEBUG::2014-11-28 15:53:50,461::task::1173::TaskManager.Task::(prepare) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::Prepare: aborted: Unknown pool id, pool not connected
Thread-27::DEBUG::2014-11-28 15:53:50,461::task::990::TaskManager.Task::(_decref) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:53:50,462::task::925::TaskManager.Task::(_doAbort) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:53:50,462::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:53:50,462::task::595::TaskManager.Task::(_updateState) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:53:50,462::task::550::TaskManager.Task::(__state_aborting) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:53:50,462::task::595::TaskManager.Task::(_updateState) Task=`1d7dee3f-93ec-4fb8-8cfd-18bcc65edbc2`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:53:50,462::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:53:50,463::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:53:50,463::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-27::DEBUG::2014-11-28 15:53:50,476::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:53:50,477::task::595::TaskManager.Task::(_updateState) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:53:50,477::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24239, domainsMap=None, options=None)
Thread-27::DEBUG::2014-11-28 15:53:50,477::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`648cd08c-4f30-4e3c-a5c7-db6c6d92002d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:53:50,478::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:53:50,478::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:53:50,478::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`648cd08c-4f30-4e3c-a5c7-db6c6d92002d`::Granted request
Thread-27::DEBUG::2014-11-28 15:53:50,479::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`c463e149-a49f-41b2-aa69-f41848c6c650`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-27::DEBUG::2014-11-28 15:53:50,479::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-27::DEBUG::2014-11-28 15:53:50,479::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-27::DEBUG::2014-11-28 15:53:50,480::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`c463e149-a49f-41b2-aa69-f41848c6c650`::Granted request
Thread-27::INFO::2014-11-28 15:53:50,480::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7fc2920bcf00 to type StoragePoolDiskBackend instance 0x7fc238099680
Thread-27::INFO::2014-11-28 15:53:50,480::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24239)
Thread-27::DEBUG::2014-11-28 15:53:50,480::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,481::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,481::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,481::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,481::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,482::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,482::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-27::DEBUG::2014-11-28 15:53:50,482::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:53:50,482::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-27::DEBUG::2014-11-28 15:53:50,482::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:53:50,483::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-27::DEBUG::2014-11-28 15:53:50,483::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-27::DEBUG::2014-11-28 15:53:50,494::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::DEBUG::2014-11-28 15:53:50,496::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-27::DEBUG::2014-11-28 15:53:50,542::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-27::DEBUG::2014-11-28 15:53:50,543::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,543::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,543::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,544::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,544::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,544::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:53:50,544::misc::748::SamplingMethod::(__call__) Returning last result
Thread-22::DEBUG::2014-11-28 15:53:56,707::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-22::DEBUG::2014-11-28 15:53:56,708::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-22::INFO::2014-11-28 15:53:56,708::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:53:56,709::task::1185::TaskManager.Task::(prepare) Task=`77d99d29-54c3-40ae-9fcb-d7cc8af16897`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-22::DEBUG::2014-11-28 15:53:56,709::task::595::TaskManager.Task::(_updateState) Task=`77d99d29-54c3-40ae-9fcb-d7cc8af16897`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:53:56,710::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:53:56,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:53:56,711::task::990::TaskManager.Task::(_decref) Task=`77d99d29-54c3-40ae-9fcb-d7cc8af16897`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:53:56,717::task::595::TaskManager.Task::(_updateState) Task=`9a5f650a-cb87-4f18-a724-8bdba8a72b67`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:53:56,718::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:53:56,719::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:53:56,720::task::1185::TaskManager.Task::(prepare) Task=`9a5f650a-cb87-4f18-a724-8bdba8a72b67`::finished: {}
Thread-22::DEBUG::2014-11-28 15:53:56,720::task::595::TaskManager.Task::(_updateState) Task=`9a5f650a-cb87-4f18-a724-8bdba8a72b67`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:53:56,721::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:53:56,722::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:53:56,722::task::990::TaskManager.Task::(_decref) Task=`9a5f650a-cb87-4f18-a724-8bdba8a72b67`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:53:56,732::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-27::DEBUG::2014-11-28 15:53:56,879::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-27::DEBUG::2014-11-28 15:53:56,882::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-22::DEBUG::2014-11-28 15:54:11,816::task::595::TaskManager.Task::(_updateState) Task=`a8056054-d73b-48b5-83b5-4589a4d29ac1`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:54:11,817::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:54:11,817::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:54:11,817::task::1185::TaskManager.Task::(prepare) Task=`a8056054-d73b-48b5-83b5-4589a4d29ac1`::finished: {}
Thread-22::DEBUG::2014-11-28 15:54:11,817::task::595::TaskManager.Task::(_updateState) Task=`a8056054-d73b-48b5-83b5-4589a4d29ac1`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:54:11,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:54:11,818::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:54:11,818::task::990::TaskManager.Task::(_decref) Task=`a8056054-d73b-48b5-83b5-4589a4d29ac1`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:54:26,896::task::595::TaskManager.Task::(_updateState) Task=`d6b0390a-b78a-456e-94f0-f97eb35d0dc6`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:54:26,896::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:54:26,896::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:54:26,896::task::1185::TaskManager.Task::(prepare) Task=`d6b0390a-b78a-456e-94f0-f97eb35d0dc6`::finished: {}
Thread-22::DEBUG::2014-11-28 15:54:26,897::task::595::TaskManager.Task::(_updateState) Task=`d6b0390a-b78a-456e-94f0-f97eb35d0dc6`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:54:26,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:54:26,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:54:26,897::task::990::TaskManager.Task::(_decref) Task=`d6b0390a-b78a-456e-94f0-f97eb35d0dc6`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:54:41,994::task::595::TaskManager.Task::(_updateState) Task=`cbfbe28a-483d-4969-9208-599ab19fb924`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:54:41,994::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:54:41,995::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:54:41,995::task::1185::TaskManager.Task::(prepare) Task=`cbfbe28a-483d-4969-9208-599ab19fb924`::finished: {}
Thread-22::DEBUG::2014-11-28 15:54:41,995::task::595::TaskManager.Task::(_updateState) Task=`cbfbe28a-483d-4969-9208-599ab19fb924`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:54:41,995::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:54:41,995::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:54:41,995::task::990::TaskManager.Task::(_decref) Task=`cbfbe28a-483d-4969-9208-599ab19fb924`::ref 0 aborting False
Thread-37::WARNING::2014-11-28 15:54:50,557::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-27::DEBUG::2014-11-28 15:54:56,883::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-27::DEBUG::2014-11-28 15:54:56,884::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-27::DEBUG::2014-11-28 15:54:56,884::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:54:56,884::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-27::DEBUG::2014-11-28 15:54:56,884::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-27::DEBUG::2014-11-28 15:54:56,884::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-27::DEBUG::2014-11-28 15:54:56,885::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-27::DEBUG::2014-11-28 15:54:56,885::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-27::ERROR::2014-11-28 15:54:56,885::task::866::TaskManager.Task::(_setError) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-27::DEBUG::2014-11-28 15:54:56,886::task::885::TaskManager.Task::(_run) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::Task._run: dbd2c5c2-f7f7-4436-a894-c18821b355d6 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 4, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24239, None) {} failed - stopping task
Thread-27::DEBUG::2014-11-28 15:54:56,886::task::1211::TaskManager.Task::(stop) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::stopping in state preparing (force False)
Thread-27::DEBUG::2014-11-28 15:54:56,886::task::990::TaskManager.Task::(_decref) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::ref 1 aborting True
Thread-27::DEBUG::2014-11-28 15:54:56,886::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-27::INFO::2014-11-28 15:54:56,887::task::1168::TaskManager.Task::(prepare) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::aborting: Task is aborted: u'' - code 100
Thread-27::DEBUG::2014-11-28 15:54:56,887::task::1173::TaskManager.Task::(prepare) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::Prepare: aborted: 
Thread-27::DEBUG::2014-11-28 15:54:56,887::task::990::TaskManager.Task::(_decref) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::ref 0 aborting True
Thread-27::DEBUG::2014-11-28 15:54:56,887::task::925::TaskManager.Task::(_doAbort) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::Task._doAbort: force False
Thread-27::DEBUG::2014-11-28 15:54:56,887::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:54:56,888::task::595::TaskManager.Task::(_updateState) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::moving from state preparing -> state aborting
Thread-27::DEBUG::2014-11-28 15:54:56,888::task::550::TaskManager.Task::(__state_aborting) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::_aborting: recover policy none
Thread-27::DEBUG::2014-11-28 15:54:56,888::task::595::TaskManager.Task::(_updateState) Task=`dbd2c5c2-f7f7-4436-a894-c18821b355d6`::moving from state aborting -> state failed
Thread-27::DEBUG::2014-11-28 15:54:56,888::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:54:56,888::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2014-11-28 15:54:56,889::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-27::DEBUG::2014-11-28 15:54:57,075::task::595::TaskManager.Task::(_updateState) Task=`efb89db1-a392-4504-b8ce-71ec95f20855`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:54:57,075::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:54:57,075::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:54:57,075::task::1185::TaskManager.Task::(prepare) Task=`efb89db1-a392-4504-b8ce-71ec95f20855`::finished: {}
Thread-27::DEBUG::2014-11-28 15:54:57,075::task::595::TaskManager.Task::(_updateState) Task=`efb89db1-a392-4504-b8ce-71ec95f20855`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:54:57,076::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:54:57,076::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:54:57,076::task::990::TaskManager.Task::(_decref) Task=`efb89db1-a392-4504-b8ce-71ec95f20855`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:54:59,693::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [6e832dbc]
Thread-27::DEBUG::2014-11-28 15:54:59,693::task::595::TaskManager.Task::(_updateState) Task=`1f3184ed-1cb4-4ad5-bd2e-a2143c78dc18`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:54:59,694::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:54:59,700::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-27::ERROR::2014-11-28 15:54:59,720::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-27::ERROR::2014-11-28 15:54:59,721::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-27::DEBUG::2014-11-28 15:54:59,721::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-27::INFO::2014-11-28 15:54:59,721::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-27::DEBUG::2014-11-28 15:54:59,722::task::1185::TaskManager.Task::(prepare) Task=`1f3184ed-1cb4-4ad5-bd2e-a2143c78dc18`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-27::DEBUG::2014-11-28 15:54:59,722::task::595::TaskManager.Task::(_updateState) Task=`1f3184ed-1cb4-4ad5-bd2e-a2143c78dc18`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:54:59,722::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:54:59,722::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:54:59,722::task::990::TaskManager.Task::(_decref) Task=`1f3184ed-1cb4-4ad5-bd2e-a2143c78dc18`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:54:59,850::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5efc59e4]
Thread-27::DEBUG::2014-11-28 15:54:59,851::task::595::TaskManager.Task::(_updateState) Task=`799bb4dc-06de-4b3f-8055-eb058fc80823`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:54:59,851::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:54:59,877::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-22::DEBUG::2014-11-28 15:55:06,917::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-22::DEBUG::2014-11-28 15:55:06,918::task::595::TaskManager.Task::(_updateState) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:06,918::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-22::ERROR::2014-11-28 15:55:06,919::task::866::TaskManager.Task::(_setError) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-22::DEBUG::2014-11-28 15:55:06,919::task::885::TaskManager.Task::(_run) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::Task._run: d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 15:55:06,919::task::1211::TaskManager.Task::(stop) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 15:55:06,920::task::990::TaskManager.Task::(_decref) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::ref 1 aborting True
Thread-22::INFO::2014-11-28 15:55:06,920::task::1168::TaskManager.Task::(prepare) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-22::DEBUG::2014-11-28 15:55:06,920::task::1173::TaskManager.Task::(prepare) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::Prepare: aborted: Unknown pool id, pool not connected
Thread-22::DEBUG::2014-11-28 15:55:06,920::task::990::TaskManager.Task::(_decref) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::ref 0 aborting True
Thread-22::DEBUG::2014-11-28 15:55:06,920::task::925::TaskManager.Task::(_doAbort) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::Task._doAbort: force False
Thread-22::DEBUG::2014-11-28 15:55:06,921::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:06,921::task::595::TaskManager.Task::(_updateState) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::moving from state preparing -> state aborting
Thread-22::DEBUG::2014-11-28 15:55:06,921::task::550::TaskManager.Task::(__state_aborting) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::_aborting: recover policy none
Thread-22::DEBUG::2014-11-28 15:55:06,921::task::595::TaskManager.Task::(_updateState) Task=`d3c2f1d2-5ad5-45cb-bef0-de447ea14ab1`::moving from state aborting -> state failed
Thread-22::DEBUG::2014-11-28 15:55:06,921::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:06,921::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::ERROR::2014-11-28 15:55:06,922::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-22::DEBUG::2014-11-28 15:55:06,925::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-22::DEBUG::2014-11-28 15:55:06,925::task::595::TaskManager.Task::(_updateState) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:06,925::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-22::ERROR::2014-11-28 15:55:06,926::task::866::TaskManager.Task::(_setError) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2124, in getAllTasksStatuses
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-22::DEBUG::2014-11-28 15:55:06,926::task::885::TaskManager.Task::(_run) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::Task._run: 8e71f183-e472-4a19-8471-ef1b17260950 () {} failed - stopping task
Thread-22::DEBUG::2014-11-28 15:55:06,926::task::1211::TaskManager.Task::(stop) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 15:55:06,926::task::990::TaskManager.Task::(_decref) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::ref 1 aborting True
Thread-22::INFO::2014-11-28 15:55:06,926::task::1168::TaskManager.Task::(prepare) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::aborting: Task is aborted: 'Not SPM' - code 654
Thread-22::DEBUG::2014-11-28 15:55:06,927::task::1173::TaskManager.Task::(prepare) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::Prepare: aborted: Not SPM
Thread-22::DEBUG::2014-11-28 15:55:06,927::task::990::TaskManager.Task::(_decref) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::ref 0 aborting True
Thread-22::DEBUG::2014-11-28 15:55:06,927::task::925::TaskManager.Task::(_doAbort) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::Task._doAbort: force False
Thread-22::DEBUG::2014-11-28 15:55:06,927::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:06,927::task::595::TaskManager.Task::(_updateState) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::moving from state preparing -> state aborting
Thread-22::DEBUG::2014-11-28 15:55:06,928::task::550::TaskManager.Task::(__state_aborting) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::_aborting: recover policy none
Thread-22::DEBUG::2014-11-28 15:55:06,928::task::595::TaskManager.Task::(_updateState) Task=`8e71f183-e472-4a19-8471-ef1b17260950`::moving from state aborting -> state failed
Thread-22::DEBUG::2014-11-28 15:55:06,928::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:06,928::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::ERROR::2014-11-28 15:55:06,928::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-22::DEBUG::2014-11-28 15:55:12,158::task::595::TaskManager.Task::(_updateState) Task=`f50edec8-6896-4fca-a8e7-d58c67e451d9`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:12,158::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:55:12,158::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:55:12,159::task::1185::TaskManager.Task::(prepare) Task=`f50edec8-6896-4fca-a8e7-d58c67e451d9`::finished: {}
Thread-22::DEBUG::2014-11-28 15:55:12,159::task::595::TaskManager.Task::(_updateState) Task=`f50edec8-6896-4fca-a8e7-d58c67e451d9`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:55:12,159::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:12,159::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:12,159::task::990::TaskManager.Task::(_decref) Task=`f50edec8-6896-4fca-a8e7-d58c67e451d9`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:55:27,239::task::595::TaskManager.Task::(_updateState) Task=`deed76d1-91d3-4817-8af1-c0a841fad31a`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:27,239::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:55:27,240::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:55:27,240::task::1185::TaskManager.Task::(prepare) Task=`deed76d1-91d3-4817-8af1-c0a841fad31a`::finished: {}
Thread-22::DEBUG::2014-11-28 15:55:27,240::task::595::TaskManager.Task::(_updateState) Task=`deed76d1-91d3-4817-8af1-c0a841fad31a`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:55:27,240::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:27,240::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:27,240::task::990::TaskManager.Task::(_decref) Task=`deed76d1-91d3-4817-8af1-c0a841fad31a`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:55:42,350::task::595::TaskManager.Task::(_updateState) Task=`42347d48-b4af-4219-97af-7eda8cb4201b`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:42,351::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:55:42,351::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:55:42,351::task::1185::TaskManager.Task::(prepare) Task=`42347d48-b4af-4219-97af-7eda8cb4201b`::finished: {}
Thread-22::DEBUG::2014-11-28 15:55:42,351::task::595::TaskManager.Task::(_updateState) Task=`42347d48-b4af-4219-97af-7eda8cb4201b`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:55:42,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:42,352::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:42,352::task::990::TaskManager.Task::(_decref) Task=`42347d48-b4af-4219-97af-7eda8cb4201b`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:55:57,433::task::595::TaskManager.Task::(_updateState) Task=`c32aa603-6e9c-4603-8f4e-723566002f01`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:55:57,433::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:55:57,433::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:55:57,434::task::1185::TaskManager.Task::(prepare) Task=`c32aa603-6e9c-4603-8f4e-723566002f01`::finished: {}
Thread-22::DEBUG::2014-11-28 15:55:57,434::task::595::TaskManager.Task::(_updateState) Task=`c32aa603-6e9c-4603-8f4e-723566002f01`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:55:57,434::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:55:57,434::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:55:57,434::task::990::TaskManager.Task::(_decref) Task=`c32aa603-6e9c-4603-8f4e-723566002f01`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:56:12,518::task::595::TaskManager.Task::(_updateState) Task=`2e1cb5a9-cb7b-4654-9157-e11c1c487aab`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:56:12,518::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:56:12,518::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:56:12,519::task::1185::TaskManager.Task::(prepare) Task=`2e1cb5a9-cb7b-4654-9157-e11c1c487aab`::finished: {}
Thread-22::DEBUG::2014-11-28 15:56:12,519::task::595::TaskManager.Task::(_updateState) Task=`2e1cb5a9-cb7b-4654-9157-e11c1c487aab`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:56:12,519::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:56:12,519::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:56:12,519::task::990::TaskManager.Task::(_decref) Task=`2e1cb5a9-cb7b-4654-9157-e11c1c487aab`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:56:27,629::task::595::TaskManager.Task::(_updateState) Task=`8864b454-1eb7-4f6f-a0f8-bc0a18012452`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:56:27,630::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:56:27,630::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:56:27,630::task::1185::TaskManager.Task::(prepare) Task=`8864b454-1eb7-4f6f-a0f8-bc0a18012452`::finished: {}
Thread-22::DEBUG::2014-11-28 15:56:27,630::task::595::TaskManager.Task::(_updateState) Task=`8864b454-1eb7-4f6f-a0f8-bc0a18012452`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:56:27,630::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:56:27,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:56:27,631::task::990::TaskManager.Task::(_decref) Task=`8864b454-1eb7-4f6f-a0f8-bc0a18012452`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:56:42,712::task::595::TaskManager.Task::(_updateState) Task=`b3b70423-8641-4643-8c0d-98befe5fadd7`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:56:42,712::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:56:42,713::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:56:42,713::task::1185::TaskManager.Task::(prepare) Task=`b3b70423-8641-4643-8c0d-98befe5fadd7`::finished: {}
Thread-22::DEBUG::2014-11-28 15:56:42,713::task::595::TaskManager.Task::(_updateState) Task=`b3b70423-8641-4643-8c0d-98befe5fadd7`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:56:42,713::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:56:42,713::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:56:42,713::task::990::TaskManager.Task::(_decref) Task=`b3b70423-8641-4643-8c0d-98befe5fadd7`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:56:57,792::task::595::TaskManager.Task::(_updateState) Task=`56de5f3d-1d75-4885-9273-f82ee4cf8d09`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:56:57,793::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:56:57,793::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:56:57,793::task::1185::TaskManager.Task::(prepare) Task=`56de5f3d-1d75-4885-9273-f82ee4cf8d09`::finished: {}
Thread-22::DEBUG::2014-11-28 15:56:57,793::task::595::TaskManager.Task::(_updateState) Task=`56de5f3d-1d75-4885-9273-f82ee4cf8d09`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:56:57,793::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:56:57,793::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:56:57,794::task::990::TaskManager.Task::(_decref) Task=`56de5f3d-1d75-4885-9273-f82ee4cf8d09`::ref 0 aborting False
Thread-27::ERROR::2014-11-28 15:57:10,990::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-27::ERROR::2014-11-28 15:57:10,991::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-27::DEBUG::2014-11-28 15:57:10,991::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-27::INFO::2014-11-28 15:57:10,991::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-27::DEBUG::2014-11-28 15:57:10,992::task::1185::TaskManager.Task::(prepare) Task=`799bb4dc-06de-4b3f-8055-eb058fc80823`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-27::DEBUG::2014-11-28 15:57:10,992::task::595::TaskManager.Task::(_updateState) Task=`799bb4dc-06de-4b3f-8055-eb058fc80823`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:57:10,992::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:57:10,992::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:57:10,992::task::990::TaskManager.Task::(_decref) Task=`799bb4dc-06de-4b3f-8055-eb058fc80823`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:57:11,195::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5879c05d]
Thread-27::DEBUG::2014-11-28 15:57:11,195::task::595::TaskManager.Task::(_updateState) Task=`2e826737-dd73-46aa-8a5c-da3aaa430069`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:57:11,195::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:57:11,201::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-22::DEBUG::2014-11-28 15:57:12,872::task::595::TaskManager.Task::(_updateState) Task=`7d5e2564-05bd-45f6-85e0-08e1cf71754e`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:57:12,872::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:57:12,872::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:57:12,872::task::1185::TaskManager.Task::(prepare) Task=`7d5e2564-05bd-45f6-85e0-08e1cf71754e`::finished: {}
Thread-22::DEBUG::2014-11-28 15:57:12,872::task::595::TaskManager.Task::(_updateState) Task=`7d5e2564-05bd-45f6-85e0-08e1cf71754e`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:57:12,872::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:57:12,873::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:57:12,873::task::990::TaskManager.Task::(_decref) Task=`7d5e2564-05bd-45f6-85e0-08e1cf71754e`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:57:27,955::task::595::TaskManager.Task::(_updateState) Task=`52ff776e-00e8-4f5a-86e4-02c4b1895d60`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:57:27,955::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:57:27,955::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:57:27,955::task::1185::TaskManager.Task::(prepare) Task=`52ff776e-00e8-4f5a-86e4-02c4b1895d60`::finished: {}
Thread-22::DEBUG::2014-11-28 15:57:27,956::task::595::TaskManager.Task::(_updateState) Task=`52ff776e-00e8-4f5a-86e4-02c4b1895d60`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:57:27,956::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:57:27,956::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:57:27,956::task::990::TaskManager.Task::(_decref) Task=`52ff776e-00e8-4f5a-86e4-02c4b1895d60`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:57:43,039::task::595::TaskManager.Task::(_updateState) Task=`0d57f6a6-e65f-4eba-b3e2-ad29c694a535`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:57:43,040::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:57:43,040::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:57:43,040::task::1185::TaskManager.Task::(prepare) Task=`0d57f6a6-e65f-4eba-b3e2-ad29c694a535`::finished: {}
Thread-22::DEBUG::2014-11-28 15:57:43,040::task::595::TaskManager.Task::(_updateState) Task=`0d57f6a6-e65f-4eba-b3e2-ad29c694a535`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:57:43,041::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:57:43,041::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:57:43,041::task::990::TaskManager.Task::(_decref) Task=`0d57f6a6-e65f-4eba-b3e2-ad29c694a535`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 15:57:58,127::task::595::TaskManager.Task::(_updateState) Task=`9b6be670-7736-4fc1-980c-87b49352f345`::moving from state init -> state preparing
Thread-22::INFO::2014-11-28 15:57:58,127::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-22::INFO::2014-11-28 15:57:58,128::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-22::DEBUG::2014-11-28 15:57:58,128::task::1185::TaskManager.Task::(prepare) Task=`9b6be670-7736-4fc1-980c-87b49352f345`::finished: {}
Thread-22::DEBUG::2014-11-28 15:57:58,128::task::595::TaskManager.Task::(_updateState) Task=`9b6be670-7736-4fc1-980c-87b49352f345`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 15:57:58,128::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-22::DEBUG::2014-11-28 15:57:58,128::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-22::DEBUG::2014-11-28 15:57:58,128::task::990::TaskManager.Task::(_decref) Task=`9b6be670-7736-4fc1-980c-87b49352f345`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:11,202::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x1608230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x1608230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-27::DEBUG::2014-11-28 15:58:11,202::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-27::INFO::2014-11-28 15:58:11,203::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,203::task::1185::TaskManager.Task::(prepare) Task=`2e826737-dd73-46aa-8a5c-da3aaa430069`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,203::task::595::TaskManager.Task::(_updateState) Task=`2e826737-dd73-46aa-8a5c-da3aaa430069`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:11,203::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:11,203::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:11,204::task::990::TaskManager.Task::(_decref) Task=`2e826737-dd73-46aa-8a5c-da3aaa430069`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:11,305::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [4d052b33]
Thread-27::DEBUG::2014-11-28 15:58:11,305::task::595::TaskManager.Task::(_updateState) Task=`69326e6a-951b-427d-9c08-55610a87f194`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:11,305::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:58:11,311::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-27::DEBUG::2014-11-28 15:58:11,460::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-27::DEBUG::2014-11-28 15:58:11,460::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-27::INFO::2014-11-28 15:58:11,460::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,460::task::1185::TaskManager.Task::(prepare) Task=`69326e6a-951b-427d-9c08-55610a87f194`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,461::task::595::TaskManager.Task::(_updateState) Task=`69326e6a-951b-427d-9c08-55610a87f194`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:11,461::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:11,461::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:11,461::task::990::TaskManager.Task::(_decref) Task=`69326e6a-951b-427d-9c08-55610a87f194`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:11,583::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3a10d22e]
Thread-27::DEBUG::2014-11-28 15:58:11,583::task::595::TaskManager.Task::(_updateState) Task=`e279c896-896c-43ba-a6e3-d60e63f655b3`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:11,583::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:58:11,590::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-27::DEBUG::2014-11-28 15:58:11,591::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-27::DEBUG::2014-11-28 15:58:11,591::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-27::INFO::2014-11-28 15:58:11,591::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,592::task::1185::TaskManager.Task::(prepare) Task=`e279c896-896c-43ba-a6e3-d60e63f655b3`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-27::DEBUG::2014-11-28 15:58:11,592::task::595::TaskManager.Task::(_updateState) Task=`e279c896-896c-43ba-a6e3-d60e63f655b3`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:11,592::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:11,592::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:11,592::task::990::TaskManager.Task::(_decref) Task=`e279c896-896c-43ba-a6e3-d60e63f655b3`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:13,213::task::595::TaskManager.Task::(_updateState) Task=`ee690750-0e72-43dc-a436-5cdae97248c0`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:13,213::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:58:13,214::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:58:13,214::task::1185::TaskManager.Task::(prepare) Task=`ee690750-0e72-43dc-a436-5cdae97248c0`::finished: {}
Thread-27::DEBUG::2014-11-28 15:58:13,214::task::595::TaskManager.Task::(_updateState) Task=`ee690750-0e72-43dc-a436-5cdae97248c0`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:13,214::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:13,214::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:13,214::task::990::TaskManager.Task::(_decref) Task=`ee690750-0e72-43dc-a436-5cdae97248c0`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:28,305::task::595::TaskManager.Task::(_updateState) Task=`6ba52283-73d8-4f6d-a78a-c92bb59734cc`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:28,306::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:58:28,306::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:58:28,306::task::1185::TaskManager.Task::(prepare) Task=`6ba52283-73d8-4f6d-a78a-c92bb59734cc`::finished: {}
Thread-27::DEBUG::2014-11-28 15:58:28,306::task::595::TaskManager.Task::(_updateState) Task=`6ba52283-73d8-4f6d-a78a-c92bb59734cc`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:28,306::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:28,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:28,307::task::990::TaskManager.Task::(_decref) Task=`6ba52283-73d8-4f6d-a78a-c92bb59734cc`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:43,398::task::595::TaskManager.Task::(_updateState) Task=`84f1a6a6-11af-4fb5-8ec8-f922fc046654`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:43,398::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:58:43,398::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:58:43,398::task::1185::TaskManager.Task::(prepare) Task=`84f1a6a6-11af-4fb5-8ec8-f922fc046654`::finished: {}
Thread-27::DEBUG::2014-11-28 15:58:43,398::task::595::TaskManager.Task::(_updateState) Task=`84f1a6a6-11af-4fb5-8ec8-f922fc046654`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:43,399::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:43,399::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:43,399::task::990::TaskManager.Task::(_decref) Task=`84f1a6a6-11af-4fb5-8ec8-f922fc046654`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:58:58,479::task::595::TaskManager.Task::(_updateState) Task=`c7e06bff-27dd-45cf-bd4a-bec34bf839d9`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:58:58,479::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:58:58,479::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:58:58,480::task::1185::TaskManager.Task::(prepare) Task=`c7e06bff-27dd-45cf-bd4a-bec34bf839d9`::finished: {}
Thread-27::DEBUG::2014-11-28 15:58:58,480::task::595::TaskManager.Task::(_updateState) Task=`c7e06bff-27dd-45cf-bd4a-bec34bf839d9`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:58:58,480::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:58:58,480::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:58:58,480::task::990::TaskManager.Task::(_decref) Task=`c7e06bff-27dd-45cf-bd4a-bec34bf839d9`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:59:10,553::task::595::TaskManager.Task::(_updateState) Task=`781b27ab-5ba7-477e-a6a7-c78c412764e9`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:59:10,553::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-27::INFO::2014-11-28 15:59:10,553::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-27::DEBUG::2014-11-28 15:59:10,554::task::1185::TaskManager.Task::(prepare) Task=`781b27ab-5ba7-477e-a6a7-c78c412764e9`::finished: {}
Thread-27::DEBUG::2014-11-28 15:59:10,554::task::595::TaskManager.Task::(_updateState) Task=`781b27ab-5ba7-477e-a6a7-c78c412764e9`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:59:10,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:59:10,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:59:10,554::task::990::TaskManager.Task::(_decref) Task=`781b27ab-5ba7-477e-a6a7-c78c412764e9`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:59:10,586::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-27::DEBUG::2014-11-28 15:59:10,586::task::595::TaskManager.Task::(_updateState) Task=`dc77628c-0179-4c7f-b79f-9914a743553f`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:59:10,586::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, remove=False, options=None)
Thread-27::WARNING::2014-11-28 15:59:10,586::hsm::1126::Storage.HSM::(disconnectStoragePool) disconnect sp: 45995288-a7d6-4cbf-8b2f-63fa35642bad failed. Known pools {}
Thread-27::INFO::2014-11-28 15:59:10,587::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
Thread-27::DEBUG::2014-11-28 15:59:10,587::task::1185::TaskManager.Task::(prepare) Task=`dc77628c-0179-4c7f-b79f-9914a743553f`::finished: None
Thread-27::DEBUG::2014-11-28 15:59:10,587::task::595::TaskManager.Task::(_updateState) Task=`dc77628c-0179-4c7f-b79f-9914a743553f`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:59:10,587::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:59:10,587::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:59:10,587::task::990::TaskManager.Task::(_decref) Task=`dc77628c-0179-4c7f-b79f-9914a743553f`::ref 0 aborting False
Thread-27::DEBUG::2014-11-28 15:59:10,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [67ed792d]
Thread-27::DEBUG::2014-11-28 15:59:10,680::task::595::TaskManager.Task::(_updateState) Task=`821428ad-20a4-457f-af6a-684d9f70eb85`::moving from state init -> state preparing
Thread-27::INFO::2014-11-28 15:59:10,680::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-27::DEBUG::2014-11-28 15:59:10,681::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-27::ERROR::2014-11-28 15:59:10,695::hsm::2487::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2483, in disconnectStorageServer
    conObj.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
    return self._mountCon.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
    self._mount.umount(True, True)
  File "/usr/share/vdsm/storage/mount.py", line 229, in umount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';umount: /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso: mountpoint not found\n')
Thread-27::DEBUG::2014-11-28 15:59:10,696::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-27::DEBUG::2014-11-28 15:59:10,721::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-27::DEBUG::2014-11-28 15:59:10,745::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-27::ERROR::2014-11-28 15:59:10,756::hsm::2487::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2483, in disconnectStorageServer
    conObj.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
    return self._mountCon.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
    self._mount.umount(True, True)
  File "/usr/share/vdsm/storage/mount.py", line 229, in umount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';umount: /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01: mountpoint not found\n')
Thread-27::DEBUG::2014-11-28 15:59:10,757::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-27::DEBUG::2014-11-28 15:59:10,771::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-27::DEBUG::2014-11-28 15:59:10,771::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:59:10,771::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-27::DEBUG::2014-11-28 15:59:10,771::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-27::DEBUG::2014-11-28 15:59:10,772::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-27::DEBUG::2014-11-28 15:59:10,772::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-27::DEBUG::2014-11-28 15:59:10,784::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::DEBUG::2014-11-28 15:59:10,785::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-27::DEBUG::2014-11-28 15:59:10,814::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-27::DEBUG::2014-11-28 15:59:10,814::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,814::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,815::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,815::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,815::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,815::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 15:59:10,816::misc::748::SamplingMethod::(__call__) Returning last result
Thread-27::INFO::2014-11-28 15:59:10,816::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-27::DEBUG::2014-11-28 15:59:10,816::task::1185::TaskManager.Task::(prepare) Task=`821428ad-20a4-457f-af6a-684d9f70eb85`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-27::DEBUG::2014-11-28 15:59:10,816::task::595::TaskManager.Task::(_updateState) Task=`821428ad-20a4-457f-af6a-684d9f70eb85`::moving from state preparing -> state finished
Thread-27::DEBUG::2014-11-28 15:59:10,817::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2014-11-28 15:59:10,817::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2014-11-28 15:59:10,817::task::990::TaskManager.Task::(_decref) Task=`821428ad-20a4-457f-af6a-684d9f70eb85`::ref 0 aborting False
MainThread::DEBUG::2014-11-28 16:06:13,589::vdsm::58::vds::(sigtermHandler) Received signal 15
MainThread::INFO::2014-11-28 16:06:14,405::vmChannels::187::vds::(stop) VM channels listener was stopped.
MainThread::INFO::2014-11-28 16:06:14,406::momIF::91::MOM::(stop) Shutting down MOM
MainThread::DEBUG::2014-11-28 16:06:14,406::task::595::TaskManager.Task::(_updateState) Task=`e56acb0c-8f26-484d-8dca-64f680bab421`::moving from state init -> state preparing
MainThread::INFO::2014-11-28 16:06:14,407::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None)
Thread-11::DEBUG::2014-11-28 16:06:14,407::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped
VM Channels Listener::INFO::2014-11-28 16:06:14,788::vmChannels::182::vds::(run) VM channels listener thread has ended.
MainThread::INFO::2014-11-28 16:06:14,829::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors
MainThread::DEBUG::2014-11-28 16:06:14,829::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks
MainThread::INFO::2014-11-28 16:06:14,829::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None
MainThread::DEBUG::2014-11-28 16:06:14,829::task::1185::TaskManager.Task::(prepare) Task=`e56acb0c-8f26-484d-8dca-64f680bab421`::finished: None
MainThread::DEBUG::2014-11-28 16:06:14,829::task::595::TaskManager.Task::(_updateState) Task=`e56acb0c-8f26-484d-8dca-64f680bab421`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 16:06:14,830::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 16:06:14,830::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 16:06:14,830::task::990::TaskManager.Task::(_decref) Task=`e56acb0c-8f26-484d-8dca-64f680bab421`::ref 0 aborting False
MainThread::INFO::2014-11-28 16:06:14,830::vdsm::137::vds::(run) VDSM main thread ended. Waiting for 15 other threads...
MainThread::INFO::2014-11-28 16:06:14,830::vdsm::141::vds::(run) <Thread(libvirtEventLoop, started daemon 140473365440256)>
MainThread::INFO::2014-11-28 16:06:14,830::vdsm::141::vds::(run) <WorkerThread(Thread-2, started daemon 140473280755456)>
MainThread::INFO::2014-11-28 16:06:14,830::vdsm::141::vds::(run) <WorkerThread(Thread-8, started daemon 140473230399232)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-9, started daemon 140472676775680)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-7, started daemon 140473238791936)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-6, started daemon 140473247184640)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-5, started daemon 140473255577344)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <HostMonitor(HostMonitor, started daemon 140472114599680)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-10, started daemon 140472668382976)>
MainThread::INFO::2014-11-28 16:06:14,831::vdsm::141::vds::(run) <WorkerThread(Thread-1, started daemon 140473357047552)>
MainThread::INFO::2014-11-28 16:06:14,832::momIF::91::MOM::(stop) Shutting down MOM
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <MomThread(MOM, started 140472634812160)>
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <_MainThread(MainThread, started 140473653024576)>
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <GuestManager(GuestManager, started daemon 140472106206976)>
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <PolicyEngine(PolicyEngine, started daemon 140472651597568)>
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <WorkerThread(Thread-4, started daemon 140473263970048)>
MainThread::INFO::2014-11-28 16:06:14,832::vdsm::141::vds::(run) <WorkerThread(Thread-3, started daemon 140473272362752)>
MainThread::INFO::2014-11-28 16:08:02,311::vdsm::131::vds::(run) (PID: 2413) I am the actual vdsm 4.14.17-0.el7 cti11.ctisandbox.cfcc.edu (3.10.0-123.9.3.el7.x86_64)
MainThread::DEBUG::2014-11-28 16:08:02,325::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-11-28 16:08:02,325::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::WARNING::2014-11-28 16:08:02,328::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
MainThread::DEBUG::2014-11-28 16:08:02,792::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-11-28 16:08:02,825::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
MainThread::DEBUG::2014-11-28 16:08:02,825::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-11-28 16:08:02,850::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 16:08:02,851::hsm::458::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-11-28 16:08:02,856::hsm::490::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-11-28 16:08:02,856::hsm::491::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-11-28 16:08:02,857::hsm::493::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-11-28 16:08:02,858::hsm::536::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-11-28 16:08:02,859::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
MainThread::INFO::2014-11-28 16:08:02,860::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-11-28 16:08:02,860::misc::740::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-11-28 16:08:03,118::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
storageRefresh::DEBUG::2014-11-28 16:08:03,136::misc::740::SamplingMethod::(__call__) Got in to sampling method
MainThread::DEBUG::2014-11-28 16:08:03,136::task::595::TaskManager.Task::(_updateState) Task=`dc89d68e-4f26-4244-afda-d16a3e4e66ef`::moving from state init -> state preparing
storageRefresh::DEBUG::2014-11-28 16:08:03,137::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
MainThread::INFO::2014-11-28 16:08:03,137::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x14784d0>>)
storageRefresh::DEBUG::2014-11-28 16:08:03,137::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
MainThread::INFO::2014-11-28 16:08:03,137::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
MainThread::DEBUG::2014-11-28 16:08:03,138::task::1185::TaskManager.Task::(prepare) Task=`dc89d68e-4f26-4244-afda-d16a3e4e66ef`::finished: None
MainThread::DEBUG::2014-11-28 16:08:03,138::task::595::TaskManager.Task::(_updateState) Task=`dc89d68e-4f26-4244-afda-d16a3e4e66ef`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 16:08:03,141::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 16:08:03,141::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 16:08:03,141::task::990::TaskManager.Task::(_decref) Task=`dc89d68e-4f26-4244-afda-d16a3e4e66ef`::ref 0 aborting False
MainThread::INFO::2014-11-28 16:08:03,142::momIF::49::MOM::(__init__) Starting up MOM
MainThread::INFO::2014-11-28 16:08:03,156::vmChannels::191::vds::(settimeout) Setting channels' timeout to 30 seconds.
VM Channels Listener::INFO::2014-11-28 16:08:03,157::vmChannels::174::vds::(run) Starting VM channels listener thread.
clientIFinit::DEBUG::2014-11-28 16:08:03,192::libvirtconnection::145::root::(get) trying to connect libvirt
storageRefresh::DEBUG::2014-11-28 16:08:03,203::misc::748::SamplingMethod::(__call__) Returning last result
MainThread::WARNING::2014-11-28 16:08:03,205::clientIF::180::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-11-28 16:08:03,209::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:03,268::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:03,269::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,270::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,271::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,272::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,273::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,274::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,276::misc::748::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-11-28 16:08:03,279::lvm::325::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,290::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:03,316::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:03,317::lvm::350::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,319::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,321::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:03,341::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:03,342::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,344::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:03,363::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:03,364::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:03,367::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:03,386::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:03,387::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::WARNING::2014-11-28 16:08:03,387::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
storageRefresh::DEBUG::2014-11-28 16:08:03,388::hsm::393::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-11-28 16:09:56,549::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [38a4b235]
Thread-13::DEBUG::2014-11-28 16:09:56,588::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 16:09:56,595::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:09:56,846::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:12:d6', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '2a:5d:1f:fa:cd:68', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.994', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 16:09:56,864::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [38a4b235]
Thread-13::DEBUG::2014-11-28 16:09:56,951::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0046-4710-8038-CAC04F325031', 'systemSerialNumber': 'JFG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 16:09:57,141::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2f5c8ffe]
Thread-13::DEBUG::2014-11-28 16:09:57,142::task::595::TaskManager.Task::(_updateState) Task=`b5a156a7-7cc5-4fec-b26c-ca117a813c1c`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:09:57,142::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:09:57,148::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::ERROR::2014-11-28 16:09:57,265::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::ERROR::2014-11-28 16:09:57,311::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::DEBUG::2014-11-28 16:09:57,317::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {}
Thread-13::DEBUG::2014-11-28 16:09:57,325::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.2:/data/bs02store /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-13::DEBUG::2014-11-28 16:09:57,519::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 16:09:57,522::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 16:09:57,522::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:09:57,528::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.99:/mnt/storage /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 16:09:57,564::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-13::DEBUG::2014-11-28 16:09:57,565::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 16:09:57,565::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:09:57,572::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-13::ERROR::2014-11-28 16:12:08,688::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 16:12:08,689::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 16:12:08,690::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:12:08,697::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 16:12:08,735::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-13::DEBUG::2014-11-28 16:12:08,736::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
Thread-13::DEBUG::2014-11-28 16:12:08,736::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:12:08,736::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:12:08,736::task::1185::TaskManager.Task::(prepare) Task=`b5a156a7-7cc5-4fec-b26c-ca117a813c1c`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:12:08,737::task::595::TaskManager.Task::(_updateState) Task=`b5a156a7-7cc5-4fec-b26c-ca117a813c1c`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:12:08,737::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:12:08,737::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:12:08,737::task::990::TaskManager.Task::(_decref) Task=`b5a156a7-7cc5-4fec-b26c-ca117a813c1c`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:12:08,795::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 16:12:08,795::task::595::TaskManager.Task::(_updateState) Task=`94c46d96-cd43-4e1f-abec-01cea2a2e63f`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:12:08,795::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=4, msdUUID='e7eaae86-11ea-4c08-8c11-8b8c54373389', masterVersion=24241, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 16:12:08,796::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`b8f0f397-f302-4828-943c-b4fbeb234ae0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:12:08,796::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:12:08,796::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:12:08,796::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`b8f0f397-f302-4828-943c-b4fbeb234ae0`::Granted request
Thread-13::DEBUG::2014-11-28 16:12:08,797::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`359b7584-b831-41e0-881e-c2e54a6ed219`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:12:08,797::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:12:08,798::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:12:08,798::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`359b7584-b831-41e0-881e-c2e54a6ed219`::Granted request
Thread-13::INFO::2014-11-28 16:12:08,798::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7ff3c8074f00 to type StoragePoolDiskBackend instance 0x7ff3700a1290
Thread-13::INFO::2014-11-28 16:12:08,798::sp::623::Storage.StoragePool::(connect) Connect host #4 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: e7eaae86-11ea-4c08-8c11-8b8c54373389 (ver = 24241)
Thread-13::DEBUG::2014-11-28 16:12:08,799::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,799::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,799::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,799::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,800::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,800::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,800::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 16:12:08,800::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:12:08,800::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 16:12:08,801::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:12:08,801::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 16:12:08,801::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 16:12:08,813::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:12:08,815::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 16:12:08,864::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:12:08,865::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,865::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,865::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,865::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,866::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,866::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:12:08,866::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:12:09,027::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:12:09,174::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 16:12:09,177::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24241', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=9c3452c6e6a060519e38f2de3fc04201d2386f6d']
Thread-13::DEBUG::2014-11-28 16:12:09,180::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::DEBUG::2014-11-28 16:12:09,180::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_imageNS'
Thread-13::DEBUG::2014-11-28 16:12:09,180::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_volumeNS'
Thread-13::DEBUG::2014-11-28 16:12:09,181::sp::1375::Storage.StoragePool::(setMasterDomain) Master domain e7eaae86-11ea-4c08-8c11-8b8c54373389 verified, version 24241
Thread-13::DEBUG::2014-11-28 16:12:09,181::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
Thread-13::DEBUG::2014-11-28 16:12:09,181::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:12:09,184::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24241', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=9c3452c6e6a060519e38f2de3fc04201d2386f6d']
Thread-13::INFO::2014-11-28 16:12:09,184::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:12:09,185::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-13::DEBUG::2014-11-28 16:12:09,185::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `1c4abda0-906b-4faf-a2e3-b6683729cc2f`
Thread-17::ERROR::2014-11-28 16:12:09,185::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-13::INFO::2014-11-28 16:12:09,186::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-17::ERROR::2014-11-28 16:12:09,186::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:12:09,187::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:12:09,188::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::DEBUG::2014-11-28 16:12:09,188::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `54b2d645-fc18-4784-99eb-0e17e296eee7`
Thread-18::ERROR::2014-11-28 16:12:09,189::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:12:09,189::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::INFO::2014-11-28 16:12:09,189::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-18::DEBUG::2014-11-28 16:12:09,191::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-19::DEBUG::2014-11-28 16:12:09,192::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-13::DEBUG::2014-11-28 16:12:09,192::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `644b4e6d-aa9e-4628-b56c-7fb08ddc9584`
Thread-13::INFO::2014-11-28 16:12:09,195::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:12:09,198::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `e7eaae86-11ea-4c08-8c11-8b8c54373389`
Thread-22::DEBUG::2014-11-28 16:12:09,199::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::INFO::2014-11-28 16:12:09,200::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-13::DEBUG::2014-11-28 16:12:09,209::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `474f0809-26c6-4fca-ab8c-72a2e6dcd912`
Thread-24::DEBUG::2014-11-28 16:12:09,210::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-13::DEBUG::2014-11-28 16:12:09,211::misc::748::SamplingMethod::(__call__) Returning last result
Thread-18::DEBUG::2014-11-28 16:12:09,212::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-17::DEBUG::2014-11-28 16:12:09,213::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-24::ERROR::2014-11-28 16:12:09,213::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::ERROR::2014-11-28 16:12:09,225::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:12:09,228::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-24::DEBUG::2014-11-28 16:12:09,234::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 474f0809-26c6-4fca-ab8c-72a2e6dcd912' (cwd None)
Thread-18::DEBUG::2014-11-28 16:12:09,248::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:12:09,250::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-17::DEBUG::2014-11-28 16:12:09,254::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-13::DEBUG::2014-11-28 16:12:09,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24241', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=9c3452c6e6a060519e38f2de3fc04201d2386f6d']
Thread-13::DEBUG::2014-11-28 16:12:09,264::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-17::WARNING::2014-11-28 16:12:09,265::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-24::DEBUG::2014-11-28 16:12:09,268::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found\n  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912\n'; <rc> = 5
Thread-24::WARNING::2014-11-28 16:12:09,281::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found', '  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912']
Thread-13::DEBUG::2014-11-28 16:12:09,284::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f 54b2d645-fc18-4784-99eb-0e17e296eee7 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 474f0809-26c6-4fca-ab8c-72a2e6dcd912' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:09,295::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-11-28 16:12:09,313::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n  Volume group "644b4e6d-aa9e-4628-b56c-7fb08ddc9584" not found\n  Skipping volume group 644b4e6d-aa9e-4628-b56c-7fb08ddc9584\n  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found\n  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912\n'; <rc> = 5
Thread-13::WARNING::2014-11-28 16:12:09,315::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7', '  Volume group "644b4e6d-aa9e-4628-b56c-7fb08ddc9584" not found', '  Skipping volume group 644b4e6d-aa9e-4628-b56c-7fb08ddc9584', '  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found', '  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912']
Thread-13::DEBUG::2014-11-28 16:12:09,315::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:12:09,323::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.0169933 s, 37.6 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:12:09,363::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e7eaae86-11ea-4c08-8c11-8b8c54373389 changed its status to Valid
Thread-22::INFO::2014-11-28 16:12:09,375::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 (id: 4)
Thread-22::DEBUG::2014-11-28 16:12:09,376::clusterlock::202::SANLock::(acquireHostId) Host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 successfully acquired (id: 4)
Thread-37::DEBUG::2014-11-28 16:12:09,385::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-37::DEBUG::2014-11-28 16:12:09,385::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-37::DEBUG::2014-11-28 16:12:09,405::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-37::DEBUG::2014-11-28 16:12:09,420::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-39::INFO::2014-11-28 16:12:09,433::clientIF::125::vds::(contEIOVms) vmContainerLock acquired
Thread-22::DEBUG::2014-11-28 16:12:19,385::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:19,392::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n640 bytes (640 B) copied, 0.000260614 s, 2.5 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:12:29,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:29,409::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000370801 s, 1.7 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:12:39,420::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:39,426::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000280762 s, 2.3 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:12:49,438::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:49,444::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000246532 s, 2.6 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:12:59,455::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:12:59,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000409839 s, 1.6 MB/s\n'; <rc> = 0
Thread-15::WARNING::2014-11-28 16:13:08,883::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-15::DEBUG::2014-11-28 16:13:08,909::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-21::WARNING::2014-11-28 16:13:09,211::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-21::DEBUG::2014-11-28 16:13:09,215::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:13:09,220::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:13:09,231::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::DEBUG::2014-11-28 16:13:09,236::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 changed its status to Invalid
Thread-40::DEBUG::2014-11-28 16:13:09,240::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-40::DEBUG::2014-11-28 16:13:09,243::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-40::DEBUG::2014-11-28 16:13:09,249::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-40::DEBUG::2014-11-28 16:13:09,254::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-26::DEBUG::2014-11-28 16:13:09,287::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-26::WARNING::2014-11-28 16:13:09,290::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-26::DEBUG::2014-11-28 16:13:09,295::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:13:09,298::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:13:09,301::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::DEBUG::2014-11-28 16:13:09,304::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 54b2d645-fc18-4784-99eb-0e17e296eee7 changed its status to Invalid
Thread-43::DEBUG::2014-11-28 16:13:09,307::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-43::DEBUG::2014-11-28 16:13:09,308::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-43::DEBUG::2014-11-28 16:13:09,311::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-43::DEBUG::2014-11-28 16:13:09,314::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-31::WARNING::2014-11-28 16:13:09,317::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-24::ERROR::2014-11-28 16:13:09,319::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-24::ERROR::2014-11-28 16:13:09,320::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-24::DEBUG::2014-11-28 16:13:09,322::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 changed its status to Invalid
Thread-46::DEBUG::2014-11-28 16:13:09,324::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-46::DEBUG::2014-11-28 16:13:09,326::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-46::DEBUG::2014-11-28 16:13:09,328::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-46::DEBUG::2014-11-28 16:13:09,331::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-29::WARNING::2014-11-28 16:13:09,331::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-29::DEBUG::2014-11-28 16:13:09,333::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-17::ERROR::2014-11-28 16:13:09,335::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:13:09,335::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::DEBUG::2014-11-28 16:13:09,336::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f changed its status to Invalid
Thread-49::DEBUG::2014-11-28 16:13:09,337::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-49::DEBUG::2014-11-28 16:13:09,338::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-49::DEBUG::2014-11-28 16:13:09,339::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-49::DEBUG::2014-11-28 16:13:09,340::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-34::WARNING::2014-11-28 16:13:09,363::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::INFO::2014-11-28 16:13:09,363::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::INFO::2014-11-28 16:13:09,380::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd
Thread-13::DEBUG::2014-11-28 16:13:09,402::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0
Thread-13::DEBUG::2014-11-28 16:13:09,404::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd if=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=32' (cwd None)
Thread-13::DEBUG::2014-11-28 16:13:09,417::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.00685902 s, 597 kB/s\n'; <rc> = 0
Thread-13::INFO::2014-11-28 16:13:09,417::storage_mailbox::385::Storage.MailBox.HsmMailMonitor::(_sendMail) HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=512', 'seek=32']
Thread-13::DEBUG::2014-11-28 16:13:09,418::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd of=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/inbox iflag=fullblock oflag=direct conv=notrunc bs=512 seek=32' (cwd None)
Thread-13::DEBUG::2014-11-28 16:13:09,462::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.0394537 s, 104 kB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:13:09,463::storage_mailbox::207::Storage.Mailbox.HSM::(__init__) HSM_MailboxMonitor created for pool 45995288-a7d6-4cbf-8b2f-63fa35642bad
Thread-13::DEBUG::2014-11-28 16:13:09,463::sp::465::Storage.StoragePool::(__createMailboxMonitor) HSM mailbox ready for pool 45995288-a7d6-4cbf-8b2f-63fa35642bad on master domain e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:13:09,464::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 16:13:09,464::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 16:13:09,464::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:13:09,464::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 16:13:09,465::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-11-28 16:13:09,466::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 16:13:09,467::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:13:09,468::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::INFO::2014-11-28 16:13:09,469::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-13::DEBUG::2014-11-28 16:13:09,469::task::1185::TaskManager.Task::(prepare) Task=`94c46d96-cd43-4e1f-abec-01cea2a2e63f`::finished: True
Thread-13::DEBUG::2014-11-28 16:13:09,470::task::595::TaskManager.Task::(_updateState) Task=`94c46d96-cd43-4e1f-abec-01cea2a2e63f`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:09,471::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:09,471::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:09,472::task::990::TaskManager.Task::(_decref) Task=`94c46d96-cd43-4e1f-abec-01cea2a2e63f`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:13:09,480::task::595::TaskManager.Task::(_updateState) Task=`5263e9f9-cd54-49d7-8bf9-3062da72d775`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:13:09,481::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:13:09,482::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000409839', 'lastCheck': '10.0', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}}
Thread-22::DEBUG::2014-11-28 16:13:09,482::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-11-28 16:13:09,483::task::1185::TaskManager.Task::(prepare) Task=`5263e9f9-cd54-49d7-8bf9-3062da72d775`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000409839', 'lastCheck': '10.0', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.2', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:09,483::task::595::TaskManager.Task::(_updateState) Task=`5263e9f9-cd54-49d7-8bf9-3062da72d775`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:09,489::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:09,490::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:09,491::task::990::TaskManager.Task::(_decref) Task=`5263e9f9-cd54-49d7-8bf9-3062da72d775`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:13:09,491::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000267668 s, 2.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:13:09,510::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': False},) {} flowID [2f5c8ffe]
Thread-13::DEBUG::2014-11-28 16:13:09,580::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 16:13:12,613::task::595::TaskManager.Task::(_updateState) Task=`be2e8805-167a-4b84-a926-de962cd9924a`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:13:12,613::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:13:12,614::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000267668', 'lastCheck': '3.1', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:12,614::task::1185::TaskManager.Task::(prepare) Task=`be2e8805-167a-4b84-a926-de962cd9924a`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000267668', 'lastCheck': '3.1', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:12,614::task::595::TaskManager.Task::(_updateState) Task=`be2e8805-167a-4b84-a926-de962cd9924a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:12,614::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:12,614::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:12,615::task::990::TaskManager.Task::(_decref) Task=`be2e8805-167a-4b84-a926-de962cd9924a`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 16:13:19,307::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:13:19,307::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:13:19,308::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:13:19,308::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-24::ERROR::2014-11-28 16:13:19,325::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::ERROR::2014-11-28 16:13:19,325::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:13:19,325::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-24::DEBUG::2014-11-28 16:13:19,326::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 474f0809-26c6-4fca-ab8c-72a2e6dcd912' (cwd None)
Thread-18::DEBUG::2014-11-28 16:13:19,332::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:13:19,333::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-17::ERROR::2014-11-28 16:13:19,341::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:13:19,341::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:13:19,342::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-17::DEBUG::2014-11-28 16:13:19,343::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-24::DEBUG::2014-11-28 16:13:19,350::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found\n  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912\n'; <rc> = 5
Thread-24::WARNING::2014-11-28 16:13:19,351::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found', '  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912']
Thread-17::DEBUG::2014-11-28 16:13:19,365::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:13:19,366::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:13:19,367::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:13:19,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:13:19,544::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.010692 s, 59.8 kB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:13:27,720::task::595::TaskManager.Task::(_updateState) Task=`7f803464-1ec6-454e-b285-11f8d6fbfc06`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:13:27,721::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:13:27,721::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.010692', 'lastCheck': '8.2', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:27,721::task::1185::TaskManager.Task::(prepare) Task=`7f803464-1ec6-454e-b285-11f8d6fbfc06`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.010692', 'lastCheck': '8.2', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.4', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:27,722::task::595::TaskManager.Task::(_updateState) Task=`7f803464-1ec6-454e-b285-11f8d6fbfc06`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:27,722::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:27,722::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:27,722::task::990::TaskManager.Task::(_decref) Task=`7f803464-1ec6-454e-b285-11f8d6fbfc06`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:13:29,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:13:29,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000286652 s, 2.2 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:13:39,577::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:13:39,584::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000374523 s, 1.7 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:13:42,795::task::595::TaskManager.Task::(_updateState) Task=`75d93501-452c-4ef1-b8b3-c150dbd8292c`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:13:42,795::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:13:42,796::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000374523', 'lastCheck': '3.2', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:42,796::task::1185::TaskManager.Task::(prepare) Task=`75d93501-452c-4ef1-b8b3-c150dbd8292c`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000374523', 'lastCheck': '3.2', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.5', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:42,796::task::595::TaskManager.Task::(_updateState) Task=`75d93501-452c-4ef1-b8b3-c150dbd8292c`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:42,796::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:42,797::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:42,797::task::990::TaskManager.Task::(_decref) Task=`75d93501-452c-4ef1-b8b3-c150dbd8292c`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:13:49,596::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:13:49,602::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000229414 s, 2.8 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:13:57,873::task::595::TaskManager.Task::(_updateState) Task=`34bf554e-1bee-4458-9c59-5959a6d9afad`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:13:57,873::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:13:57,874::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000229414', 'lastCheck': '8.3', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:57,874::task::1185::TaskManager.Task::(prepare) Task=`34bf554e-1bee-4458-9c59-5959a6d9afad`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000229414', 'lastCheck': '8.3', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:13:57,874::task::595::TaskManager.Task::(_updateState) Task=`34bf554e-1bee-4458-9c59-5959a6d9afad`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:13:57,874::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:13:57,874::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:13:57,875::task::990::TaskManager.Task::(_decref) Task=`34bf554e-1bee-4458-9c59-5959a6d9afad`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:13:59,614::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:13:59,620::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000259541 s, 2.5 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:14:09,632::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:14:09,639::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000317745 s, 2.0 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:14:12,951::task::595::TaskManager.Task::(_updateState) Task=`c2d88d26-4da5-4635-9676-bdd76d217860`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:12,952::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:14:12,952::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.7', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000317745', 'lastCheck': '3.3', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:12,952::task::1185::TaskManager.Task::(prepare) Task=`c2d88d26-4da5-4635-9676-bdd76d217860`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.7', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000317745', 'lastCheck': '3.3', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:12,952::task::595::TaskManager.Task::(_updateState) Task=`c2d88d26-4da5-4635-9676-bdd76d217860`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:14:12,953::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:14:12,953::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:14:12,953::task::990::TaskManager.Task::(_decref) Task=`c2d88d26-4da5-4635-9676-bdd76d217860`::ref 0 aborting False
Thread-59::WARNING::2014-11-28 16:14:19,250::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-60::WARNING::2014-11-28 16:14:19,253::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-19::ERROR::2014-11-28 16:14:19,265::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:14:19,268::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-62::WARNING::2014-11-28 16:14:19,354::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-62::DEBUG::2014-11-28 16:14:19,361::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-63::WARNING::2014-11-28 16:14:19,369::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-63::DEBUG::2014-11-28 16:14:19,375::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:14:19,379::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:14:19,382::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-66::WARNING::2014-11-28 16:14:19,394::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-66::DEBUG::2014-11-28 16:14:19,399::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-67::WARNING::2014-11-28 16:14:19,404::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-67::DEBUG::2014-11-28 16:14:19,407::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-69::WARNING::2014-11-28 16:14:19,454::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-24::ERROR::2014-11-28 16:14:19,456::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-24::ERROR::2014-11-28 16:14:19,457::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-68::WARNING::2014-11-28 16:14:19,501::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:14:19,501::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:14:19,502::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-22::DEBUG::2014-11-28 16:14:19,649::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:14:19,655::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000391554 s, 1.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:14:28,035::task::595::TaskManager.Task::(_updateState) Task=`fa2e88c9-48e6-4cf1-9ee1-68557d26a021`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:28,035::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:14:28,036::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000391554', 'lastCheck': '8.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:28,036::task::1185::TaskManager.Task::(prepare) Task=`fa2e88c9-48e6-4cf1-9ee1-68557d26a021`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000391554', 'lastCheck': '8.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.6', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:28,036::task::595::TaskManager.Task::(_updateState) Task=`fa2e88c9-48e6-4cf1-9ee1-68557d26a021`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:14:28,036::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:14:28,037::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:14:28,037::task::990::TaskManager.Task::(_decref) Task=`fa2e88c9-48e6-4cf1-9ee1-68557d26a021`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 16:14:29,386::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:14:29,386::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:14:29,387::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:14:29,387::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:14:29,407::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:14:29,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:14:29,408::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::ERROR::2014-11-28 16:14:29,459::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::ERROR::2014-11-28 16:14:29,459::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:14:29,459::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-24::DEBUG::2014-11-28 16:14:29,460::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 474f0809-26c6-4fca-ab8c-72a2e6dcd912' (cwd None)
Thread-24::DEBUG::2014-11-28 16:14:29,480::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found\n  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912\n'; <rc> = 5
Thread-24::WARNING::2014-11-28 16:14:29,481::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found', '  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912']
Thread-24::DEBUG::2014-11-28 16:14:29,481::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:14:29,502::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:14:29,503::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:14:29,503::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:14:29,504::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:14:29,565::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:14:29,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:14:29,567::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:14:29,666::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:14:29,678::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.00537373 s, 119 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:14:39,689::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:14:39,696::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000364866 s, 1.8 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:14:43,108::task::595::TaskManager.Task::(_updateState) Task=`ce3c1e98-e6e3-4963-8592-2493d94736e0`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:43,108::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:14:43,109::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364866', 'lastCheck': '3.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.7', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:43,109::task::1185::TaskManager.Task::(prepare) Task=`ce3c1e98-e6e3-4963-8592-2493d94736e0`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000364866', 'lastCheck': '3.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.7', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:43,109::task::595::TaskManager.Task::(_updateState) Task=`ce3c1e98-e6e3-4963-8592-2493d94736e0`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:14:43,109::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:14:43,110::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:14:43,110::task::990::TaskManager.Task::(_decref) Task=`ce3c1e98-e6e3-4963-8592-2493d94736e0`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:14:49,708::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:14:49,714::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n639 bytes (639 B) copied, 0.000244167 s, 2.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:14:58,181::task::595::TaskManager.Task::(_updateState) Task=`135c026c-1c8a-4939-8d0c-39c2eb88c1cb`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:58,181::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:14:58,182::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '8.5', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:58,182::task::1185::TaskManager.Task::(prepare) Task=`135c026c-1c8a-4939-8d0c-39c2eb88c1cb`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '8.5', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}}
Thread-13::DEBUG::2014-11-28 16:14:58,182::task::595::TaskManager.Task::(_updateState) Task=`135c026c-1c8a-4939-8d0c-39c2eb88c1cb`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:14:58,182::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:14:58,183::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:14:58,183::task::990::TaskManager.Task::(_decref) Task=`135c026c-1c8a-4939-8d0c-39c2eb88c1cb`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:14:59,675::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [351f98fe]
Thread-13::DEBUG::2014-11-28 16:14:59,676::task::595::TaskManager.Task::(_updateState) Task=`a15d37af-f57e-4e6e-b127-de1c226f4e74`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:59,676::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:14:59,683::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::DEBUG::2014-11-28 16:14:59,737::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso
Thread-13::DEBUG::2014-11-28 16:14:59,739::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-13::DEBUG::2014-11-28 16:14:59,739::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:14:59,739::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 16:14:59,739::task::1185::TaskManager.Task::(prepare) Task=`a15d37af-f57e-4e6e-b127-de1c226f4e74`::finished: {'statuslist': [{'status': 0, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 16:14:59,740::task::595::TaskManager.Task::(_updateState) Task=`a15d37af-f57e-4e6e-b127-de1c226f4e74`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:14:59,740::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:14:59,740::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:14:59,740::task::990::TaskManager.Task::(_decref) Task=`a15d37af-f57e-4e6e-b127-de1c226f4e74`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:14:59,903::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [79d06db1]
Thread-13::DEBUG::2014-11-28 16:14:59,904::task::595::TaskManager.Task::(_updateState) Task=`37d55c0e-e0a0-46dd-9f86-55b5cc5f8ce5`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:14:59,904::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:14:59,913::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-82::DEBUG::2014-11-28 16:15:13,280::task::595::TaskManager.Task::(_updateState) Task=`22aac7c0-8e97-49d8-8dd8-ddc07e0155c4`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:15:13,280::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:15:13,280::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '23.6', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}}
Thread-82::DEBUG::2014-11-28 16:15:13,281::task::1185::TaskManager.Task::(prepare) Task=`22aac7c0-8e97-49d8-8dd8-ddc07e0155c4`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '23.6', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}}
Thread-82::DEBUG::2014-11-28 16:15:13,281::task::595::TaskManager.Task::(_updateState) Task=`22aac7c0-8e97-49d8-8dd8-ddc07e0155c4`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:15:13,281::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:15:13,281::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:15:13,281::task::990::TaskManager.Task::(_decref) Task=`22aac7c0-8e97-49d8-8dd8-ddc07e0155c4`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:15:28,364::task::595::TaskManager.Task::(_updateState) Task=`1f06eda9-3a1a-428d-94d9-b6120e868657`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:15:28,367::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:15:28,373::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '38.7', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}}
Thread-82::DEBUG::2014-11-28 16:15:28,376::task::1185::TaskManager.Task::(prepare) Task=`1f06eda9-3a1a-428d-94d9-b6120e868657`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '38.7', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}}
Thread-82::DEBUG::2014-11-28 16:15:28,379::task::595::TaskManager.Task::(_updateState) Task=`1f06eda9-3a1a-428d-94d9-b6120e868657`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:15:28,381::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:15:28,383::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:15:28,387::task::990::TaskManager.Task::(_decref) Task=`1f06eda9-3a1a-428d-94d9-b6120e868657`::ref 0 aborting False
Thread-72::WARNING::2014-11-28 16:15:29,286::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-71::WARNING::2014-11-28 16:15:29,288::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-72::DEBUG::2014-11-28 16:15:29,294::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-71::DEBUG::2014-11-28 16:15:29,300::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:15:29,308::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:15:29,312::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-74::WARNING::2014-11-28 16:15:29,416::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-74::DEBUG::2014-11-28 16:15:29,424::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-75::WARNING::2014-11-28 16:15:29,430::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-75::DEBUG::2014-11-28 16:15:29,434::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:15:29,438::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:15:29,441::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-77::WARNING::2014-11-28 16:15:29,496::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-77::DEBUG::2014-11-28 16:15:29,499::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-78::WARNING::2014-11-28 16:15:29,497::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-78::DEBUG::2014-11-28 16:15:29,502::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-24::ERROR::2014-11-28 16:15:29,505::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-24::ERROR::2014-11-28 16:15:29,507::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-80::WARNING::2014-11-28 16:15:29,575::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-80::DEBUG::2014-11-28 16:15:29,578::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-81::WARNING::2014-11-28 16:15:29,600::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:15:29,600::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:15:29,601::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-19::DEBUG::2014-11-28 16:15:39,319::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-19::DEBUG::2014-11-28 16:15:39,319::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-19::DEBUG::2014-11-28 16:15:39,320::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-19::DEBUG::2014-11-28 16:15:39,320::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-19::DEBUG::2014-11-28 16:15:39,320::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-19::DEBUG::2014-11-28 16:15:39,320::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-19::DEBUG::2014-11-28 16:15:39,332::misc::748::SamplingMethod::(__call__) Returning last result
Thread-19::DEBUG::2014-11-28 16:15:39,335::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-19::DEBUG::2014-11-28 16:15:39,416::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-19::DEBUG::2014-11-28 16:15:39,417::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,417::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,417::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,417::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,418::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,419::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-19::DEBUG::2014-11-28 16:15:39,419::misc::748::SamplingMethod::(__call__) Returning last result
Thread-18::ERROR::2014-11-28 16:15:39,446::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:15:39,447::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:15:39,447::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:15:39,448::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:15:39,521::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:15:39,522::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:15:39,522::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:15:39,602::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:15:39,622::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:15:39,658::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:15:39,661::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:15:39,696::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:15:39,699::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:15:39,699::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:15:39,884::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:15:40,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-24::DEBUG::2014-11-28 16:15:40,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.100:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=474f0809-26c6-4fca-ab8c-72a2e6dcd912', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=80de07a9b66165f2715b4747ae39d51945e0db7f']
Thread-24::DEBUG::2014-11-28 16:15:40,104::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-24::DEBUG::2014-11-28 16:15:40,105::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '474f0809-26c6-4fca-ab8c-72a2e6dcd912_imageNS'
Thread-24::DEBUG::2014-11-28 16:15:40,105::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '474f0809-26c6-4fca-ab8c-72a2e6dcd912_volumeNS'
Thread-24::DEBUG::2014-11-28 16:15:40,113::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:15:40,119::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000222448 s, 1.6 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:15:40,120::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 changed its status to Valid
Thread-103::DEBUG::2014-11-28 16:15:40,121::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-103::DEBUG::2014-11-28 16:15:40,121::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-103::DEBUG::2014-11-28 16:15:40,122::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-103::DEBUG::2014-11-28 16:15:40,123::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-105::INFO::2014-11-28 16:15:40,124::clientIF::125::vds::(contEIOVms) vmContainerLock acquired
Thread-82::DEBUG::2014-11-28 16:15:43,516::task::595::TaskManager.Task::(_updateState) Task=`a9be000e-f25e-49dc-8533-0a42c48d7554`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:15:43,516::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:15:43,516::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '53.8', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000222448', 'lastCheck': '3.4', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:15:43,517::task::1185::TaskManager.Task::(prepare) Task=`a9be000e-f25e-49dc-8533-0a42c48d7554`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '53.8', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000222448', 'lastCheck': '3.4', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:15:43,517::task::595::TaskManager.Task::(_updateState) Task=`a9be000e-f25e-49dc-8533-0a42c48d7554`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:15:43,517::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:15:43,517::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:15:43,518::task::990::TaskManager.Task::(_decref) Task=`a9be000e-f25e-49dc-8533-0a42c48d7554`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:15:50,130::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:15:50,136::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000265307 s, 1.4 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:15:58,589::task::595::TaskManager.Task::(_updateState) Task=`ad5846b0-2139-4d78-bc7c-2854f3216660`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:15:58,590::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:15:58,590::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '68.9', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265307', 'lastCheck': '8.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:15:58,590::task::1185::TaskManager.Task::(prepare) Task=`ad5846b0-2139-4d78-bc7c-2854f3216660`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000244167', 'lastCheck': '68.9', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000265307', 'lastCheck': '8.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:15:58,591::task::595::TaskManager.Task::(_updateState) Task=`ad5846b0-2139-4d78-bc7c-2854f3216660`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:15:58,591::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:15:58,591::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:15:58,591::task::990::TaskManager.Task::(_decref) Task=`ad5846b0-2139-4d78-bc7c-2854f3216660`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 16:15:59,725::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
    fileSD.FileStorageDomain.selftest(self)
  File "/usr/share/vdsm/storage/fileSD.py", line 586, in selftest
    self.oop.os.statvfs(self.domaindir)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::DEBUG::2014-11-28 16:15:59,725::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e7eaae86-11ea-4c08-8c11-8b8c54373389 changed its status to Invalid
Thread-106::DEBUG::2014-11-28 16:15:59,726::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-106::DEBUG::2014-11-28 16:15:59,726::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-106::DEBUG::2014-11-28 16:15:59,727::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-106::DEBUG::2014-11-28 16:15:59,728::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-24::DEBUG::2014-11-28 16:16:00,150::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:00,157::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000262291 s, 1.4 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:16:10,167::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:10,173::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00027082 s, 1.3 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:16:13,662::task::595::TaskManager.Task::(_updateState) Task=`6daee049-4404-4cd4-9eef-2e592f964658`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:13,662::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:16:13,663::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00027082', 'lastCheck': '3.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:13,663::task::1185::TaskManager.Task::(prepare) Task=`6daee049-4404-4cd4-9eef-2e592f964658`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00027082', 'lastCheck': '3.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:13,663::task::595::TaskManager.Task::(_updateState) Task=`6daee049-4404-4cd4-9eef-2e592f964658`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:13,663::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:13,663::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:13,664::task::990::TaskManager.Task::(_decref) Task=`6daee049-4404-4cd4-9eef-2e592f964658`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:14,144::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:14,145::task::595::TaskManager.Task::(_updateState) Task=`69806a96-aee5-4d4d-8a97-e4a554593b01`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:14,145::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-82::INFO::2014-11-28 16:16:14,145::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'Free', 'spmLver': 0}}
Thread-82::DEBUG::2014-11-28 16:16:14,146::task::1185::TaskManager.Task::(prepare) Task=`69806a96-aee5-4d4d-8a97-e4a554593b01`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'Free', 'spmLver': 0}}
Thread-82::DEBUG::2014-11-28 16:16:14,146::task::595::TaskManager.Task::(_updateState) Task=`69806a96-aee5-4d4d-8a97-e4a554593b01`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:14,146::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:14,146::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:14,146::task::990::TaskManager.Task::(_decref) Task=`69806a96-aee5-4d4d-8a97-e4a554593b01`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:14,230::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:14,230::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:14,231::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', prevID=2, prevLVER='0', maxHostID=250, domVersion='3', options=None)
Thread-82::DEBUG::2014-11-28 16:16:14,231::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`8ecda555-f3f7-4af9-8893-c013c18e0940`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '588' at 'spmStart'
Thread-82::DEBUG::2014-11-28 16:16:14,231::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-82::DEBUG::2014-11-28 16:16:14,232::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-82::DEBUG::2014-11-28 16:16:14,232::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`8ecda555-f3f7-4af9-8893-c013c18e0940`::Granted request
Thread-82::DEBUG::2014-11-28 16:16:14,232::task::827::TaskManager.Task::(resourceAcquired) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::_resourcesAcquired: Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad (exclusive)
Thread-82::DEBUG::2014-11-28 16:16:14,232::task::990::TaskManager.Task::(_decref) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::ref 1 aborting False
Thread-82::INFO::2014-11-28 16:16:14,233::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-82::DEBUG::2014-11-28 16:16:14,233::task::1180::TaskManager.Task::(prepare) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Prepare: 1 jobs exist, move to acquiring
Thread-82::DEBUG::2014-11-28 16:16:14,233::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state preparing -> state acquiring
Thread-82::DEBUG::2014-11-28 16:16:14,233::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state acquiring -> state queued
Thread-82::DEBUG::2014-11-28 16:16:14,234::taskManager::50::TaskManager::(_queueTask) queuing task: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:14,234::taskManager::56::TaskManager::(_queueTask) task queued: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-5::DEBUG::2014-11-28 16:16:14,234::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-82::DEBUG::2014-11-28 16:16:14,235::task::1182::TaskManager.Task::(prepare) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::returning
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,235::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 84c395bb-1362-491c-a275-d3b0766622b4 running: <bound method Task.commit of <storage.task.Task instance at 0x7ff36c0d6710>> with: None
Thread-82::DEBUG::2014-11-28 16:16:14,236::task::990::TaskManager.Task::(_decref) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::ref 0 aborting False
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,236::task::1194::TaskManager.Task::(commit) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::committing task: 84c395bb-1362-491c-a275-d3b0766622b4
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,237::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state queued -> state running
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,237::task::905::TaskManager.Task::(_runJobs) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x118acd0>> (args: (2, '0', 250, 3) kwargs: {})
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,238::task::331::TaskManager.Task::(run) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x118acd0>> (args: (2, '0', 250, 3) kwargs: {}) callback None
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,238::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:16:14,238::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-82::DEBUG::2014-11-28 16:16:15,242::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:15,243::task::595::TaskManager.Task::(_updateState) Task=`bc734226-0b42-4c02-941b-ce266f756450`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:15,243::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:15,243::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:15,243::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:15,244::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:15,244::task::1185::TaskManager.Task::(prepare) Task=`bc734226-0b42-4c02-941b-ce266f756450`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:15,244::task::595::TaskManager.Task::(_updateState) Task=`bc734226-0b42-4c02-941b-ce266f756450`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:15,244::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:15,244::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:15,244::task::990::TaskManager.Task::(_decref) Task=`bc734226-0b42-4c02-941b-ce266f756450`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:16,248::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:16,248::task::595::TaskManager.Task::(_updateState) Task=`5e00b2d2-798f-4c91-ad32-dd952b36f7cc`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:16,248::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:16,249::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:16,249::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:16,249::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:16,249::task::1185::TaskManager.Task::(prepare) Task=`5e00b2d2-798f-4c91-ad32-dd952b36f7cc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:16,249::task::595::TaskManager.Task::(_updateState) Task=`5e00b2d2-798f-4c91-ad32-dd952b36f7cc`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:16,250::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:16,250::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:16,250::task::990::TaskManager.Task::(_decref) Task=`5e00b2d2-798f-4c91-ad32-dd952b36f7cc`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:17,253::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:17,254::task::595::TaskManager.Task::(_updateState) Task=`ff2dab6f-478a-427e-a695-b66b1b821426`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:17,254::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:17,254::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:17,254::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:17,254::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:17,254::task::1185::TaskManager.Task::(prepare) Task=`ff2dab6f-478a-427e-a695-b66b1b821426`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:17,255::task::595::TaskManager.Task::(_updateState) Task=`ff2dab6f-478a-427e-a695-b66b1b821426`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:17,255::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:17,255::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:17,255::task::990::TaskManager.Task::(_decref) Task=`ff2dab6f-478a-427e-a695-b66b1b821426`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:18,259::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:18,259::task::595::TaskManager.Task::(_updateState) Task=`b50a8e42-0594-4ac2-9f2d-b5250c6d6656`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:18,259::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:18,260::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:18,260::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:18,260::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:18,260::task::1185::TaskManager.Task::(prepare) Task=`b50a8e42-0594-4ac2-9f2d-b5250c6d6656`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:18,260::task::595::TaskManager.Task::(_updateState) Task=`b50a8e42-0594-4ac2-9f2d-b5250c6d6656`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:18,260::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:18,261::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:18,261::task::990::TaskManager.Task::(_decref) Task=`b50a8e42-0594-4ac2-9f2d-b5250c6d6656`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:19,264::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:19,265::task::595::TaskManager.Task::(_updateState) Task=`48f32122-f560-414d-b5cf-2b1e894cda7f`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:19,265::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:19,265::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:19,265::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:19,265::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:19,266::task::1185::TaskManager.Task::(prepare) Task=`48f32122-f560-414d-b5cf-2b1e894cda7f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:19,266::task::595::TaskManager.Task::(_updateState) Task=`48f32122-f560-414d-b5cf-2b1e894cda7f`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:19,266::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:19,266::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:19,266::task::990::TaskManager.Task::(_decref) Task=`48f32122-f560-414d-b5cf-2b1e894cda7f`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:16:20,183::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:20,189::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000244228 s, 1.5 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:16:20,270::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:20,271::task::595::TaskManager.Task::(_updateState) Task=`653c6525-a716-4580-badb-ab025e109ab6`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:20,271::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:20,271::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:20,271::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:20,272::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:20,272::task::1185::TaskManager.Task::(prepare) Task=`653c6525-a716-4580-badb-ab025e109ab6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:20,272::task::595::TaskManager.Task::(_updateState) Task=`653c6525-a716-4580-badb-ab025e109ab6`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:20,272::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:20,272::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:20,273::task::990::TaskManager.Task::(_decref) Task=`653c6525-a716-4580-badb-ab025e109ab6`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:21,276::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:21,276::task::595::TaskManager.Task::(_updateState) Task=`84065ca3-5726-4614-9621-822b7f395e20`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:21,277::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:21,277::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:21,277::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:21,277::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:21,278::task::1185::TaskManager.Task::(prepare) Task=`84065ca3-5726-4614-9621-822b7f395e20`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:21,278::task::595::TaskManager.Task::(_updateState) Task=`84065ca3-5726-4614-9621-822b7f395e20`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:21,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:21,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:21,279::task::990::TaskManager.Task::(_decref) Task=`84065ca3-5726-4614-9621-822b7f395e20`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:22,282::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:22,282::task::595::TaskManager.Task::(_updateState) Task=`eeb409e4-5913-4a9d-a565-8a3348ce4352`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:22,283::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:22,283::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:22,283::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:22,283::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:22,284::task::1185::TaskManager.Task::(prepare) Task=`eeb409e4-5913-4a9d-a565-8a3348ce4352`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:22,284::task::595::TaskManager.Task::(_updateState) Task=`eeb409e4-5913-4a9d-a565-8a3348ce4352`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:22,284::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:22,284::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:22,284::task::990::TaskManager.Task::(_decref) Task=`eeb409e4-5913-4a9d-a565-8a3348ce4352`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:23,288::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:23,288::task::595::TaskManager.Task::(_updateState) Task=`60b61de3-9629-417d-ab56-7990662e5e0b`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:23,288::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:23,288::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:23,289::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:23,289::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:23,289::task::1185::TaskManager.Task::(prepare) Task=`60b61de3-9629-417d-ab56-7990662e5e0b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:23,289::task::595::TaskManager.Task::(_updateState) Task=`60b61de3-9629-417d-ab56-7990662e5e0b`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:23,290::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:23,290::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:23,290::task::990::TaskManager.Task::(_decref) Task=`60b61de3-9629-417d-ab56-7990662e5e0b`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:24,293::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:24,294::task::595::TaskManager.Task::(_updateState) Task=`a62365ab-ad69-4f72-8f7d-1956e30e94ac`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:24,294::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:24,294::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:24,295::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:24,295::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:24,295::task::1185::TaskManager.Task::(prepare) Task=`a62365ab-ad69-4f72-8f7d-1956e30e94ac`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:24,295::task::595::TaskManager.Task::(_updateState) Task=`a62365ab-ad69-4f72-8f7d-1956e30e94ac`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:24,296::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:24,296::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:24,296::task::990::TaskManager.Task::(_decref) Task=`a62365ab-ad69-4f72-8f7d-1956e30e94ac`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:25,300::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:25,301::task::595::TaskManager.Task::(_updateState) Task=`4775dd32-744b-4034-a5a8-64e18893a2c1`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:25,301::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:25,301::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:25,301::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:25,301::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:25,302::task::1185::TaskManager.Task::(prepare) Task=`4775dd32-744b-4034-a5a8-64e18893a2c1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:25,302::task::595::TaskManager.Task::(_updateState) Task=`4775dd32-744b-4034-a5a8-64e18893a2c1`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:25,302::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:25,302::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:25,302::task::990::TaskManager.Task::(_decref) Task=`4775dd32-744b-4034-a5a8-64e18893a2c1`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:26,306::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:26,306::task::595::TaskManager.Task::(_updateState) Task=`a748cefb-7e6e-458f-8fd2-5a2713193510`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:26,307::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:26,307::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:26,307::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:26,307::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:26,307::task::1185::TaskManager.Task::(prepare) Task=`a748cefb-7e6e-458f-8fd2-5a2713193510`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:26,308::task::595::TaskManager.Task::(_updateState) Task=`a748cefb-7e6e-458f-8fd2-5a2713193510`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:26,308::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:26,308::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:26,308::task::990::TaskManager.Task::(_decref) Task=`a748cefb-7e6e-458f-8fd2-5a2713193510`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:27,312::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:27,312::task::595::TaskManager.Task::(_updateState) Task=`85f72431-090f-4eee-ace1-4c5e153fb8b7`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:27,312::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:27,312::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:27,313::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:27,313::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:27,313::task::1185::TaskManager.Task::(prepare) Task=`85f72431-090f-4eee-ace1-4c5e153fb8b7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:27,313::task::595::TaskManager.Task::(_updateState) Task=`85f72431-090f-4eee-ace1-4c5e153fb8b7`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:27,314::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:27,314::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:27,314::task::990::TaskManager.Task::(_decref) Task=`85f72431-090f-4eee-ace1-4c5e153fb8b7`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:28,317::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:28,318::task::595::TaskManager.Task::(_updateState) Task=`da8b2ecb-4b7a-4225-90f7-bf3c6db87499`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:28,318::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:28,318::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:28,319::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:28,319::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:28,319::task::1185::TaskManager.Task::(prepare) Task=`da8b2ecb-4b7a-4225-90f7-bf3c6db87499`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:28,319::task::595::TaskManager.Task::(_updateState) Task=`da8b2ecb-4b7a-4225-90f7-bf3c6db87499`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:28,319::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:28,320::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:28,320::task::990::TaskManager.Task::(_decref) Task=`da8b2ecb-4b7a-4225-90f7-bf3c6db87499`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:28,736::task::595::TaskManager.Task::(_updateState) Task=`9842a1e5-565f-47e6-b8af-b962d3de1f81`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:28,737::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:16:28,737::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000244228', 'lastCheck': '8.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:28,737::task::1185::TaskManager.Task::(prepare) Task=`9842a1e5-565f-47e6-b8af-b962d3de1f81`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000244228', 'lastCheck': '8.5', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:28,738::task::595::TaskManager.Task::(_updateState) Task=`9842a1e5-565f-47e6-b8af-b962d3de1f81`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:28,738::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:28,738::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:28,738::task::990::TaskManager.Task::(_decref) Task=`9842a1e5-565f-47e6-b8af-b962d3de1f81`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:29,323::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:29,324::task::595::TaskManager.Task::(_updateState) Task=`c6f691fe-c2ba-45c1-9c18-2716abab4c2d`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:29,324::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:29,324::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:29,324::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:29,325::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:29,325::task::1185::TaskManager.Task::(prepare) Task=`c6f691fe-c2ba-45c1-9c18-2716abab4c2d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:29,325::task::595::TaskManager.Task::(_updateState) Task=`c6f691fe-c2ba-45c1-9c18-2716abab4c2d`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:29,325::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:29,325::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:29,326::task::990::TaskManager.Task::(_decref) Task=`c6f691fe-c2ba-45c1-9c18-2716abab4c2d`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:16:30,199::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:30,205::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000292593 s, 1.2 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:16:30,329::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:30,330::task::595::TaskManager.Task::(_updateState) Task=`b21d01b6-f7ec-4b98-91b7-6a46413f41c7`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:30,330::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:30,330::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:30,330::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:30,331::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:30,331::task::1185::TaskManager.Task::(prepare) Task=`b21d01b6-f7ec-4b98-91b7-6a46413f41c7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:30,331::task::595::TaskManager.Task::(_updateState) Task=`b21d01b6-f7ec-4b98-91b7-6a46413f41c7`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:30,331::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:30,331::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:30,332::task::990::TaskManager.Task::(_decref) Task=`b21d01b6-f7ec-4b98-91b7-6a46413f41c7`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:31,335::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:31,335::task::595::TaskManager.Task::(_updateState) Task=`fb476114-166a-488a-a972-893141001597`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:31,336::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:31,336::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:31,336::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:31,337::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:31,337::task::1185::TaskManager.Task::(prepare) Task=`fb476114-166a-488a-a972-893141001597`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:31,337::task::595::TaskManager.Task::(_updateState) Task=`fb476114-166a-488a-a972-893141001597`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:31,337::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:31,337::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:31,338::task::990::TaskManager.Task::(_decref) Task=`fb476114-166a-488a-a972-893141001597`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:32,341::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:32,342::task::595::TaskManager.Task::(_updateState) Task=`37652652-cafe-4d1b-8b0d-7ffe6f734700`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:32,342::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:32,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:32,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:32,343::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:32,343::task::1185::TaskManager.Task::(prepare) Task=`37652652-cafe-4d1b-8b0d-7ffe6f734700`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:32,343::task::595::TaskManager.Task::(_updateState) Task=`37652652-cafe-4d1b-8b0d-7ffe6f734700`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:32,343::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:32,344::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:32,344::task::990::TaskManager.Task::(_decref) Task=`37652652-cafe-4d1b-8b0d-7ffe6f734700`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:33,347::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:33,348::task::595::TaskManager.Task::(_updateState) Task=`adb69c88-b798-439a-8dd1-edc7276dcd9b`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:33,348::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:33,348::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:33,348::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:33,349::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:33,349::task::1185::TaskManager.Task::(prepare) Task=`adb69c88-b798-439a-8dd1-edc7276dcd9b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:33,349::task::595::TaskManager.Task::(_updateState) Task=`adb69c88-b798-439a-8dd1-edc7276dcd9b`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:33,349::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:33,350::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:33,350::task::990::TaskManager.Task::(_decref) Task=`adb69c88-b798-439a-8dd1-edc7276dcd9b`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:34,353::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:34,353::task::595::TaskManager.Task::(_updateState) Task=`0cbb4a7c-1421-4201-a564-ece655e77b29`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:34,354::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:34,354::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:34,354::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:34,355::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:34,355::task::1185::TaskManager.Task::(prepare) Task=`0cbb4a7c-1421-4201-a564-ece655e77b29`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:34,355::task::595::TaskManager.Task::(_updateState) Task=`0cbb4a7c-1421-4201-a564-ece655e77b29`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:34,355::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:34,355::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:34,356::task::990::TaskManager.Task::(_decref) Task=`0cbb4a7c-1421-4201-a564-ece655e77b29`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:35,359::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:35,360::task::595::TaskManager.Task::(_updateState) Task=`477ad85a-197c-4bcf-9508-a6214c8b9b22`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:35,361::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:35,361::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:35,361::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:35,362::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:35,362::task::1185::TaskManager.Task::(prepare) Task=`477ad85a-197c-4bcf-9508-a6214c8b9b22`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:35,362::task::595::TaskManager.Task::(_updateState) Task=`477ad85a-197c-4bcf-9508-a6214c8b9b22`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:35,363::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:35,363::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:35,363::task::990::TaskManager.Task::(_decref) Task=`477ad85a-197c-4bcf-9508-a6214c8b9b22`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:36,366::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:36,367::task::595::TaskManager.Task::(_updateState) Task=`e2cb77e8-7902-456b-b63c-57c0aa93d0b7`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:36,367::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:36,367::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:36,367::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:36,368::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:36,368::task::1185::TaskManager.Task::(prepare) Task=`e2cb77e8-7902-456b-b63c-57c0aa93d0b7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:36,368::task::595::TaskManager.Task::(_updateState) Task=`e2cb77e8-7902-456b-b63c-57c0aa93d0b7`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:36,368::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:36,369::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:36,369::task::990::TaskManager.Task::(_decref) Task=`e2cb77e8-7902-456b-b63c-57c0aa93d0b7`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:37,373::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:37,374::task::595::TaskManager.Task::(_updateState) Task=`474a92dd-c46b-4fce-b034-7b171e7456c2`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:37,375::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:37,375::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:37,375::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:37,376::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:37,376::task::1185::TaskManager.Task::(prepare) Task=`474a92dd-c46b-4fce-b034-7b171e7456c2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:37,376::task::595::TaskManager.Task::(_updateState) Task=`474a92dd-c46b-4fce-b034-7b171e7456c2`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:37,377::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:37,377::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:37,377::task::990::TaskManager.Task::(_decref) Task=`474a92dd-c46b-4fce-b034-7b171e7456c2`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:38,384::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:38,386::task::595::TaskManager.Task::(_updateState) Task=`97347d3a-35bb-448c-85de-25e0a0dc909c`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:38,388::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:38,389::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:38,390::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:38,391::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:38,392::task::1185::TaskManager.Task::(prepare) Task=`97347d3a-35bb-448c-85de-25e0a0dc909c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:38,394::task::595::TaskManager.Task::(_updateState) Task=`97347d3a-35bb-448c-85de-25e0a0dc909c`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:38,395::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:38,396::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:38,398::task::990::TaskManager.Task::(_decref) Task=`97347d3a-35bb-448c-85de-25e0a0dc909c`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:39,409::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:39,416::task::595::TaskManager.Task::(_updateState) Task=`cbf526af-b7f8-4375-aa55-1777da569c39`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:39,421::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:39,425::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:39,430::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:39,433::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-83::WARNING::2014-11-28 16:16:39,424::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:16:39,436::task::1185::TaskManager.Task::(prepare) Task=`cbf526af-b7f8-4375-aa55-1777da569c39`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:39,441::task::595::TaskManager.Task::(_updateState) Task=`cbf526af-b7f8-4375-aa55-1777da569c39`::moving from state preparing -> state finished
Thread-83::DEBUG::2014-11-28 16:16:39,442::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::DEBUG::2014-11-28 16:16:39,445::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-84::WARNING::2014-11-28 16:16:39,439::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:16:39,450::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-85::WARNING::2014-11-28 16:16:39,444::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:16:39,458::task::990::TaskManager.Task::(_decref) Task=`cbf526af-b7f8-4375-aa55-1777da569c39`::ref 0 aborting False
Thread-84::DEBUG::2014-11-28 16:16:39,459::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-85::DEBUG::2014-11-28 16:16:39,466::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:16:39,478::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:16:39,483::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-88::WARNING::2014-11-28 16:16:39,549::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-88::DEBUG::2014-11-28 16:16:39,567::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-91::WARNING::2014-11-28 16:16:39,659::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-89::WARNING::2014-11-28 16:16:39,662::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-91::DEBUG::2014-11-28 16:16:39,670::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-89::DEBUG::2014-11-28 16:16:39,675::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-90::WARNING::2014-11-28 16:16:39,672::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-90::DEBUG::2014-11-28 16:16:39,682::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-94::WARNING::2014-11-28 16:16:39,696::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-94::DEBUG::2014-11-28 16:16:39,701::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-95::WARNING::2014-11-28 16:16:39,722::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-95::DEBUG::2014-11-28 16:16:39,727::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:16:39,732::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:16:39,733::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-98::WARNING::2014-11-28 16:16:39,773::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-98::DEBUG::2014-11-28 16:16:39,775::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-99::WARNING::2014-11-28 16:16:39,820::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-99::DEBUG::2014-11-28 16:16:39,822::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-100::WARNING::2014-11-28 16:16:39,877::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:16:39,878::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:16:39,878::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:16:40,214::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:40,220::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00044219 s, 812 kB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:16:40,475::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:40,475::task::595::TaskManager.Task::(_updateState) Task=`62cfba8c-80fb-4c79-b0a2-7232e4e4637c`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:40,475::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:40,476::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:40,476::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:40,476::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:40,476::task::1185::TaskManager.Task::(prepare) Task=`62cfba8c-80fb-4c79-b0a2-7232e4e4637c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:40,476::task::595::TaskManager.Task::(_updateState) Task=`62cfba8c-80fb-4c79-b0a2-7232e4e4637c`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:40,477::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:40,477::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:40,477::task::990::TaskManager.Task::(_decref) Task=`62cfba8c-80fb-4c79-b0a2-7232e4e4637c`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:41,480::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:41,480::task::595::TaskManager.Task::(_updateState) Task=`c21176f5-70ea-4e85-afb3-f8892a668605`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:41,481::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:41,481::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:41,481::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:41,481::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:41,481::task::1185::TaskManager.Task::(prepare) Task=`c21176f5-70ea-4e85-afb3-f8892a668605`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:41,482::task::595::TaskManager.Task::(_updateState) Task=`c21176f5-70ea-4e85-afb3-f8892a668605`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:41,482::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:41,482::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:41,482::task::990::TaskManager.Task::(_decref) Task=`c21176f5-70ea-4e85-afb3-f8892a668605`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:42,486::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:42,486::task::595::TaskManager.Task::(_updateState) Task=`c7b30ea5-0984-462c-9e71-28bfb16ca363`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:42,487::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:42,487::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:42,487::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:42,487::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:42,487::task::1185::TaskManager.Task::(prepare) Task=`c7b30ea5-0984-462c-9e71-28bfb16ca363`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:42,488::task::595::TaskManager.Task::(_updateState) Task=`c7b30ea5-0984-462c-9e71-28bfb16ca363`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:42,488::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:42,488::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:42,488::task::990::TaskManager.Task::(_decref) Task=`c7b30ea5-0984-462c-9e71-28bfb16ca363`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:43,491::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:43,492::task::595::TaskManager.Task::(_updateState) Task=`ed4756d5-9d11-4c4d-9044-43f20e5452e9`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:43,492::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:43,492::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:43,492::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:43,492::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:43,493::task::1185::TaskManager.Task::(prepare) Task=`ed4756d5-9d11-4c4d-9044-43f20e5452e9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:43,493::task::595::TaskManager.Task::(_updateState) Task=`ed4756d5-9d11-4c4d-9044-43f20e5452e9`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:43,493::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:43,493::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:43,493::task::990::TaskManager.Task::(_decref) Task=`ed4756d5-9d11-4c4d-9044-43f20e5452e9`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:43,813::task::595::TaskManager.Task::(_updateState) Task=`94508d3f-3e9f-4bed-ba9b-b130fcf65846`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:43,813::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:16:43,814::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00044219', 'lastCheck': '3.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:43,814::task::1185::TaskManager.Task::(prepare) Task=`94508d3f-3e9f-4bed-ba9b-b130fcf65846`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00044219', 'lastCheck': '3.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:43,814::task::595::TaskManager.Task::(_updateState) Task=`94508d3f-3e9f-4bed-ba9b-b130fcf65846`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:43,814::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:43,814::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:43,815::task::990::TaskManager.Task::(_decref) Task=`94508d3f-3e9f-4bed-ba9b-b130fcf65846`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:44,497::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:44,497::task::595::TaskManager.Task::(_updateState) Task=`525d10c8-8c5f-419a-be84-f0454f378362`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:44,497::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:44,497::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:44,498::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:44,498::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:44,498::task::1185::TaskManager.Task::(prepare) Task=`525d10c8-8c5f-419a-be84-f0454f378362`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:44,498::task::595::TaskManager.Task::(_updateState) Task=`525d10c8-8c5f-419a-be84-f0454f378362`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:44,498::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:44,498::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:44,499::task::990::TaskManager.Task::(_decref) Task=`525d10c8-8c5f-419a-be84-f0454f378362`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:45,503::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:45,503::task::595::TaskManager.Task::(_updateState) Task=`4d492e68-99de-43fb-b488-2253022534b3`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:45,503::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:45,504::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:45,504::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:45,504::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:45,504::task::1185::TaskManager.Task::(prepare) Task=`4d492e68-99de-43fb-b488-2253022534b3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:45,504::task::595::TaskManager.Task::(_updateState) Task=`4d492e68-99de-43fb-b488-2253022534b3`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:45,505::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:45,505::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:45,505::task::990::TaskManager.Task::(_decref) Task=`4d492e68-99de-43fb-b488-2253022534b3`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:46,508::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:46,509::task::595::TaskManager.Task::(_updateState) Task=`a719c5f3-7ffe-46f4-830a-e6ab6352f15c`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:46,509::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:46,509::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:46,509::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:46,509::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:46,510::task::1185::TaskManager.Task::(prepare) Task=`a719c5f3-7ffe-46f4-830a-e6ab6352f15c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:46,510::task::595::TaskManager.Task::(_updateState) Task=`a719c5f3-7ffe-46f4-830a-e6ab6352f15c`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:46,510::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:46,510::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:46,510::task::990::TaskManager.Task::(_decref) Task=`a719c5f3-7ffe-46f4-830a-e6ab6352f15c`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:47,514::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:47,514::task::595::TaskManager.Task::(_updateState) Task=`5f3971e2-df9c-4ff5-92db-e9686089fe43`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:47,515::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:47,515::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:47,515::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:47,515::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:47,515::task::1185::TaskManager.Task::(prepare) Task=`5f3971e2-df9c-4ff5-92db-e9686089fe43`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:47,515::task::595::TaskManager.Task::(_updateState) Task=`5f3971e2-df9c-4ff5-92db-e9686089fe43`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:47,516::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:47,516::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:47,516::task::990::TaskManager.Task::(_decref) Task=`5f3971e2-df9c-4ff5-92db-e9686089fe43`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:48,519::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:48,520::task::595::TaskManager.Task::(_updateState) Task=`bae4e3bd-f11e-4e64-b465-953dbf36ea8a`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:48,520::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:48,520::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:48,520::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:48,520::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:48,521::task::1185::TaskManager.Task::(prepare) Task=`bae4e3bd-f11e-4e64-b465-953dbf36ea8a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:48,521::task::595::TaskManager.Task::(_updateState) Task=`bae4e3bd-f11e-4e64-b465-953dbf36ea8a`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:48,521::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:48,521::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:48,521::task::990::TaskManager.Task::(_decref) Task=`bae4e3bd-f11e-4e64-b465-953dbf36ea8a`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:49,525::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:49,525::task::595::TaskManager.Task::(_updateState) Task=`440a3ca2-a3be-4bc5-96d5-1eda17ac35b5`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:49,525::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:49,526::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:49,526::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:49,526::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:49,526::task::1185::TaskManager.Task::(prepare) Task=`440a3ca2-a3be-4bc5-96d5-1eda17ac35b5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:49,526::task::595::TaskManager.Task::(_updateState) Task=`440a3ca2-a3be-4bc5-96d5-1eda17ac35b5`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:49,527::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:49,527::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:49,527::task::990::TaskManager.Task::(_decref) Task=`440a3ca2-a3be-4bc5-96d5-1eda17ac35b5`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 16:16:49,737::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:16:49,737::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:16:49,738::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:16:49,738::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:16:49,758::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:16:49,759::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:16:49,760::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:16:49,879::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:16:49,879::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:16:49,879::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:16:49,880::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:16:49,912::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:16:49,913::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:16:49,913::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:16:50,230::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:16:50,237::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000281232 s, 1.3 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:16:50,531::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:50,531::task::595::TaskManager.Task::(_updateState) Task=`57ad0dba-552f-4c8a-bfd3-16d0c548c31e`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:50,531::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:50,532::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:50,532::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:50,532::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:50,532::task::1185::TaskManager.Task::(prepare) Task=`57ad0dba-552f-4c8a-bfd3-16d0c548c31e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:50,532::task::595::TaskManager.Task::(_updateState) Task=`57ad0dba-552f-4c8a-bfd3-16d0c548c31e`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:50,533::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:50,533::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:50,533::task::990::TaskManager.Task::(_decref) Task=`57ad0dba-552f-4c8a-bfd3-16d0c548c31e`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:51,536::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:51,537::task::595::TaskManager.Task::(_updateState) Task=`fceac40f-a86c-4f65-a0e5-505f8291d7e8`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:51,537::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:51,537::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:51,537::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:51,538::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:51,538::task::1185::TaskManager.Task::(prepare) Task=`fceac40f-a86c-4f65-a0e5-505f8291d7e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:51,538::task::595::TaskManager.Task::(_updateState) Task=`fceac40f-a86c-4f65-a0e5-505f8291d7e8`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:51,538::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:51,538::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:51,539::task::990::TaskManager.Task::(_decref) Task=`fceac40f-a86c-4f65-a0e5-505f8291d7e8`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:52,542::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:52,542::task::595::TaskManager.Task::(_updateState) Task=`301fb548-8cc1-4b89-8775-02398b8358fe`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:52,543::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:52,543::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:52,543::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:52,543::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:52,543::task::1185::TaskManager.Task::(prepare) Task=`301fb548-8cc1-4b89-8775-02398b8358fe`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:52,543::task::595::TaskManager.Task::(_updateState) Task=`301fb548-8cc1-4b89-8775-02398b8358fe`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:52,544::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:52,544::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:52,544::task::990::TaskManager.Task::(_decref) Task=`301fb548-8cc1-4b89-8775-02398b8358fe`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:53,547::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:53,548::task::595::TaskManager.Task::(_updateState) Task=`998ed31f-76fe-430f-9e79-927a7826c97e`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:53,548::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:53,548::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:53,548::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:53,548::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:53,549::task::1185::TaskManager.Task::(prepare) Task=`998ed31f-76fe-430f-9e79-927a7826c97e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:53,549::task::595::TaskManager.Task::(_updateState) Task=`998ed31f-76fe-430f-9e79-927a7826c97e`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:53,549::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:53,549::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:53,549::task::990::TaskManager.Task::(_decref) Task=`998ed31f-76fe-430f-9e79-927a7826c97e`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:54,552::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:54,553::task::595::TaskManager.Task::(_updateState) Task=`901f4861-1e73-4352-87c4-5b6c9a345b76`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:54,553::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:54,553::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:54,553::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:54,554::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:54,554::task::1185::TaskManager.Task::(prepare) Task=`901f4861-1e73-4352-87c4-5b6c9a345b76`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:54,554::task::595::TaskManager.Task::(_updateState) Task=`901f4861-1e73-4352-87c4-5b6c9a345b76`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:54,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:54,555::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:54,555::task::990::TaskManager.Task::(_decref) Task=`901f4861-1e73-4352-87c4-5b6c9a345b76`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:55,558::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:55,559::task::595::TaskManager.Task::(_updateState) Task=`57f75707-2325-45eb-9542-73cae253776f`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:55,559::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:55,560::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:55,560::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:55,560::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:55,560::task::1185::TaskManager.Task::(prepare) Task=`57f75707-2325-45eb-9542-73cae253776f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:55,560::task::595::TaskManager.Task::(_updateState) Task=`57f75707-2325-45eb-9542-73cae253776f`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:55,561::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:55,561::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:55,561::task::990::TaskManager.Task::(_decref) Task=`57f75707-2325-45eb-9542-73cae253776f`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:56,564::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:56,565::task::595::TaskManager.Task::(_updateState) Task=`e8f2ae33-bce2-43d8-acf9-1eb30f3a9c2a`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:56,565::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:56,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:56,565::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:56,566::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:56,566::task::1185::TaskManager.Task::(prepare) Task=`e8f2ae33-bce2-43d8-acf9-1eb30f3a9c2a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:56,566::task::595::TaskManager.Task::(_updateState) Task=`e8f2ae33-bce2-43d8-acf9-1eb30f3a9c2a`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:56,566::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:56,566::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:56,567::task::990::TaskManager.Task::(_decref) Task=`e8f2ae33-bce2-43d8-acf9-1eb30f3a9c2a`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:57,570::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:57,571::task::595::TaskManager.Task::(_updateState) Task=`ee4bb0db-b180-48df-94e5-9ea5851c6627`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:57,571::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:57,571::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:57,571::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:57,571::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:57,571::task::1185::TaskManager.Task::(prepare) Task=`ee4bb0db-b180-48df-94e5-9ea5851c6627`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:57,572::task::595::TaskManager.Task::(_updateState) Task=`ee4bb0db-b180-48df-94e5-9ea5851c6627`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:57,572::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:57,572::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:57,572::task::990::TaskManager.Task::(_decref) Task=`ee4bb0db-b180-48df-94e5-9ea5851c6627`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:58,575::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:58,576::task::595::TaskManager.Task::(_updateState) Task=`c86ba94c-d48f-4ff9-a38f-f01941e9993d`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:58,576::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:58,576::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:58,576::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:58,576::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:58,577::task::1185::TaskManager.Task::(prepare) Task=`c86ba94c-d48f-4ff9-a38f-f01941e9993d`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:58,577::task::595::TaskManager.Task::(_updateState) Task=`c86ba94c-d48f-4ff9-a38f-f01941e9993d`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:58,577::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:58,577::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:58,577::task::990::TaskManager.Task::(_decref) Task=`c86ba94c-d48f-4ff9-a38f-f01941e9993d`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:58,887::task::595::TaskManager.Task::(_updateState) Task=`9d5eaec5-ad7e-4ac9-a975-2fcc554dcd20`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:58,888::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:16:58,888::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281232', 'lastCheck': '8.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:58,888::task::1185::TaskManager.Task::(prepare) Task=`9d5eaec5-ad7e-4ac9-a975-2fcc554dcd20`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281232', 'lastCheck': '8.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:16:58,888::task::595::TaskManager.Task::(_updateState) Task=`9d5eaec5-ad7e-4ac9-a975-2fcc554dcd20`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:58,889::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:58,889::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:58,889::task::990::TaskManager.Task::(_decref) Task=`9d5eaec5-ad7e-4ac9-a975-2fcc554dcd20`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:16:59,580::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:16:59,581::task::595::TaskManager.Task::(_updateState) Task=`cd6dc825-1d8e-4a52-9bee-2bb0d9d35038`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:16:59,581::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:16:59,581::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:16:59,582::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:16:59,582::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:59,582::task::1185::TaskManager.Task::(prepare) Task=`cd6dc825-1d8e-4a52-9bee-2bb0d9d35038`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:16:59,582::task::595::TaskManager.Task::(_updateState) Task=`cd6dc825-1d8e-4a52-9bee-2bb0d9d35038`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:16:59,582::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:16:59,582::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:16:59,583::task::990::TaskManager.Task::(_decref) Task=`cd6dc825-1d8e-4a52-9bee-2bb0d9d35038`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:17:00,247::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:00,252::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00024636 s, 1.5 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:17:00,586::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:00,586::task::595::TaskManager.Task::(_updateState) Task=`2b9be0e1-d73f-494f-bb23-7570ed2ec517`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:00,587::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:00,587::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:00,587::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:00,587::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:00,587::task::1185::TaskManager.Task::(prepare) Task=`2b9be0e1-d73f-494f-bb23-7570ed2ec517`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:00,588::task::595::TaskManager.Task::(_updateState) Task=`2b9be0e1-d73f-494f-bb23-7570ed2ec517`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:00,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:00,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:00,588::task::990::TaskManager.Task::(_decref) Task=`2b9be0e1-d73f-494f-bb23-7570ed2ec517`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:01,591::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:01,592::task::595::TaskManager.Task::(_updateState) Task=`2041a75c-a2d1-42e5-bfd8-e2444d361430`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:01,592::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:01,592::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:01,592::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:01,593::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:01,593::task::1185::TaskManager.Task::(prepare) Task=`2041a75c-a2d1-42e5-bfd8-e2444d361430`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:01,593::task::595::TaskManager.Task::(_updateState) Task=`2041a75c-a2d1-42e5-bfd8-e2444d361430`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:01,593::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:01,593::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:01,593::task::990::TaskManager.Task::(_decref) Task=`2041a75c-a2d1-42e5-bfd8-e2444d361430`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:02,597::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:02,597::task::595::TaskManager.Task::(_updateState) Task=`9af280cf-02c1-4ad0-972f-6ff9caeb14b9`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:02,597::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:02,598::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:02,598::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:02,598::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:02,598::task::1185::TaskManager.Task::(prepare) Task=`9af280cf-02c1-4ad0-972f-6ff9caeb14b9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:02,598::task::595::TaskManager.Task::(_updateState) Task=`9af280cf-02c1-4ad0-972f-6ff9caeb14b9`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:02,599::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:02,599::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:02,599::task::990::TaskManager.Task::(_decref) Task=`9af280cf-02c1-4ad0-972f-6ff9caeb14b9`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:03,628::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:03,629::task::595::TaskManager.Task::(_updateState) Task=`e5ef7069-1f64-41c1-ba08-10090d8eb532`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:03,629::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:03,629::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:03,629::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:03,630::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:03,630::task::1185::TaskManager.Task::(prepare) Task=`e5ef7069-1f64-41c1-ba08-10090d8eb532`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:03,630::task::595::TaskManager.Task::(_updateState) Task=`e5ef7069-1f64-41c1-ba08-10090d8eb532`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:03,630::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:03,630::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:03,631::task::990::TaskManager.Task::(_decref) Task=`e5ef7069-1f64-41c1-ba08-10090d8eb532`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:04,634::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:04,634::task::595::TaskManager.Task::(_updateState) Task=`f4f361c4-9e42-429c-99b5-a3704e75df42`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:04,635::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:04,635::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:04,635::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:04,635::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:04,635::task::1185::TaskManager.Task::(prepare) Task=`f4f361c4-9e42-429c-99b5-a3704e75df42`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:04,636::task::595::TaskManager.Task::(_updateState) Task=`f4f361c4-9e42-429c-99b5-a3704e75df42`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:04,636::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:04,636::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:04,636::task::990::TaskManager.Task::(_decref) Task=`f4f361c4-9e42-429c-99b5-a3704e75df42`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:05,640::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:05,641::task::595::TaskManager.Task::(_updateState) Task=`16c4b68c-0910-4f65-8536-9c211c8a634b`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:05,641::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:05,641::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:05,641::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:05,642::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:05,642::task::1185::TaskManager.Task::(prepare) Task=`16c4b68c-0910-4f65-8536-9c211c8a634b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:05,642::task::595::TaskManager.Task::(_updateState) Task=`16c4b68c-0910-4f65-8536-9c211c8a634b`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:05,642::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:05,642::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:05,643::task::990::TaskManager.Task::(_decref) Task=`16c4b68c-0910-4f65-8536-9c211c8a634b`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:06,646::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:06,646::task::595::TaskManager.Task::(_updateState) Task=`bdffbc9a-853c-47c3-b4b9-e1167b287fe3`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:06,647::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:06,647::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:06,647::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:06,647::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:06,648::task::1185::TaskManager.Task::(prepare) Task=`bdffbc9a-853c-47c3-b4b9-e1167b287fe3`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:06,648::task::595::TaskManager.Task::(_updateState) Task=`bdffbc9a-853c-47c3-b4b9-e1167b287fe3`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:06,648::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:06,648::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:06,648::task::990::TaskManager.Task::(_decref) Task=`bdffbc9a-853c-47c3-b4b9-e1167b287fe3`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:07,651::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:07,652::task::595::TaskManager.Task::(_updateState) Task=`3e40c0d9-aec2-4ba5-9289-ee8161a20188`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:07,652::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:07,652::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:07,653::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:07,653::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:07,653::task::1185::TaskManager.Task::(prepare) Task=`3e40c0d9-aec2-4ba5-9289-ee8161a20188`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:07,653::task::595::TaskManager.Task::(_updateState) Task=`3e40c0d9-aec2-4ba5-9289-ee8161a20188`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:07,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:07,654::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:07,654::task::990::TaskManager.Task::(_decref) Task=`3e40c0d9-aec2-4ba5-9289-ee8161a20188`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:08,657::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:08,657::task::595::TaskManager.Task::(_updateState) Task=`68842385-9985-4466-9c8a-2e2a32ee192e`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:08,658::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:08,658::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:08,658::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:08,658::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:08,658::task::1185::TaskManager.Task::(prepare) Task=`68842385-9985-4466-9c8a-2e2a32ee192e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:08,659::task::595::TaskManager.Task::(_updateState) Task=`68842385-9985-4466-9c8a-2e2a32ee192e`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:08,659::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:08,659::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:08,659::task::990::TaskManager.Task::(_decref) Task=`68842385-9985-4466-9c8a-2e2a32ee192e`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:09,664::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:09,665::task::595::TaskManager.Task::(_updateState) Task=`0fc530b3-9b0d-4ae8-9135-adbb90375b48`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:09,666::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:09,667::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:09,668::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:09,669::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:09,670::task::1185::TaskManager.Task::(prepare) Task=`0fc530b3-9b0d-4ae8-9135-adbb90375b48`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:09,671::task::595::TaskManager.Task::(_updateState) Task=`0fc530b3-9b0d-4ae8-9135-adbb90375b48`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:09,672::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:09,673::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:09,674::task::990::TaskManager.Task::(_decref) Task=`0fc530b3-9b0d-4ae8-9135-adbb90375b48`::ref 0 aborting False
Thread-22::ERROR::2014-11-28 16:17:09,738::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/nfsSD.py", line 113, in selftest
    fileSD.FileStorageDomain.selftest(self)
  File "/usr/share/vdsm/storage/fileSD.py", line 586, in selftest
    self.oop.os.statvfs(self.domaindir)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-24::DEBUG::2014-11-28 16:17:10,254::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:17:10,291::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:17:10,293::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-24::DEBUG::2014-11-28 16:17:10,314::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.100:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=474f0809-26c6-4fca-ab8c-72a2e6dcd912', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=80de07a9b66165f2715b4747ae39d51945e0db7f']
Thread-24::DEBUG::2014-11-28 16:17:10,315::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-24::INFO::2014-11-28 16:17:10,315::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_imageNS already registered
Thread-24::INFO::2014-11-28 16:17:10,316::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_volumeNS already registered
Thread-24::DEBUG::2014-11-28 16:17:10,324::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:10,341::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00128491 s, 279 kB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:17:10,679::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:10,680::task::595::TaskManager.Task::(_updateState) Task=`18d3b9bd-c2bb-437b-960b-96f0d7c9abde`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:10,680::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:10,680::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:10,680::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:10,681::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:10,681::task::1185::TaskManager.Task::(prepare) Task=`18d3b9bd-c2bb-437b-960b-96f0d7c9abde`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:10,681::task::595::TaskManager.Task::(_updateState) Task=`18d3b9bd-c2bb-437b-960b-96f0d7c9abde`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:10,681::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:10,681::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:10,682::task::990::TaskManager.Task::(_decref) Task=`18d3b9bd-c2bb-437b-960b-96f0d7c9abde`::ref 0 aborting False
Thread-13::ERROR::2014-11-28 16:17:11,025::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 16:17:11,026::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 16:17:11,026::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:17:11,026::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 16:17:11,026::task::1185::TaskManager.Task::(prepare) Task=`37d55c0e-e0a0-46dd-9f86-55b5cc5f8ce5`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 16:17:11,026::task::595::TaskManager.Task::(_updateState) Task=`37d55c0e-e0a0-46dd-9f86-55b5cc5f8ce5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:17:11,027::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:17:11,027::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:17:11,027::task::990::TaskManager.Task::(_decref) Task=`37d55c0e-e0a0-46dd-9f86-55b5cc5f8ce5`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:17:11,126::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5454a50]
Thread-13::DEBUG::2014-11-28 16:17:11,126::task::595::TaskManager.Task::(_updateState) Task=`c72585e0-8a46-4ecf-98ec-42759519976d`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:17:11,126::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:17:11,133::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-82::DEBUG::2014-11-28 16:17:11,685::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:11,686::task::595::TaskManager.Task::(_updateState) Task=`53a48c80-c656-43dd-8133-32d85442e53e`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:11,686::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:11,686::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:11,686::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:11,686::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:11,687::task::1185::TaskManager.Task::(prepare) Task=`53a48c80-c656-43dd-8133-32d85442e53e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:11,687::task::595::TaskManager.Task::(_updateState) Task=`53a48c80-c656-43dd-8133-32d85442e53e`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:11,687::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:11,687::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:11,687::task::990::TaskManager.Task::(_decref) Task=`53a48c80-c656-43dd-8133-32d85442e53e`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:12,691::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:12,691::task::595::TaskManager.Task::(_updateState) Task=`79b641b8-e523-49eb-9867-687e5d5dea5c`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:12,691::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:12,692::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:12,692::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:12,692::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:12,692::task::1185::TaskManager.Task::(prepare) Task=`79b641b8-e523-49eb-9867-687e5d5dea5c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:12,693::task::595::TaskManager.Task::(_updateState) Task=`79b641b8-e523-49eb-9867-687e5d5dea5c`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:12,693::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:12,693::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:12,693::task::990::TaskManager.Task::(_decref) Task=`79b641b8-e523-49eb-9867-687e5d5dea5c`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:13,697::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:13,699::task::595::TaskManager.Task::(_updateState) Task=`33cf71f3-3a82-47ed-9a84-5ec37f4f5285`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:13,700::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:13,700::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:13,701::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:13,702::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:13,702::task::1185::TaskManager.Task::(prepare) Task=`33cf71f3-3a82-47ed-9a84-5ec37f4f5285`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:13,703::task::595::TaskManager.Task::(_updateState) Task=`33cf71f3-3a82-47ed-9a84-5ec37f4f5285`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:13,704::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:13,705::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:13,706::task::990::TaskManager.Task::(_decref) Task=`33cf71f3-3a82-47ed-9a84-5ec37f4f5285`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:13,964::task::595::TaskManager.Task::(_updateState) Task=`668a10b1-cf54-4787-b341-45f0c20b2ff1`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:13,966::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:17:13,968::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00128491', 'lastCheck': '3.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:17:13,969::task::1185::TaskManager.Task::(prepare) Task=`668a10b1-cf54-4787-b341-45f0c20b2ff1`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00128491', 'lastCheck': '3.6', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:17:13,970::task::595::TaskManager.Task::(_updateState) Task=`668a10b1-cf54-4787-b341-45f0c20b2ff1`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:13,970::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:13,971::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:13,972::task::990::TaskManager.Task::(_decref) Task=`668a10b1-cf54-4787-b341-45f0c20b2ff1`::ref 0 aborting False
84c395bb-1362-491c-a275-d3b0766622b4::ERROR::2014-11-28 16:17:14,244::task::866::TaskManager.Task::(_setError) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 247, in startSpm
    self.updateMonitoringThreads()
  File "/usr/share/vdsm/storage/misc.py", line 757, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 742, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1385, in updateMonitoringThreads
    activeDomains = self.getDomains(activeOnly=True)
  File "/usr/share/vdsm/storage/sp.py", line 1412, in getDomains
    in self._backend.getDomainsMap().iteritems()
  File "/usr/share/vdsm/storage/spbackends.py", line 245, in getDomainsMap
    return self.getMetaParam(PMDK_DOMAINS)
  File "/usr/share/vdsm/storage/spbackends.py", line 348, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 89, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 199, in __getitem__
    with self._accessWrapper():
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper
    self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
    lines = self._metaRW.readlines()
  File "/usr/share/vdsm/storage/fileSD.py", line 118, in readlines
    return stripNewLines(self._oop.directReadLines(self._metafile))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,247::task::885::TaskManager.Task::(_run) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Task._run: 84c395bb-1362-491c-a275-d3b0766622b4 () {} failed - stopping task
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,247::task::1211::TaskManager.Task::(stop) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::stopping in state running (force False)
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,248::task::990::TaskManager.Task::(_decref) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::ref 1 aborting True
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,248::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,248::task::916::TaskManager.Task::(_runJobs) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::aborting: Task is aborted: u'' - code 100
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,248::task::990::TaskManager.Task::(_decref) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::ref 0 aborting True
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::task::925::TaskManager.Task::(_doAbort) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Task._doAbort: force False
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state running -> state aborting
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::task::550::TaskManager.Task::(__state_aborting) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::_aborting: recover policy auto
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state aborting -> state racquiring
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,249::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state racquiring -> state recovering
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,250::task::798::TaskManager.Task::(_recover) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::_recover
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,250::task::805::TaskManager.Task::(_recover) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::running recovery None
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,250::task::786::TaskManager.Task::(_recoverDone) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::Recover Done: state recovering
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,250::task::595::TaskManager.Task::(_updateState) Task=`84c395bb-1362-491c-a275-d3b0766622b4`::moving from state recovering -> state recovered
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,250::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad': < ResourceRef 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', isValid: 'True' obj: 'None'>}
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,251::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,251::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,251::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,251::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,252::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
84c395bb-1362-491c-a275-d3b0766622b4::DEBUG::2014-11-28 16:17:14,252::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-82::DEBUG::2014-11-28 16:17:14,711::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:14,711::task::595::TaskManager.Task::(_updateState) Task=`d9b85d04-c3dd-4afa-af43-5f2659990a8a`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:14,712::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:17:14,712::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:17:14,712::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': u'', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}
Thread-82::INFO::2014-11-28 16:17:14,712::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 100, 'message': u'', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:14,712::task::1185::TaskManager.Task::(prepare) Task=`d9b85d04-c3dd-4afa-af43-5f2659990a8a`::finished: {'taskStatus': {'code': 100, 'message': u'', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '84c395bb-1362-491c-a275-d3b0766622b4'}}
Thread-82::DEBUG::2014-11-28 16:17:14,713::task::595::TaskManager.Task::(_updateState) Task=`d9b85d04-c3dd-4afa-af43-5f2659990a8a`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:17:14,713::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:17:14,713::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:17:14,713::task::990::TaskManager.Task::(_decref) Task=`d9b85d04-c3dd-4afa-af43-5f2659990a8a`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:17:14,722::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:17:14,722::task::595::TaskManager.Task::(_updateState) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:17:14,723::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-22::DEBUG::2014-11-28 16:17:19,739::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-22::DEBUG::2014-11-28 16:17:19,739::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-22::DEBUG::2014-11-28 16:17:19,740::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:17:19,740::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-22::DEBUG::2014-11-28 16:17:19,740::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:17:19,740::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-22::DEBUG::2014-11-28 16:17:19,740::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-22::DEBUG::2014-11-28 16:17:19,753::misc::748::SamplingMethod::(__call__) Returning last result
Thread-22::DEBUG::2014-11-28 16:17:19,756::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-22::DEBUG::2014-11-28 16:17:19,782::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:17:19,783::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,783::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,783::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,783::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,784::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,784::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:17:19,784::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::DEBUG::2014-11-28 16:17:20,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:20,356::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00024022 s, 1.5 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:17:30,366::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:30,373::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000288401 s, 1.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:17:40,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:40,391::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.0002738 s, 1.3 MB/s\n'; <rc> = 0
Thread-111::WARNING::2014-11-28 16:17:49,512::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-111::DEBUG::2014-11-28 16:17:49,522::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-110::WARNING::2014-11-28 16:17:49,514::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-109::WARNING::2014-11-28 16:17:49,517::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-110::DEBUG::2014-11-28 16:17:49,532::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-109::DEBUG::2014-11-28 16:17:49,536::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:17:49,544::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:17:49,549::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-114::WARNING::2014-11-28 16:17:49,775::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-115::WARNING::2014-11-28 16:17:49,776::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-114::DEBUG::2014-11-28 16:17:49,784::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-115::DEBUG::2014-11-28 16:17:49,785::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-116::WARNING::2014-11-28 16:17:49,781::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-116::DEBUG::2014-11-28 16:17:49,792::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:17:49,794::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:17:49,797::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-119::WARNING::2014-11-28 16:17:49,929::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-119::DEBUG::2014-11-28 16:17:49,933::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-121::WARNING::2014-11-28 16:17:49,935::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-121::DEBUG::2014-11-28 16:17:49,937::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-120::WARNING::2014-11-28 16:17:49,936::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-120::DEBUG::2014-11-28 16:17:49,938::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-17::ERROR::2014-11-28 16:17:49,938::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:17:49,939::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:17:50,401::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:17:50,407::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000330046 s, 1.1 MB/s\n'; <rc> = 0
Thread-19::DEBUG::2014-11-28 16:17:59,555::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-18::DEBUG::2014-11-28 16:17:59,801::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:17:59,801::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:17:59,802::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:17:59,802::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:17:59,803::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:17:59,826::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:17:59,827::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:17:59,827::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 16:17:59,939::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:17:59,940::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:17:59,940::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:17:59,940::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:17:59,941::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:17:59,977::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:17:59,978::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:17:59,978::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:18:00,417::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:00,423::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000247837 s, 1.4 MB/s\n'; <rc> = 0
Thread-126::WARNING::2014-11-28 16:18:10,280::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-126::DEBUG::2014-11-28 16:18:10,283::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-125::WARNING::2014-11-28 16:18:10,281::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-125::DEBUG::2014-11-28 16:18:10,286::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-124::WARNING::2014-11-28 16:18:10,294::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-124::DEBUG::2014-11-28 16:18:10,296::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-24::DEBUG::2014-11-28 16:18:10,468::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:10,476::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000245367 s, 1.5 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:18:11,134::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'474f0809-26c6-4fca-ab8c-72a2e6dcd912': <function findDomain at 0x10c7230>, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x10c7230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x10c7230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 16:18:11,134::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:18:11,135::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:18:11,135::task::1185::TaskManager.Task::(prepare) Task=`c72585e0-8a46-4ecf-98ec-42759519976d`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:18:11,135::task::595::TaskManager.Task::(_updateState) Task=`c72585e0-8a46-4ecf-98ec-42759519976d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:18:11,135::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:18:11,136::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:18:11,136::task::990::TaskManager.Task::(_decref) Task=`c72585e0-8a46-4ecf-98ec-42759519976d`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:18:11,190::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [49a145fe]
Thread-13::DEBUG::2014-11-28 16:18:11,190::task::595::TaskManager.Task::(_updateState) Task=`f31947ce-15d7-43f5-b663-2e56f1df7994`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:18:11,191::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:18:11,198::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-82::ERROR::2014-11-28 16:18:14,729::task::866::TaskManager.Task::(_setError) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 631, in getSpmStatus
    status = self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 625, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 223, in getSpmStatus
    if (poolMeta[PMDK_SPM_ID] == self.id
  File "/usr/share/vdsm/storage/persistentDict.py", line 89, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 199, in __getitem__
    with self._accessWrapper():
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper
    self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
    lines = self._metaRW.readlines()
  File "/usr/share/vdsm/storage/fileSD.py", line 118, in readlines
    return stripNewLines(self._oop.directReadLines(self._metafile))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:18:14,734::task::885::TaskManager.Task::(_run) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::Task._run: dff90f43-869f-4c8b-a237-6acbf0efe442 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-82::DEBUG::2014-11-28 16:18:14,734::task::1211::TaskManager.Task::(stop) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::stopping in state preparing (force False)
Thread-82::DEBUG::2014-11-28 16:18:14,734::task::990::TaskManager.Task::(_decref) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::ref 1 aborting True
Thread-82::DEBUG::2014-11-28 16:18:14,734::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::INFO::2014-11-28 16:18:14,735::task::1168::TaskManager.Task::(prepare) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::aborting: Task is aborted: u'' - code 100
Thread-82::DEBUG::2014-11-28 16:18:14,735::task::1173::TaskManager.Task::(prepare) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::Prepare: aborted: 
Thread-82::DEBUG::2014-11-28 16:18:14,735::task::990::TaskManager.Task::(_decref) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::ref 0 aborting True
Thread-82::DEBUG::2014-11-28 16:18:14,735::task::925::TaskManager.Task::(_doAbort) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::Task._doAbort: force False
Thread-82::DEBUG::2014-11-28 16:18:14,736::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:18:14,736::task::595::TaskManager.Task::(_updateState) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::moving from state preparing -> state aborting
Thread-82::DEBUG::2014-11-28 16:18:14,736::task::550::TaskManager.Task::(__state_aborting) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::_aborting: recover policy none
Thread-82::DEBUG::2014-11-28 16:18:14,736::task::595::TaskManager.Task::(_updateState) Task=`dff90f43-869f-4c8b-a237-6acbf0efe442`::moving from state aborting -> state failed
Thread-82::DEBUG::2014-11-28 16:18:14,736::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:18:14,737::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::ERROR::2014-11-28 16:18:14,737::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-82::DEBUG::2014-11-28 16:18:14,755::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2c01d796]
Thread-82::DEBUG::2014-11-28 16:18:14,755::task::595::TaskManager.Task::(_updateState) Task=`34166fad-ba60-4a12-8447-1eb6b8c22f2b`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:18:14,756::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='84c395bb-1362-491c-a275-d3b0766622b4', spUUID=None, options=None)
Thread-82::DEBUG::2014-11-28 16:18:14,756::taskManager::161::TaskManager::(clearTask) Entry. taskID: 84c395bb-1362-491c-a275-d3b0766622b4
Thread-82::DEBUG::2014-11-28 16:18:14,756::taskManager::166::TaskManager::(clearTask) Return.
Thread-82::INFO::2014-11-28 16:18:14,756::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-82::DEBUG::2014-11-28 16:18:14,756::task::1185::TaskManager.Task::(prepare) Task=`34166fad-ba60-4a12-8447-1eb6b8c22f2b`::finished: None
Thread-82::DEBUG::2014-11-28 16:18:14,756::task::595::TaskManager.Task::(_updateState) Task=`34166fad-ba60-4a12-8447-1eb6b8c22f2b`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:18:14,757::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:18:14,757::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:18:14,757::task::990::TaskManager.Task::(_decref) Task=`34166fad-ba60-4a12-8447-1eb6b8c22f2b`::ref 0 aborting False
Thread-131::WARNING::2014-11-28 16:18:19,804::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-129::WARNING::2014-11-28 16:18:19,804::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-131::DEBUG::2014-11-28 16:18:19,807::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-129::DEBUG::2014-11-28 16:18:19,808::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-130::WARNING::2014-11-28 16:18:19,809::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-130::DEBUG::2014-11-28 16:18:19,810::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-22::ERROR::2014-11-28 16:18:19,810::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:18:19,811::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-24::DEBUG::2014-11-28 16:18:20,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:20,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000293236 s, 1.2 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:18:23,778::task::595::TaskManager.Task::(_updateState) Task=`4487a10c-3860-4dd6-86f5-d6138a2c9823`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:18:23,778::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:18:23,779::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000293236', 'lastCheck': '3.3', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:18:23,779::task::1185::TaskManager.Task::(prepare) Task=`4487a10c-3860-4dd6-86f5-d6138a2c9823`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000293236', 'lastCheck': '3.3', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:18:23,779::task::595::TaskManager.Task::(_updateState) Task=`4487a10c-3860-4dd6-86f5-d6138a2c9823`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:18:23,779::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:18:23,779::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:18:23,780::task::990::TaskManager.Task::(_decref) Task=`4487a10c-3860-4dd6-86f5-d6138a2c9823`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:18:24,800::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-82::DEBUG::2014-11-28 16:18:24,800::task::595::TaskManager.Task::(_updateState) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:18:24,800::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-82::DEBUG::2014-11-28 16:18:24,801::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-82::DEBUG::2014-11-28 16:18:24,801::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-82::DEBUG::2014-11-28 16:18:24,801::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-82::DEBUG::2014-11-28 16:18:24,801::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-82::DEBUG::2014-11-28 16:18:24,801::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-82::DEBUG::2014-11-28 16:18:24,802::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-82::DEBUG::2014-11-28 16:18:24,814::misc::748::SamplingMethod::(__call__) Returning last result
Thread-82::DEBUG::2014-11-28 16:18:24,816::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-82::DEBUG::2014-11-28 16:18:24,841::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:18:24,841::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,841::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,842::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,842::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,843::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,843::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-82::DEBUG::2014-11-28 16:18:24,843::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::DEBUG::2014-11-28 16:18:30,506::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:30,512::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000407731 s, 880 kB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:18:40,522::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:40,529::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000337278 s, 1.1 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:18:50,539::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:18:50,545::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000231347 s, 1.6 MB/s\n'; <rc> = 0
Thread-135::WARNING::2014-11-28 16:18:59,577::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-134::WARNING::2014-11-28 16:18:59,582::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-134::DEBUG::2014-11-28 16:18:59,594::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-135::DEBUG::2014-11-28 16:18:59,593::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-133::WARNING::2014-11-28 16:18:59,581::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-19::ERROR::2014-11-28 16:18:59,608::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:18:59,613::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-138::WARNING::2014-11-28 16:18:59,851::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-139::WARNING::2014-11-28 16:18:59,852::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-138::DEBUG::2014-11-28 16:18:59,856::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-137::WARNING::2014-11-28 16:18:59,853::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-139::DEBUG::2014-11-28 16:18:59,861::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-137::DEBUG::2014-11-28 16:18:59,862::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:18:59,866::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:18:59,869::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-141::WARNING::2014-11-28 16:18:59,986::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-141::DEBUG::2014-11-28 16:18:59,989::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-142::WARNING::2014-11-28 16:19:00,008::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-143::WARNING::2014-11-28 16:19:00,008::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:19:00,009::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:19:00,009::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:19:00,555::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:00,562::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000364709 s, 984 kB/s\n'; <rc> = 0
Thread-18::ERROR::2014-11-28 16:19:09,873::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:19:09,873::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:19:09,873::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:19:09,875::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:19:09,895::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:19:09,896::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:19:09,896::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:19:10,010::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:19:10,010::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:19:10,011::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:19:10,011::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:19:10,046::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:19:10,048::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:19:10,049::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:19:10,594::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:10,601::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000244406 s, 1.5 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:19:11,204::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'474f0809-26c6-4fca-ab8c-72a2e6dcd912': <function findDomain at 0x10c7230>, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x10c7230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x10c7230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 16:19:11,204::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:19:11,205::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 16:19:11,205::task::1185::TaskManager.Task::(prepare) Task=`f31947ce-15d7-43f5-b663-2e56f1df7994`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 16:19:11,205::task::595::TaskManager.Task::(_updateState) Task=`f31947ce-15d7-43f5-b663-2e56f1df7994`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:19:11,206::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:19:11,206::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:19:11,206::task::990::TaskManager.Task::(_decref) Task=`f31947ce-15d7-43f5-b663-2e56f1df7994`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:19:20,615::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:20,621::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000413564 s, 868 kB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:19:23,253::task::595::TaskManager.Task::(_updateState) Task=`b35a851c-0961-40fd-9813-8804ab343ebe`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:19:23,253::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:19:23,253::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.2', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.4', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413564', 'lastCheck': '2.6', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:19:23,254::task::1185::TaskManager.Task::(prepare) Task=`b35a851c-0961-40fd-9813-8804ab343ebe`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.2', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.4', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413564', 'lastCheck': '2.6', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:19:23,254::task::595::TaskManager.Task::(_updateState) Task=`b35a851c-0961-40fd-9813-8804ab343ebe`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:19:23,254::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:19:23,254::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:19:23,255::task::990::TaskManager.Task::(_decref) Task=`b35a851c-0961-40fd-9813-8804ab343ebe`::ref 0 aborting False
Thread-145::WARNING::2014-11-28 16:19:24,863::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-147::WARNING::2014-11-28 16:19:24,863::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-146::WARNING::2014-11-28 16:19:24,865::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-147::DEBUG::2014-11-28 16:19:24,866::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::ERROR::2014-11-28 16:19:24,867::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 16:19:24,868::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-82::ERROR::2014-11-28 16:19:24,868::hsm::637::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 631, in getSpmStatus
    status = self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 625, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 219, in getSpmStatus
    poolMeta = self._getPoolMD(self.masterDomain)
  File "/usr/share/vdsm/storage/spbackends.py", line 337, in _getPoolMD
    return DictValidator(domain._metadata._dict, SP_MD_FIELDS)
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 16:19:24,869::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:19:24,870::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-82::ERROR::2014-11-28 16:19:24,869::task::866::TaskManager.Task::(_setError) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 638, in getSpmStatus
    raise se.StorageDomainMasterError("MD read error")
StorageDomainMasterError: Error validating master storage domain: ('MD read error',)
Thread-22::DEBUG::2014-11-28 16:19:24,870::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-82::DEBUG::2014-11-28 16:19:24,871::task::885::TaskManager.Task::(_run) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::Task._run: 7632ae79-6ac3-41be-ac5d-87b49dcca9f1 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 16:19:24,871::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-82::DEBUG::2014-11-28 16:19:24,872::task::1211::TaskManager.Task::(stop) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 16:19:24,873::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-82::DEBUG::2014-11-28 16:19:24,873::task::990::TaskManager.Task::(_decref) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::ref 1 aborting True
Thread-82::INFO::2014-11-28 16:19:24,880::task::1168::TaskManager.Task::(prepare) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::aborting: Task is aborted: 'Error validating master storage domain' - code 355
Thread-82::DEBUG::2014-11-28 16:19:24,880::task::1173::TaskManager.Task::(prepare) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::Prepare: aborted: Error validating master storage domain
Thread-82::DEBUG::2014-11-28 16:19:24,880::task::990::TaskManager.Task::(_decref) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::ref 0 aborting True
Thread-82::DEBUG::2014-11-28 16:19:24,880::task::925::TaskManager.Task::(_doAbort) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::Task._doAbort: force False
Thread-82::DEBUG::2014-11-28 16:19:24,880::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:19:24,881::task::595::TaskManager.Task::(_updateState) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::moving from state preparing -> state aborting
Thread-82::DEBUG::2014-11-28 16:19:24,881::task::550::TaskManager.Task::(__state_aborting) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::_aborting: recover policy none
Thread-82::DEBUG::2014-11-28 16:19:24,881::task::595::TaskManager.Task::(_updateState) Task=`7632ae79-6ac3-41be-ac5d-87b49dcca9f1`::moving from state aborting -> state failed
Thread-82::DEBUG::2014-11-28 16:19:24,881::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:19:24,881::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::ERROR::2014-11-28 16:19:24,882::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}}
Thread-22::DEBUG::2014-11-28 16:19:24,885::misc::748::SamplingMethod::(__call__) Returning last result
Thread-22::DEBUG::2014-11-28 16:19:24,886::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-82::DEBUG::2014-11-28 16:19:24,892::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-82::DEBUG::2014-11-28 16:19:24,893::task::595::TaskManager.Task::(_updateState) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:19:24,893::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-82::ERROR::2014-11-28 16:19:24,893::task::866::TaskManager.Task::(_setError) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2125, in getAllTasksStatuses
    allTasksStatus = sp.getAllTasksStatuses()
  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
    raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-82::DEBUG::2014-11-28 16:19:24,894::task::885::TaskManager.Task::(_run) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::Task._run: 9b77c7bd-f0f9-43a0-87ee-f0f8900c8690 () {} failed - stopping task
Thread-82::DEBUG::2014-11-28 16:19:24,894::task::1211::TaskManager.Task::(stop) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::stopping in state preparing (force False)
Thread-82::DEBUG::2014-11-28 16:19:24,894::task::990::TaskManager.Task::(_decref) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::ref 1 aborting True
Thread-82::INFO::2014-11-28 16:19:24,894::task::1168::TaskManager.Task::(prepare) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-82::DEBUG::2014-11-28 16:19:24,895::task::1173::TaskManager.Task::(prepare) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::Prepare: aborted: Secured object is not in safe state
Thread-82::DEBUG::2014-11-28 16:19:24,895::task::990::TaskManager.Task::(_decref) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::ref 0 aborting True
Thread-82::DEBUG::2014-11-28 16:19:24,895::task::925::TaskManager.Task::(_doAbort) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::Task._doAbort: force False
Thread-82::DEBUG::2014-11-28 16:19:24,895::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:19:24,895::task::595::TaskManager.Task::(_updateState) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::moving from state preparing -> state aborting
Thread-82::DEBUG::2014-11-28 16:19:24,896::task::550::TaskManager.Task::(__state_aborting) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::_aborting: recover policy none
Thread-82::DEBUG::2014-11-28 16:19:24,896::task::595::TaskManager.Task::(_updateState) Task=`9b77c7bd-f0f9-43a0-87ee-f0f8900c8690`::moving from state aborting -> state failed
Thread-82::DEBUG::2014-11-28 16:19:24,896::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:19:24,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::ERROR::2014-11-28 16:19:24,896::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
SecureError: Secured object is not in safe state
Thread-22::DEBUG::2014-11-28 16:19:24,909::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:19:24,910::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,910::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,910::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,911::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,911::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,911::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:19:24,911::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::DEBUG::2014-11-28 16:19:30,631::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:30,638::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000428352 s, 838 kB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:19:38,332::task::595::TaskManager.Task::(_updateState) Task=`13689c1f-7ff6-4d09-b6d7-94e75ea26ce0`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:19:38,332::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:19:38,332::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.3', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '78.5', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428352', 'lastCheck': '7.7', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:19:38,333::task::1185::TaskManager.Task::(prepare) Task=`13689c1f-7ff6-4d09-b6d7-94e75ea26ce0`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.3', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.7', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '78.5', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428352', 'lastCheck': '7.7', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:19:38,333::task::595::TaskManager.Task::(_updateState) Task=`13689c1f-7ff6-4d09-b6d7-94e75ea26ce0`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:19:38,333::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:19:38,333::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:19:38,333::task::990::TaskManager.Task::(_decref) Task=`13689c1f-7ff6-4d09-b6d7-94e75ea26ce0`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:19:40,648::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:40,654::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00022345 s, 1.6 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:19:50,664::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:19:50,670::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000373274 s, 962 kB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:19:53,407::task::595::TaskManager.Task::(_updateState) Task=`2d6c3379-493a-4069-b85b-1831623ab517`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:19:53,408::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-82::INFO::2014-11-28 16:19:53,408::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '93.6', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000373274', 'lastCheck': '2.7', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:19:53,408::task::1185::TaskManager.Task::(prepare) Task=`2d6c3379-493a-4069-b85b-1831623ab517`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.8', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '93.6', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000373274', 'lastCheck': '2.7', 'valid': True}}
Thread-82::DEBUG::2014-11-28 16:19:53,408::task::595::TaskManager.Task::(_updateState) Task=`2d6c3379-493a-4069-b85b-1831623ab517`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:19:53,409::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:19:53,409::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:19:53,409::task::990::TaskManager.Task::(_decref) Task=`2d6c3379-493a-4069-b85b-1831623ab517`::ref 0 aborting False
Thread-82::DEBUG::2014-11-28 16:19:59,675::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [b931141]
Thread-82::DEBUG::2014-11-28 16:19:59,676::task::595::TaskManager.Task::(_updateState) Task=`6c799e2c-6fc6-4846-9b9e-83601d76f2c2`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:19:59,676::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-82::DEBUG::2014-11-28 16:19:59,684::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:00,681::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:00,688::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000243037 s, 1.5 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:20:08,484::task::595::TaskManager.Task::(_updateState) Task=`17d29727-c825-4b12-9db9-f3bb380b6486`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:20:08,485::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:20:08,487::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '108.7', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000243037', 'lastCheck': '7.8', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:08,488::task::1185::TaskManager.Task::(prepare) Task=`17d29727-c825-4b12-9db9-f3bb380b6486`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '108.7', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000243037', 'lastCheck': '7.8', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:08,488::task::595::TaskManager.Task::(_updateState) Task=`17d29727-c825-4b12-9db9-f3bb380b6486`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:20:08,489::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:20:08,489::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:20:08,490::task::990::TaskManager.Task::(_decref) Task=`17d29727-c825-4b12-9db9-f3bb380b6486`::ref 0 aborting False
Thread-149::WARNING::2014-11-28 16:20:09,638::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-150::WARNING::2014-11-28 16:20:09,641::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-149::DEBUG::2014-11-28 16:20:09,654::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-151::WARNING::2014-11-28 16:20:09,645::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-150::DEBUG::2014-11-28 16:20:09,664::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-151::DEBUG::2014-11-28 16:20:09,668::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:20:09,678::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:20:09,683::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-153::WARNING::2014-11-28 16:20:09,912::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-155::WARNING::2014-11-28 16:20:09,918::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-153::DEBUG::2014-11-28 16:20:09,919::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-154::WARNING::2014-11-28 16:20:09,919::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-154::DEBUG::2014-11-28 16:20:09,925::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:20:09,928::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:20:09,930::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-157::WARNING::2014-11-28 16:20:10,057::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-157::DEBUG::2014-11-28 16:20:10,060::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-158::WARNING::2014-11-28 16:20:10,072::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-159::WARNING::2014-11-28 16:20:10,072::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:20:10,074::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:20:10,074::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:20:10,698::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:10,704::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000301306 s, 1.2 MB/s\n'; <rc> = 0
Thread-18::ERROR::2014-11-28 16:20:19,934::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:20:19,934::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:20:19,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:20:19,936::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:20:19,957::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:20:19,958::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:20:19,958::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:20:20,075::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:20:20,075::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:20:20,075::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:20:20,076::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:20:20,113::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:20:20,114::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:20:20,114::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:20:20,715::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:20,722::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000281236 s, 1.3 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:20:23,567::task::595::TaskManager.Task::(_updateState) Task=`9eb8ca22-47d6-4a2f-91a6-59fe9c15a6c8`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:20:23,568::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:20:23,568::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '123.8', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281236', 'lastCheck': '2.8', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:23,568::task::1185::TaskManager.Task::(prepare) Task=`9eb8ca22-47d6-4a2f-91a6-59fe9c15a6c8`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.6', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.9', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '123.8', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281236', 'lastCheck': '2.8', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:23,569::task::595::TaskManager.Task::(_updateState) Task=`9eb8ca22-47d6-4a2f-91a6-59fe9c15a6c8`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:20:23,569::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:20:23,569::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:20:23,569::task::990::TaskManager.Task::(_decref) Task=`9eb8ca22-47d6-4a2f-91a6-59fe9c15a6c8`::ref 0 aborting False
Thread-163::WARNING::2014-11-28 16:20:24,930::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-161::WARNING::2014-11-28 16:20:24,932::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-162::WARNING::2014-11-28 16:20:24,931::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::ERROR::2014-11-28 16:20:24,934::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:20:24,935::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-24::DEBUG::2014-11-28 16:20:30,731::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:30,738::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00027991 s, 1.3 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:20:38,663::task::595::TaskManager.Task::(_updateState) Task=`60c539db-77df-4890-be05-052a02f88da5`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:20:38,663::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:20:38,664::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.7', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00027991', 'lastCheck': '7.9', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:38,664::task::1185::TaskManager.Task::(prepare) Task=`60c539db-77df-4890-be05-052a02f88da5`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.7', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '13.7', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00027991', 'lastCheck': '7.9', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:38,664::task::595::TaskManager.Task::(_updateState) Task=`60c539db-77df-4890-be05-052a02f88da5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:20:38,664::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:20:38,664::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:20:38,665::task::990::TaskManager.Task::(_decref) Task=`60c539db-77df-4890-be05-052a02f88da5`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:20:40,748::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:40,755::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000304809 s, 1.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:20:50,764::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:20:50,771::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000253833 s, 1.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:20:53,755::task::595::TaskManager.Task::(_updateState) Task=`8fc9e39a-31bc-49ed-b5aa-144eeb7fc4a4`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:20:53,755::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:20:53,756::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.7', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.8', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000253833', 'lastCheck': '3.0', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:53,756::task::1185::TaskManager.Task::(prepare) Task=`8fc9e39a-31bc-49ed-b5aa-144eeb7fc4a4`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.7', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '28.8', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000253833', 'lastCheck': '3.0', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:20:53,756::task::595::TaskManager.Task::(_updateState) Task=`8fc9e39a-31bc-49ed-b5aa-144eeb7fc4a4`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:20:53,756::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:20:53,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:20:53,757::task::990::TaskManager.Task::(_decref) Task=`8fc9e39a-31bc-49ed-b5aa-144eeb7fc4a4`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:21:00,781::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:00,788::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000244738 s, 1.5 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:21:08,831::task::595::TaskManager.Task::(_updateState) Task=`6f8f1989-ca2b-47d4-a80c-1ead3cad481e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:21:08,832::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:21:08,832::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000244738', 'lastCheck': '8.0', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:08,833::task::1185::TaskManager.Task::(prepare) Task=`6f8f1989-ca2b-47d4-a80c-1ead3cad481e`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.1', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000244738', 'lastCheck': '8.0', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:08,833::task::595::TaskManager.Task::(_updateState) Task=`6f8f1989-ca2b-47d4-a80c-1ead3cad481e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:21:08,833::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:21:08,834::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:21:08,834::task::990::TaskManager.Task::(_decref) Task=`6f8f1989-ca2b-47d4-a80c-1ead3cad481e`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:21:10,798::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:10,805::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000315593 s, 1.1 MB/s\n'; <rc> = 0
Thread-167::DEBUG::2014-11-28 16:21:19,713::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-165::WARNING::2014-11-28 16:21:19,712::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-166::WARNING::2014-11-28 16:21:19,713::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-167::WARNING::2014-11-28 16:21:19,719::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-166::DEBUG::2014-11-28 16:21:19,725::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-167::DEBUG::2014-11-28 16:21:19,727::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:21:19,734::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:21:19,739::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-170::WARNING::2014-11-28 16:21:19,982::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-172::WARNING::2014-11-28 16:21:19,983::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-171::WARNING::2014-11-28 16:21:19,986::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-18::ERROR::2014-11-28 16:21:19,992::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:21:19,995::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-175::WARNING::2014-11-28 16:21:20,124::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-175::DEBUG::2014-11-28 16:21:20,127::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-177::WARNING::2014-11-28 16:21:20,138::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-176::WARNING::2014-11-28 16:21:20,138::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:21:20,140::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:21:20,140::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:21:20,815::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:20,821::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000300993 s, 1.2 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:21:23,937::task::595::TaskManager.Task::(_updateState) Task=`0cca7df1-6a69-4770-9a9a-f86c1904a867`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:21:23,938::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:21:23,938::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000300993', 'lastCheck': '3.1', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:23,938::task::1185::TaskManager.Task::(prepare) Task=`0cca7df1-6a69-4770-9a9a-f86c1904a867`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.8', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.9', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.2', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.0', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000300993', 'lastCheck': '3.1', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:23,938::task::595::TaskManager.Task::(_updateState) Task=`0cca7df1-6a69-4770-9a9a-f86c1904a867`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:21:23,939::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:21:23,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:21:23,939::task::990::TaskManager.Task::(_decref) Task=`0cca7df1-6a69-4770-9a9a-f86c1904a867`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 16:21:29,998::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:21:29,998::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:21:29,999::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:21:29,999::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:21:30,019::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:21:30,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:21:30,020::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:21:30,140::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:21:30,141::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:21:30,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:21:30,142::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:21:30,170::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:21:30,171::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:21:30,171::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:21:30,831::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:30,838::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000272807 s, 1.3 MB/s\n'; <rc> = 0
Thread-182::WARNING::2014-11-28 16:21:34,958::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-182::DEBUG::2014-11-28 16:21:34,961::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-181::WARNING::2014-11-28 16:21:34,960::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-181::DEBUG::2014-11-28 16:21:34,964::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-180::WARNING::2014-11-28 16:21:34,964::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-22::ERROR::2014-11-28 16:21:34,966::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:21:34,966::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 16:21:39,040::task::595::TaskManager.Task::(_updateState) Task=`f54e8cee-2c7c-4c3f-9363-1a16312c1847`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:21:39,041::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:21:39,041::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000272807', 'lastCheck': '8.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:39,041::task::1185::TaskManager.Task::(prepare) Task=`f54e8cee-2c7c-4c3f-9363-1a16312c1847`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.9', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.0', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.3', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.1', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000272807', 'lastCheck': '8.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:39,042::task::595::TaskManager.Task::(_updateState) Task=`f54e8cee-2c7c-4c3f-9363-1a16312c1847`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:21:39,042::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:21:39,042::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:21:39,042::task::990::TaskManager.Task::(_decref) Task=`f54e8cee-2c7c-4c3f-9363-1a16312c1847`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:21:40,848::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:40,854::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00032692 s, 1.1 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:21:50,864::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:21:50,871::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000255463 s, 1.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:21:54,124::task::595::TaskManager.Task::(_updateState) Task=`1ee397c5-e6c3-43b3-b604-84d73b2e7de9`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:21:54,124::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:21:54,125::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000255463', 'lastCheck': '3.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:54,125::task::1185::TaskManager.Task::(prepare) Task=`1ee397c5-e6c3-43b3-b604-84d73b2e7de9`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.0', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.1', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.4', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000255463', 'lastCheck': '3.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:21:54,125::task::595::TaskManager.Task::(_updateState) Task=`1ee397c5-e6c3-43b3-b604-84d73b2e7de9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:21:54,126::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:21:54,126::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:21:54,126::task::990::TaskManager.Task::(_decref) Task=`1ee397c5-e6c3-43b3-b604-84d73b2e7de9`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:22:00,880::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:00,887::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000236628 s, 1.5 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:22:09,201::task::595::TaskManager.Task::(_updateState) Task=`169b3e5d-4cca-439f-904c-c3a96fc37f01`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:22:09,201::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:22:09,202::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000236628', 'lastCheck': '8.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:09,202::task::1185::TaskManager.Task::(prepare) Task=`169b3e5d-4cca-439f-904c-c3a96fc37f01`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.2', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '34.2', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000236628', 'lastCheck': '8.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:09,202::task::595::TaskManager.Task::(_updateState) Task=`169b3e5d-4cca-439f-904c-c3a96fc37f01`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:22:09,202::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:22:09,203::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:22:09,203::task::990::TaskManager.Task::(_decref) Task=`169b3e5d-4cca-439f-904c-c3a96fc37f01`::ref 0 aborting False
Thread-82::ERROR::2014-11-28 16:22:10,796::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-82::ERROR::2014-11-28 16:22:10,797::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-82::DEBUG::2014-11-28 16:22:10,797::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-82::INFO::2014-11-28 16:22:10,797::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-82::DEBUG::2014-11-28 16:22:10,798::task::1185::TaskManager.Task::(prepare) Task=`6c799e2c-6fc6-4846-9b9e-83601d76f2c2`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-82::DEBUG::2014-11-28 16:22:10,798::task::595::TaskManager.Task::(_updateState) Task=`6c799e2c-6fc6-4846-9b9e-83601d76f2c2`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:22:10,798::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:22:10,798::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:22:10,799::task::990::TaskManager.Task::(_decref) Task=`6c799e2c-6fc6-4846-9b9e-83601d76f2c2`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:22:10,889::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:22:10,889::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-24::DEBUG::2014-11-28 16:22:10,890::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-24::DEBUG::2014-11-28 16:22:10,890::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-24::DEBUG::2014-11-28 16:22:10,890::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-24::DEBUG::2014-11-28 16:22:10,890::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-24::DEBUG::2014-11-28 16:22:10,891::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-82::DEBUG::2014-11-28 16:22:10,899::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [286d6c39]
Thread-82::DEBUG::2014-11-28 16:22:10,900::task::595::TaskManager.Task::(_updateState) Task=`3693e886-36ad-4fc3-81f3-aab86f5fae38`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:22:10,900::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-24::DEBUG::2014-11-28 16:22:10,904::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::DEBUG::2014-11-28 16:22:10,908::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-82::DEBUG::2014-11-28 16:22:10,918::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.98:_mnt_storage
Thread-24::DEBUG::2014-11-28 16:22:10,933::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:22:10,934::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,934::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,934::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,934::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,935::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,935::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:10,935::misc::748::SamplingMethod::(__call__) Returning last result
Thread-24::DEBUG::2014-11-28 16:22:10,973::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:22:11,025::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-24::DEBUG::2014-11-28 16:22:11,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.100:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=474f0809-26c6-4fca-ab8c-72a2e6dcd912', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=80de07a9b66165f2715b4747ae39d51945e0db7f']
Thread-24::DEBUG::2014-11-28 16:22:11,083::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-24::INFO::2014-11-28 16:22:11,083::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_imageNS already registered
Thread-24::INFO::2014-11-28 16:22:11,083::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_volumeNS already registered
Thread-24::DEBUG::2014-11-28 16:22:11,091::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:11,100::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.00207095 s, 173 kB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:22:21,111::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:21,118::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000262349 s, 1.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:22:24,278::task::595::TaskManager.Task::(_updateState) Task=`5dc87f98-b501-45e9-b488-9b091c8e7711`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:22:24,279::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:22:24,279::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.3', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262349', 'lastCheck': '3.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:24,280::task::1185::TaskManager.Task::(prepare) Task=`5dc87f98-b501-45e9-b488-9b091c8e7711`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.1', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.5', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '49.3', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000262349', 'lastCheck': '3.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:24,280::task::595::TaskManager.Task::(_updateState) Task=`5dc87f98-b501-45e9-b488-9b091c8e7711`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:22:24,280::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:22:24,280::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:22:24,280::task::990::TaskManager.Task::(_decref) Task=`5dc87f98-b501-45e9-b488-9b091c8e7711`::ref 0 aborting False
Thread-185::WARNING::2014-11-28 16:22:29,762::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-187::WARNING::2014-11-28 16:22:29,768::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-185::DEBUG::2014-11-28 16:22:29,774::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-186::WARNING::2014-11-28 16:22:29,770::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-187::DEBUG::2014-11-28 16:22:29,779::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-186::DEBUG::2014-11-28 16:22:29,783::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:22:29,790::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:22:29,794::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-190::WARNING::2014-11-28 16:22:30,037::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-191::WARNING::2014-11-28 16:22:30,038::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-190::DEBUG::2014-11-28 16:22:30,043::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-191::DEBUG::2014-11-28 16:22:30,044::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-192::WARNING::2014-11-28 16:22:30,043::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-192::DEBUG::2014-11-28 16:22:30,050::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:22:30,053::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:22:30,057::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-195::WARNING::2014-11-28 16:22:30,183::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-195::DEBUG::2014-11-28 16:22:30,186::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-196::WARNING::2014-11-28 16:22:30,197::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-197::WARNING::2014-11-28 16:22:30,197::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:22:30,198::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:22:30,199::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:22:31,128::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:31,134::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000288308 s, 1.2 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:22:39,371::task::595::TaskManager.Task::(_updateState) Task=`f06bc984-9ada-488d-9e94-7cf6f7809e39`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:22:39,371::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:22:39,372::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.4', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000288308', 'lastCheck': '8.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:39,372::task::1185::TaskManager.Task::(prepare) Task=`f06bc984-9ada-488d-9e94-7cf6f7809e39`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.3', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.6', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '64.4', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000288308', 'lastCheck': '8.2', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:22:39,372::task::595::TaskManager.Task::(_updateState) Task=`f06bc984-9ada-488d-9e94-7cf6f7809e39`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:22:39,372::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:22:39,372::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:22:39,373::task::990::TaskManager.Task::(_decref) Task=`f06bc984-9ada-488d-9e94-7cf6f7809e39`::ref 0 aborting False
Thread-18::ERROR::2014-11-28 16:22:40,059::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:22:40,060::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:22:40,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:22:40,061::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:22:40,086::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:22:40,088::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:22:40,088::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:22:40,199::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:22:40,200::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:22:40,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:22:40,201::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:22:40,239::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:22:40,239::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:22:40,240::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:22:41,145::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:41,152::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000380781 s, 943 kB/s\n'; <rc> = 0
Thread-201::DEBUG::2014-11-28 16:22:44,992::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-201::WARNING::2014-11-28 16:22:44,993::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-200::WARNING::2014-11-28 16:22:44,993::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-201::DEBUG::2014-11-28 16:22:44,994::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-202::WARNING::2014-11-28 16:22:44,992::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-200::DEBUG::2014-11-28 16:22:44,996::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-22::ERROR::2014-11-28 16:22:44,997::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:22:44,997::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 16:22:48,865::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1f79d02e]
Thread-13::DEBUG::2014-11-28 16:22:48,866::task::595::TaskManager.Task::(_updateState) Task=`c720fd40-c862-4063-a822-a10770ec6903`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:22:48,866::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-24::DEBUG::2014-11-28 16:22:51,161::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:22:51,168::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000282776 s, 1.3 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:22:54,998::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-24::DEBUG::2014-11-28 16:23:01,178::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:01,184::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000262358 s, 1.4 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:23:10,925::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'474f0809-26c6-4fca-ab8c-72a2e6dcd912': <function findDomain at 0x10c7230>, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x10c7230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x10c7230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:23:10,928::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-82::DEBUG::2014-11-28 16:23:10,929::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::INFO::2014-11-28 16:23:10,932::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-82::DEBUG::2014-11-28 16:23:10,933::task::1185::TaskManager.Task::(prepare) Task=`3693e886-36ad-4fc3-81f3-aab86f5fae38`::finished: {'statuslist': [{'status': 0, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-82::DEBUG::2014-11-28 16:23:10,935::task::595::TaskManager.Task::(_updateState) Task=`3693e886-36ad-4fc3-81f3-aab86f5fae38`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:23:10,937::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:23:10,940::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:23:10,943::task::990::TaskManager.Task::(_decref) Task=`3693e886-36ad-4fc3-81f3-aab86f5fae38`::ref 0 aborting False
Thread-205::WARNING::2014-11-28 16:23:10,952::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-206::DEBUG::2014-11-28 16:23:10,954::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-205::DEBUG::2014-11-28 16:23:10,955::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-206::WARNING::2014-11-28 16:23:10,956::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-206::DEBUG::2014-11-28 16:23:10,957::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::DEBUG::2014-11-28 16:23:10,991::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [27278bf3]
Thread-207::WARNING::2014-11-28 16:23:10,991::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:23:10,992::task::595::TaskManager.Task::(_updateState) Task=`6cad1522-429f-4366-9389-0b51cc4a3574`::moving from state init -> state preparing
Thread-207::DEBUG::2014-11-28 16:23:10,993::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-82::INFO::2014-11-28 16:23:10,993::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-82::DEBUG::2014-11-28 16:23:11,001::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-24::DEBUG::2014-11-28 16:23:11,194::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:11,201::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000426629 s, 841 kB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:23:21,211::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:21,217::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000259311 s, 1.4 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:23:31,228::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:31,235::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000367521 s, 977 kB/s\n'; <rc> = 0
Thread-210::WARNING::2014-11-28 16:23:39,821::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-211::DEBUG::2014-11-28 16:23:39,825::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-210::DEBUG::2014-11-28 16:23:39,836::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-209::WARNING::2014-11-28 16:23:39,830::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-209::DEBUG::2014-11-28 16:23:39,844::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-211::WARNING::2014-11-28 16:23:39,842::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-211::DEBUG::2014-11-28 16:23:39,856::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:23:39,865::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:23:39,869::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-215::WARNING::2014-11-28 16:23:40,109::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-213::WARNING::2014-11-28 16:23:40,109::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-215::DEBUG::2014-11-28 16:23:40,113::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-213::DEBUG::2014-11-28 16:23:40,114::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-214::WARNING::2014-11-28 16:23:40,111::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-214::DEBUG::2014-11-28 16:23:40,119::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:23:40,124::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:23:40,127::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-217::WARNING::2014-11-28 16:23:40,250::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-217::DEBUG::2014-11-28 16:23:40,254::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-218::WARNING::2014-11-28 16:23:40,256::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-219::WARNING::2014-11-28 16:23:40,261::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:23:40,262::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:23:40,262::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:23:41,245::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:41,251::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000276662 s, 1.3 MB/s\n'; <rc> = 0
Thread-222::WARNING::2014-11-28 16:23:48,881::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-221::WARNING::2014-11-28 16:23:48,881::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-221::DEBUG::2014-11-28 16:23:48,884::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-223::WARNING::2014-11-28 16:23:48,892::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::ERROR::2014-11-28 16:23:48,893::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 16:23:48,893::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::ERROR::2014-11-28 16:23:48,893::hsm::637::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 631, in getSpmStatus
    status = self._getSpmStatusInfo(pool)
  File "/usr/share/vdsm/storage/hsm.py", line 625, in _getSpmStatusInfo
    (pool.spmRole,) + pool.getSpmStatus()))
  File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
    return self._backend.getSpmStatus()
  File "/usr/share/vdsm/storage/spbackends.py", line 219, in getSpmStatus
    poolMeta = self._getPoolMD(self.masterDomain)
  File "/usr/share/vdsm/storage/spbackends.py", line 337, in _getPoolMD
    return DictValidator(domain._metadata._dict, SP_MD_FIELDS)
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::DEBUG::2014-11-28 16:23:48,894::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:23:48,895::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::ERROR::2014-11-28 16:23:48,894::task::866::TaskManager.Task::(_setError) Task=`c720fd40-c862-4063-a822-a10770ec6903`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 638, in getSpmStatus
    raise se.StorageDomainMasterError("MD read error")
StorageDomainMasterError: Error validating master storage domain: ('MD read error',)
Thread-22::DEBUG::2014-11-28 16:23:48,895::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:23:48,895::task::885::TaskManager.Task::(_run) Task=`c720fd40-c862-4063-a822-a10770ec6903`::Task._run: c720fd40-c862-4063-a822-a10770ec6903 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-22::DEBUG::2014-11-28 16:23:48,896::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 16:23:48,896::task::1211::TaskManager.Task::(stop) Task=`c720fd40-c862-4063-a822-a10770ec6903`::stopping in state preparing (force False)
Thread-22::DEBUG::2014-11-28 16:23:48,896::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 16:23:48,896::task::990::TaskManager.Task::(_decref) Task=`c720fd40-c862-4063-a822-a10770ec6903`::ref 1 aborting True
Thread-13::INFO::2014-11-28 16:23:48,897::task::1168::TaskManager.Task::(prepare) Task=`c720fd40-c862-4063-a822-a10770ec6903`::aborting: Task is aborted: 'Error validating master storage domain' - code 355
Thread-13::DEBUG::2014-11-28 16:23:48,902::task::1173::TaskManager.Task::(prepare) Task=`c720fd40-c862-4063-a822-a10770ec6903`::Prepare: aborted: Error validating master storage domain
Thread-13::DEBUG::2014-11-28 16:23:48,903::task::990::TaskManager.Task::(_decref) Task=`c720fd40-c862-4063-a822-a10770ec6903`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 16:23:48,903::task::925::TaskManager.Task::(_doAbort) Task=`c720fd40-c862-4063-a822-a10770ec6903`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 16:23:48,903::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:23:48,904::task::595::TaskManager.Task::(_updateState) Task=`c720fd40-c862-4063-a822-a10770ec6903`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 16:23:48,904::task::550::TaskManager.Task::(__state_aborting) Task=`c720fd40-c862-4063-a822-a10770ec6903`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 16:23:48,904::task::595::TaskManager.Task::(_updateState) Task=`c720fd40-c862-4063-a822-a10770ec6903`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 16:23:48,904::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:23:48,904::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 16:23:48,904::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}}
Thread-13::DEBUG::2014-11-28 16:23:48,907::task::595::TaskManager.Task::(_updateState) Task=`b481ce2d-0070-4f0e-a6a5-7e5b1154beb2`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:23:48,908::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:23:48,908::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276662', 'lastCheck': '7.7', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:23:48,908::task::1185::TaskManager.Task::(prepare) Task=`b481ce2d-0070-4f0e-a6a5-7e5b1154beb2`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.6', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.8', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.0', 'valid': False}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.9', 'valid': False}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000276662', 'lastCheck': '7.7', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:23:48,908::task::595::TaskManager.Task::(_updateState) Task=`b481ce2d-0070-4f0e-a6a5-7e5b1154beb2`::moving from state preparing -> state finished
Thread-22::DEBUG::2014-11-28 16:23:48,909::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:23:48,909::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:23:48,910::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:23:48,910::task::990::TaskManager.Task::(_decref) Task=`b481ce2d-0070-4f0e-a6a5-7e5b1154beb2`::ref 0 aborting False
Thread-22::DEBUG::2014-11-28 16:23:48,911::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-22::DEBUG::2014-11-28 16:23:48,943::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:23:48,943::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,944::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,944::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,944::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,944::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,945::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:23:48,945::misc::748::SamplingMethod::(__call__) Returning last result
Thread-19::DEBUG::2014-11-28 16:23:49,875::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-18::DEBUG::2014-11-28 16:23:50,130::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:23:50,130::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:23:50,130::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:23:50,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:23:50,132::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:23:50,154::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:23:50,155::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:23:50,155::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 16:23:50,263::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:23:50,263::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:23:50,264::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:23:50,264::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:23:50,265::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:23:50,299::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:23:50,300::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:23:50,300::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-24::DEBUG::2014-11-28 16:23:51,261::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:23:51,267::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000251026 s, 1.4 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:24:01,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:01,284::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000275058 s, 1.3 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:24:11,007::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'474f0809-26c6-4fca-ab8c-72a2e6dcd912': <function findDomain at 0x10c7230>, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x10c7230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x10c7230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-82::DEBUG::2014-11-28 16:24:11,007::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-82::INFO::2014-11-28 16:24:11,007::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-82::DEBUG::2014-11-28 16:24:11,008::task::1185::TaskManager.Task::(prepare) Task=`6cad1522-429f-4366-9389-0b51cc4a3574`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-82::DEBUG::2014-11-28 16:24:11,008::task::595::TaskManager.Task::(_updateState) Task=`6cad1522-429f-4366-9389-0b51cc4a3574`::moving from state preparing -> state finished
Thread-82::DEBUG::2014-11-28 16:24:11,008::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-82::DEBUG::2014-11-28 16:24:11,008::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-82::DEBUG::2014-11-28 16:24:11,008::task::990::TaskManager.Task::(_decref) Task=`6cad1522-429f-4366-9389-0b51cc4a3574`::ref 0 aborting False
Thread-24::DEBUG::2014-11-28 16:24:11,293::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:11,300::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000306033 s, 1.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:24:21,309::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:21,315::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000288847 s, 1.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:24:31,326::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:31,332::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000303951 s, 1.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:24:41,343::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:41,350::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000294435 s, 1.2 MB/s\n'; <rc> = 0
Thread-225::DEBUG::2014-11-28 16:24:48,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-226::DEBUG::2014-11-28 16:24:48,962::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-227::WARNING::2014-11-28 16:24:48,969::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-225::WARNING::2014-11-28 16:24:48,971::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-227::DEBUG::2014-11-28 16:24:48,981::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-226::WARNING::2014-11-28 16:24:48,972::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-225::DEBUG::2014-11-28 16:24:48,986::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-226::DEBUG::2014-11-28 16:24:48,991::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-22::ERROR::2014-11-28 16:24:48,998::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:24:49,003::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-229::WARNING::2014-11-28 16:24:49,887::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-231::DEBUG::2014-11-28 16:24:49,900::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-229::DEBUG::2014-11-28 16:24:49,905::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-230::WARNING::2014-11-28 16:24:49,900::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-231::WARNING::2014-11-28 16:24:49,911::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-230::DEBUG::2014-11-28 16:24:49,918::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-231::DEBUG::2014-11-28 16:24:49,926::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:24:49,934::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:24:49,939::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-235::WARNING::2014-11-28 16:24:50,175::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-233::WARNING::2014-11-28 16:24:50,177::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-235::DEBUG::2014-11-28 16:24:50,180::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-234::WARNING::2014-11-28 16:24:50,179::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-233::DEBUG::2014-11-28 16:24:50,184::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-234::DEBUG::2014-11-28 16:24:50,186::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:24:50,189::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:24:50,191::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-237::WARNING::2014-11-28 16:24:50,312::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-237::DEBUG::2014-11-28 16:24:50,315::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-238::WARNING::2014-11-28 16:24:50,319::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-238::DEBUG::2014-11-28 16:24:50,321::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-239::WARNING::2014-11-28 16:24:50,327::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:24:50,328::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:24:50,328::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:24:51,358::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:24:51,364::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000304033 s, 1.2 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:24:59,009::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-22::DEBUG::2014-11-28 16:24:59,009::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:24:59,009::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-22::DEBUG::2014-11-28 16:24:59,009::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-22::DEBUG::2014-11-28 16:24:59,010::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-22::DEBUG::2014-11-28 16:24:59,010::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-22::DEBUG::2014-11-28 16:24:59,022::misc::748::SamplingMethod::(__call__) Returning last result
Thread-22::DEBUG::2014-11-28 16:24:59,023::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-22::DEBUG::2014-11-28 16:24:59,047::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:24:59,048::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,048::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,048::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,048::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,049::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,049::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-22::DEBUG::2014-11-28 16:24:59,049::misc::748::SamplingMethod::(__call__) Returning last result
Thread-18::ERROR::2014-11-28 16:25:00,194::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:25:00,195::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:25:00,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:25:00,196::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:25:00,217::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:25:00,218::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:25:00,218::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:25:00,329::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:25:00,329::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:25:00,329::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:25:00,330::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:25:00,365::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:25:00,366::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:25:00,367::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-82::DEBUG::2014-11-28 16:25:01,080::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [762f1608]
Thread-82::DEBUG::2014-11-28 16:25:01,081::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-82::DEBUG::2014-11-28 16:25:01,090::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:25:01,112::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7cbea28135'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.11', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.11', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0046-4710-8038-CAC04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d5', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '15e091f3-d7cd-469f-abfe-5cac567f73f9', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:12:D5', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:12:d6', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:12:d6', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:12d6/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7cbea28135', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '2a:5d:1f:fa:cd:68', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.994', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-82::DEBUG::2014-11-28 16:25:01,141::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [762f1608]
Thread-82::DEBUG::2014-11-28 16:25:01,142::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0046-4710-8038-CAC04F325031', 'systemSerialNumber': 'JFG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-82::DEBUG::2014-11-28 16:25:01,260::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [665d6e57]
Thread-82::DEBUG::2014-11-28 16:25:01,260::task::595::TaskManager.Task::(_updateState) Task=`fa925342-0199-4f1c-815f-8d41a73e5c83`::moving from state init -> state preparing
Thread-82::INFO::2014-11-28 16:25:01,261::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-82::DEBUG::2014-11-28 16:25:01,268::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso
Thread-24::DEBUG::2014-11-28 16:25:01,375::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:01,382::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000236132 s, 1.5 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:25:01,418::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-82::DEBUG::2014-11-28 16:25:01,418::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-82::DEBUG::2014-11-28 16:25:01,425::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-24::DEBUG::2014-11-28 16:25:11,422::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-24::DEBUG::2014-11-28 16:25:11,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-24::DEBUG::2014-11-28 16:25:11,427::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.100:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=474f0809-26c6-4fca-ab8c-72a2e6dcd912', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=80de07a9b66165f2715b4747ae39d51945e0db7f']
Thread-24::DEBUG::2014-11-28 16:25:11,429::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-24::INFO::2014-11-28 16:25:11,429::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_imageNS already registered
Thread-24::INFO::2014-11-28 16:25:11,429::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 474f0809-26c6-4fca-ab8c-72a2e6dcd912_volumeNS already registered
Thread-24::DEBUG::2014-11-28 16:25:11,443::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:11,459::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000411839 s, 872 kB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:25:21,470::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:21,478::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000282408 s, 1.3 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:25:31,488::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:31,495::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000272152 s, 1.3 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:25:41,505::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:41,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000278159 s, 1.3 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:25:51,525::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:25:51,534::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000278849 s, 1.3 MB/s\n'; <rc> = 0
Thread-82::DEBUG::2014-11-28 16:25:59,070::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-82::DEBUG::2014-11-28 16:25:59,074::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-241::WARNING::2014-11-28 16:25:59,065::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-243::WARNING::2014-11-28 16:25:59,072::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-241::DEBUG::2014-11-28 16:25:59,085::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-242::WARNING::2014-11-28 16:25:59,075::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-243::DEBUG::2014-11-28 16:25:59,087::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-22::ERROR::2014-11-28 16:25:59,097::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-22::ERROR::2014-11-28 16:25:59,102::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain e7eaae86-11ea-4c08-8c11-8b8c54373389 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in _monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-82::DEBUG::2014-11-28 16:25:59,174::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-247::WARNING::2014-11-28 16:25:59,967::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-246::WARNING::2014-11-28 16:25:59,970::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-247::DEBUG::2014-11-28 16:25:59,976::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-246::DEBUG::2014-11-28 16:25:59,979::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-19::ERROR::2014-11-28 16:25:59,985::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-19::ERROR::2014-11-28 16:25:59,988::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-251::WARNING::2014-11-28 16:26:00,243::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-250::DEBUG::2014-11-28 16:26:00,243::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-251::DEBUG::2014-11-28 16:26:00,246::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-250::WARNING::2014-11-28 16:26:00,247::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-250::DEBUG::2014-11-28 16:26:00,251::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-18::ERROR::2014-11-28 16:26:00,254::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-18::ERROR::2014-11-28 16:26:00,256::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-255::WARNING::2014-11-28 16:26:00,390::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-255::DEBUG::2014-11-28 16:26:00,392::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-254::WARNING::2014-11-28 16:26:00,396::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 16:26:00,397::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-17::ERROR::2014-11-28 16:26:00,397::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-24::DEBUG::2014-11-28 16:26:01,543::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:26:01,550::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000390694 s, 919 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:26:09,271::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-22::DEBUG::2014-11-28 16:26:09,418::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-22::DEBUG::2014-11-28 16:26:09,422::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24241', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f712eebad6dfcc88f8d75e665220ecff6e2c79e3']
Thread-22::DEBUG::2014-11-28 16:26:09,425::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-22::INFO::2014-11-28 16:26:09,425::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace e7eaae86-11ea-4c08-8c11-8b8c54373389_imageNS already registered
Thread-22::INFO::2014-11-28 16:26:09,425::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace e7eaae86-11ea-4c08-8c11-8b8c54373389_volumeNS already registered
Thread-22::DEBUG::2014-11-28 16:26:09,434::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:26:09,440::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n640 bytes (640 B) copied, 0.000219571 s, 2.9 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:26:09,444::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e7eaae86-11ea-4c08-8c11-8b8c54373389 changed its status to Valid
Thread-265::DEBUG::2014-11-28 16:26:09,445::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-22::INFO::2014-11-28 16:26:09,445::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 (id: 4)
Thread-265::DEBUG::2014-11-28 16:26:09,445::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-22::DEBUG::2014-11-28 16:26:09,446::clusterlock::202::SANLock::(acquireHostId) Host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 successfully acquired (id: 4)
Thread-265::DEBUG::2014-11-28 16:26:09,446::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-265::DEBUG::2014-11-28 16:26:09,448::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-267::INFO::2014-11-28 16:26:09,450::clientIF::125::vds::(contEIOVms) vmContainerLock acquired
Thread-18::ERROR::2014-11-28 16:26:10,260::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::ERROR::2014-11-28 16:26:10,260::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-18::DEBUG::2014-11-28 16:26:10,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-18::DEBUG::2014-11-28 16:26:10,261::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-18::DEBUG::2014-11-28 16:26:10,295::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-18::WARNING::2014-11-28 16:26:10,296::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-18::DEBUG::2014-11-28 16:26:10,296::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 16:26:10,398::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::ERROR::2014-11-28 16:26:10,398::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-17::DEBUG::2014-11-28 16:26:10,399::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 16:26:10,399::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-17::DEBUG::2014-11-28 16:26:10,425::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-17::WARNING::2014-11-28 16:26:10,428::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-17::DEBUG::2014-11-28 16:26:10,429::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-258::WARNING::2014-11-28 16:26:11,407::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-259::WARNING::2014-11-28 16:26:11,460::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.98:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-24::DEBUG::2014-11-28 16:26:11,560::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:26:11,566::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000325038 s, 1.1 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-11-28 16:26:19,456::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:26:19,463::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n640 bytes (640 B) copied, 0.00029159 s, 2.2 MB/s\n'; <rc> = 0
Thread-24::DEBUG::2014-11-28 16:26:21,576::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-24::DEBUG::2014-11-28 16:26:21,582::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000258759 s, 1.4 MB/s\n'; <rc> = 0