MainThread::INFO::2014-11-28 15:07:35,026::vdsm::131::vds::(run) (PID: 30837) I am the actual vdsm 4.14.17-0.el7 cti12.ctisandbox.cfcc.edu (3.10.0-123.el7.x86_64)
MainThread::DEBUG::2014-11-28 15:07:35,027::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-11-28 15:07:35,027::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::DEBUG::2014-11-28 15:07:35,515::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-11-28 15:07:35,548::multipath::151::Storage.Multipath::(isEnabled) multipath Defaulting to False
MainThread::DEBUG::2014-11-28 15:07:35,548::misc::459::Storage.Misc::(rotateFiles) dir: /etc, prefixName: multipath.conf, versions: 5
MainThread::DEBUG::2014-11-28 15:07:35,549::misc::480::Storage.Misc::(rotateFiles) versions found: []
MainThread::DEBUG::2014-11-28 15:07:35,549::multipath::169::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/cp /tmp/tmp1hp4Oa /etc/multipath.conf' (cwd None)
MainThread::DEBUG::2014-11-28 15:07:35,558::multipath::169::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:07:35,559::multipath::175::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /sbin/multipath -F' (cwd None)
MainThread::DEBUG::2014-11-28 15:07:35,570::multipath::175::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:07:35,570::multipath::178::Storage.Misc.excCmd::(setupMultipath) '/usr/bin/sudo -n /usr/bin/vdsm-tool service-reload multipathd' (cwd None)
MainThread::DEBUG::2014-11-28 15:07:35,861::multipath::178::Storage.Misc.excCmd::(setupMultipath) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:07:35,862::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-11-28 15:07:35,881::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 15:07:35,882::hsm::458::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-11-28 15:07:35,887::hsm::490::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-11-28 15:07:35,887::hsm::491::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-11-28 15:07:35,887::hsm::493::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-11-28 15:07:35,888::hsm::536::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-11-28 15:07:35,889::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
storageRefresh::DEBUG::2014-11-28 15:07:35,889::misc::740::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-11-28 15:07:35,889::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
storageRefresh::DEBUG::2014-11-28 15:07:35,889::misc::740::SamplingMethod::(__call__) Got in to sampling method
MainThread::INFO::2014-11-28 15:07:35,891::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-11-28 15:07:35,891::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
storageRefresh::DEBUG::2014-11-28 15:07:35,912::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
MainThread::DEBUG::2014-11-28 15:07:35,949::task::595::TaskManager.Task::(_updateState) Task=`5f4f020e-ca83-4bdb-a899-65ad5985c5d1`::moving from state init -> state preparing
storageRefresh::DEBUG::2014-11-28 15:07:35,949::misc::748::SamplingMethod::(__call__) Returning last result
MainThread::INFO::2014-11-28 15:07:35,950::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x2a947e8>>)
MainThread::INFO::2014-11-28 15:07:35,950::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
MainThread::DEBUG::2014-11-28 15:07:35,950::task::1185::TaskManager.Task::(prepare) Task=`5f4f020e-ca83-4bdb-a899-65ad5985c5d1`::finished: None
MainThread::DEBUG::2014-11-28 15:07:35,950::task::595::TaskManager.Task::(_updateState) Task=`5f4f020e-ca83-4bdb-a899-65ad5985c5d1`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 15:07:35,951::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 15:07:35,951::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 15:07:35,951::task::990::TaskManager.Task::(_decref) Task=`5f4f020e-ca83-4bdb-a899-65ad5985c5d1`::ref 0 aborting False
MainThread::INFO::2014-11-28 15:07:35,952::momIF::49::MOM::(__init__) Starting up MOM
storageRefresh::DEBUG::2014-11-28 15:07:35,953::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
MainThread::INFO::2014-11-28 15:07:35,959::vmChannels::191::vds::(settimeout) Setting channels' timeout to 30 seconds.
clientIFinit::DEBUG::2014-11-28 15:07:35,976::libvirtconnection::145::root::(get) trying to connect libvirt
VM Channels Listener::INFO::2014-11-28 15:07:35,976::vmChannels::174::vds::(run) Starting VM channels listener thread.
MainThread::WARNING::2014-11-28 15:07:35,981::clientIF::180::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-11-28 15:07:35,982::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:07:35,983::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,983::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,983::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,983::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,984::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,984::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,985::misc::748::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-11-28 15:07:35,985::lvm::325::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:35,987::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:07:36,016::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:07:36,018::lvm::350::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:36,019::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:36,027::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:07:36,049::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:07:36,052::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:36,056::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:07:36,076::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:07:36,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:36,081::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 15:07:36,101::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 15:07:36,102::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 15:07:36,104::hsm::393::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-11-28 15:07:36,344::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [6445607c]
Thread-13::DEBUG::2014-11-28 15:07:36,345::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:07:36,356::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [6445607c]
Thread-13::DEBUG::2014-11-28 15:07:36,393::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:36,407::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:07:36,442::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1404144562.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {}, 'bridges': {';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0047-4710-8038-B1C04F325031', 'lastClientIface': 'em2', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:25', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '8f04642b-5fd9-4fdc-9884-9b4187d3e718', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:13:25', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '255.255.0.0', 'addr': '172.16.0.12', 'hwaddr': 'b8:ac:6f:91:13:26', 'cfg': {'IPV6_PEERDNS': 'yes', 'IPV6INIT': 'yes', 'UUID': '01e33ef8-bd9d-4966-aa81-ecb41a479ec7', 'IPADDR': '172.16.0.12', 'DNS1': '10.255.100.3', 'GATEWAY': '172.16.0.1', 'DEFROUTE': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'HWADDR': 'B8:AC:6F:91:13:26', 'PREFIX': '16', 'BOOTPROTO': 'none', 'IPV6_DEFROUTE': 'yes', 'DNS2': '10.255.100.1', 'IPV6_AUTOCONF': 'yes', 'IPV6_FAILURE_FATAL': 'no', 'IPV6_PEERROUTES': 'yes', 'TYPE': 'Ethernet', 'ONBOOT': 'yes', 'NAME': 'em2'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'speed': 1000, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '6e:ac:d1:90:02:62', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.898', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:07:36,630::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setupNetworks with ({'ovirtmgmt': {'nic': 'em2', 'ipaddr': '172.16.0.12', 'mtu': '9000', 'netmask': '255.255.0.0', 'STP': 'no', 'bridged': 'true', 'gateway': '172.16.0.1', 'defaultRoute': True}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:36,634::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:36,634::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:07:50,160::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:50,160::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:07:50,164::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:50,164::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:07:50,169::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:50,169::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-14::DEBUG::2014-11-28 15:07:50,674::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call ping with () {} flowID [3d522862]
Thread-14::DEBUG::2014-11-28 15:07:50,674::BindingXMLRPC::1074::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:07:50,991::BindingXMLRPC::1074::vds::(wrapper) return setupNetworks with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:07:51,181::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [3d522862]
Thread-13::DEBUG::2014-11-28 15:07:51,182::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:51,187::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:07:51,207::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1404144562.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0047-4710-8038-B1C04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:25', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '8f04642b-5fd9-4fdc-9884-9b4187d3e718', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:13:25', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:26', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:13:26', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199', 'netConfigDirty': 'True', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '6e:ac:d1:90:02:62', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.898', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:07:51,311::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setSafeNetworkConfig with () {} flowID [7f852268]
Thread-13::DEBUG::2014-11-28 15:07:51,319::BindingXMLRPC::1074::vds::(wrapper) return setSafeNetworkConfig with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:07:52,271::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [37475ba1]
Thread-13::DEBUG::2014-11-28 15:07:52,272::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:52,278::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:07:52,298::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1404144562.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0047-4710-8038-B1C04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:25', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '8f04642b-5fd9-4fdc-9884-9b4187d3e718', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:13:25', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:26', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:13:26', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '6e:ac:d1:90:02:62', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.898', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:07:52,318::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [37475ba1]
Thread-13::DEBUG::2014-11-28 15:07:52,357::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0047-4710-8038-B1C04F325031', 'systemSerialNumber': '1GG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 15:07:52,401::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {} flowID [714f50b]
Thread-13::DEBUG::2014-11-28 15:07:52,402::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:52,407::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:07:52,426::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1404144562.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0047-4710-8038-B1C04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:25', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '8f04642b-5fd9-4fdc-9884-9b4187d3e718', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:13:25', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:26', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:13:26', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '6e:ac:d1:90:02:62', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.898', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 15:07:52,447::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {} flowID [714f50b]
Thread-13::DEBUG::2014-11-28 15:07:52,447::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0047-4710-8038-B1C04F325031', 'systemSerialNumber': '1GG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 15:07:52,633::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [6669b79a]
Thread-13::DEBUG::2014-11-28 15:07:52,634::task::595::TaskManager.Task::(_updateState) Task=`c6c1f2a7-4026-4ec8-af88-ded412838e46`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:07:52,634::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:07:52,640::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::ERROR::2014-11-28 15:07:52,735::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::ERROR::2014-11-28 15:07:52,736::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::DEBUG::2014-11-28 15:07:52,738::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {}
Thread-13::DEBUG::2014-11-28 15:07:52,743::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.2:/data/bs02store /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:52,936::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:07:52,938::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:07:52,938::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:07:52,944::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.99:/mnt/storage /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 15:07:52,974::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-13::DEBUG::2014-11-28 15:07:52,976::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 15:07:52,976::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:07:52,982::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-13::ERROR::2014-11-28 15:10:04,094::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 15:10:04,095::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 15:10:04,095::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 15:10:04,102::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::ERROR::2014-11-28 15:10:04,120::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::ERROR::2014-11-28 15:10:04,121::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::DEBUG::2014-11-28 15:10:04,121::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:10:04,121::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:10:04,121::task::1185::TaskManager.Task::(prepare) Task=`c6c1f2a7-4026-4ec8-af88-ded412838e46`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:10:04,121::task::595::TaskManager.Task::(_updateState) Task=`c6c1f2a7-4026-4ec8-af88-ded412838e46`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:10:04,122::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:10:04,122::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:10:04,122::task::990::TaskManager.Task::(_decref) Task=`c6c1f2a7-4026-4ec8-af88-ded412838e46`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:10:20,737::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 15:10:20,738::task::595::TaskManager.Task::(_updateState) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:10:20,738::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24235, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 15:10:20,739::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`827aa596-9e42-42bf-b622-5ace620e24a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:10:20,739::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:10:20,739::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:10:20,739::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`827aa596-9e42-42bf-b622-5ace620e24a4`::Granted request
Thread-13::DEBUG::2014-11-28 15:10:20,740::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`b0c06c20-a807-4ca7-9177-c70eb6d8ba36`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:10:20,740::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:10:20,740::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:10:20,741::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`b0c06c20-a807-4ca7-9177-c70eb6d8ba36`::Granted request
Thread-13::INFO::2014-11-28 15:10:20,741::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c0c3b90
Thread-13::INFO::2014-11-28 15:10:20,741::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24235)
Thread-13::DEBUG::2014-11-28 15:10:20,741::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,742::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,742::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,742::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,742::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,743::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,743::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 15:10:20,743::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:10:20,743::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 15:10:20,743::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:10:20,744::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 15:10:20,744::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 15:10:20,757::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:10:20,758::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 15:10:20,819::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:10:20,819::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,820::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,820::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,820::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,820::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,821::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:10:20,821::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:10:20,980::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-13::DEBUG::2014-11-28 15:10:21,128::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 15:10:21,131::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-13::DEBUG::2014-11-28 15:11:21,132::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 15:11:21,132::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 15:11:21,133::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:11:21,133::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 15:11:21,133::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-11-28 15:11:21,133::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 15:11:21,133::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:11:21,134::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::ERROR::2014-11-28 15:11:21,134::task::866::TaskManager.Task::(_setError) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:11:21,137::task::885::TaskManager.Task::(_run) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::Task._run: 276d7d4f-ccc4-41cb-aa4d-458e15417b28 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24235, None) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:11:21,137::task::1211::TaskManager.Task::(stop) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:11:21,137::task::990::TaskManager.Task::(_decref) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::ref 1 aborting True
Thread-13::DEBUG::2014-11-28 15:11:21,137::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-13::INFO::2014-11-28 15:11:21,138::task::1168::TaskManager.Task::(prepare) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::aborting: Task is aborted: u'' - code 100
Thread-13::DEBUG::2014-11-28 15:11:21,138::task::1173::TaskManager.Task::(prepare) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::Prepare: aborted: 
Thread-13::DEBUG::2014-11-28 15:11:21,138::task::990::TaskManager.Task::(_decref) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:11:21,138::task::925::TaskManager.Task::(_doAbort) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:11:21,138::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:11:21,139::task::595::TaskManager.Task::(_updateState) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:11:21,139::task::550::TaskManager.Task::(__state_aborting) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:11:21,139::task::595::TaskManager.Task::(_updateState) Task=`276d7d4f-ccc4-41cb-aa4d-458e15417b28`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:11:21,139::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:11:21,139::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:11:21,140::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-13::DEBUG::2014-11-28 15:11:21,150::task::595::TaskManager.Task::(_updateState) Task=`0f27c67d-4024-4933-a84c-b0b28a5f5508`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:11:21,151::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:11:21,151::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:11:21,151::task::1185::TaskManager.Task::(prepare) Task=`0f27c67d-4024-4933-a84c-b0b28a5f5508`::finished: {}
Thread-13::DEBUG::2014-11-28 15:11:21,151::task::595::TaskManager.Task::(_updateState) Task=`0f27c67d-4024-4933-a84c-b0b28a5f5508`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:11:21,151::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:11:21,152::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:11:21,152::task::990::TaskManager.Task::(_decref) Task=`0f27c67d-4024-4933-a84c-b0b28a5f5508`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:11:21,171::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': False},) {} flowID [6669b79a]
Thread-13::DEBUG::2014-11-28 15:11:21,241::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 15:11:30,323::task::595::TaskManager.Task::(_updateState) Task=`f0c620fb-5db0-4dd4-9207-df1f512c77da`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:11:30,323::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:11:30,324::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:11:30,324::task::1185::TaskManager.Task::(prepare) Task=`f0c620fb-5db0-4dd4-9207-df1f512c77da`::finished: {}
Thread-13::DEBUG::2014-11-28 15:11:30,324::task::595::TaskManager.Task::(_updateState) Task=`f0c620fb-5db0-4dd4-9207-df1f512c77da`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:11:30,324::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:11:30,324::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:11:30,325::task::990::TaskManager.Task::(_decref) Task=`f0c620fb-5db0-4dd4-9207-df1f512c77da`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:11:45,427::task::595::TaskManager.Task::(_updateState) Task=`5aab3eb1-89bb-4e16-a28a-668c5720f110`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:11:45,427::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:11:45,427::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:11:45,428::task::1185::TaskManager.Task::(prepare) Task=`5aab3eb1-89bb-4e16-a28a-668c5720f110`::finished: {}
Thread-13::DEBUG::2014-11-28 15:11:45,428::task::595::TaskManager.Task::(_updateState) Task=`5aab3eb1-89bb-4e16-a28a-668c5720f110`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:11:45,428::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:11:45,428::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:11:45,428::task::990::TaskManager.Task::(_decref) Task=`5aab3eb1-89bb-4e16-a28a-668c5720f110`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:00,525::task::595::TaskManager.Task::(_updateState) Task=`db04f2a3-7c1d-4a8a-bfc4-a7de75e188b3`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:00,525::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:12:00,525::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:12:00,525::task::1185::TaskManager.Task::(prepare) Task=`db04f2a3-7c1d-4a8a-bfc4-a7de75e188b3`::finished: {}
Thread-13::DEBUG::2014-11-28 15:12:00,526::task::595::TaskManager.Task::(_updateState) Task=`db04f2a3-7c1d-4a8a-bfc4-a7de75e188b3`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:00,526::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:00,526::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:00,526::task::990::TaskManager.Task::(_decref) Task=`db04f2a3-7c1d-4a8a-bfc4-a7de75e188b3`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:15,630::task::595::TaskManager.Task::(_updateState) Task=`c2865919-5db5-41ce-a7ef-c0b64888dfca`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:15,630::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:12:15,631::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:12:15,631::task::1185::TaskManager.Task::(prepare) Task=`c2865919-5db5-41ce-a7ef-c0b64888dfca`::finished: {}
Thread-13::DEBUG::2014-11-28 15:12:15,631::task::595::TaskManager.Task::(_updateState) Task=`c2865919-5db5-41ce-a7ef-c0b64888dfca`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:15,631::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:15,631::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:15,631::task::990::TaskManager.Task::(_decref) Task=`c2865919-5db5-41ce-a7ef-c0b64888dfca`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:30,738::task::595::TaskManager.Task::(_updateState) Task=`c09746d7-2907-4690-bd07-08deea07540e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:30,738::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:12:30,738::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:12:30,739::task::1185::TaskManager.Task::(prepare) Task=`c09746d7-2907-4690-bd07-08deea07540e`::finished: {}
Thread-13::DEBUG::2014-11-28 15:12:30,739::task::595::TaskManager.Task::(_updateState) Task=`c09746d7-2907-4690-bd07-08deea07540e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:30,739::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:30,739::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:30,739::task::990::TaskManager.Task::(_decref) Task=`c09746d7-2907-4690-bd07-08deea07540e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:31,962::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [36988381]
Thread-13::DEBUG::2014-11-28 15:12:31,962::task::595::TaskManager.Task::(_updateState) Task=`baaddf0b-94f2-4c0f-ab21-5fd83bf6c44d`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:31,963::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:12:31,969::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::ERROR::2014-11-28 15:12:31,987::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::ERROR::2014-11-28 15:12:31,988::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::DEBUG::2014-11-28 15:12:31,988::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:12:31,988::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:12:31,988::task::1185::TaskManager.Task::(prepare) Task=`baaddf0b-94f2-4c0f-ab21-5fd83bf6c44d`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:12:31,989::task::595::TaskManager.Task::(_updateState) Task=`baaddf0b-94f2-4c0f-ab21-5fd83bf6c44d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:31,989::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:31,989::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:31,989::task::990::TaskManager.Task::(_decref) Task=`baaddf0b-94f2-4c0f-ab21-5fd83bf6c44d`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:32,065::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3ec885ee]
Thread-13::DEBUG::2014-11-28 15:12:32,065::task::595::TaskManager.Task::(_updateState) Task=`82100e1c-cc8d-4e34-974d-f4d29d5a1e3a`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:32,066::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:12:32,071::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:12:32,073::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:12:32,073::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:12:32,073::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:12:32,073::task::1185::TaskManager.Task::(prepare) Task=`82100e1c-cc8d-4e34-974d-f4d29d5a1e3a`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:12:32,074::task::595::TaskManager.Task::(_updateState) Task=`82100e1c-cc8d-4e34-974d-f4d29d5a1e3a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:32,074::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:32,074::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:32,074::task::990::TaskManager.Task::(_decref) Task=`82100e1c-cc8d-4e34-974d-f4d29d5a1e3a`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:12:45,861::task::595::TaskManager.Task::(_updateState) Task=`c1bce499-f426-4a6f-8ea2-063a6653d85d`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:12:45,861::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:12:45,861::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:12:45,862::task::1185::TaskManager.Task::(prepare) Task=`c1bce499-f426-4a6f-8ea2-063a6653d85d`::finished: {}
Thread-13::DEBUG::2014-11-28 15:12:45,862::task::595::TaskManager.Task::(_updateState) Task=`c1bce499-f426-4a6f-8ea2-063a6653d85d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:12:45,862::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:12:45,862::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:12:45,862::task::990::TaskManager.Task::(_decref) Task=`c1bce499-f426-4a6f-8ea2-063a6653d85d`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:13:32,128::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1eb0234a]
Thread-13::DEBUG::2014-11-28 15:13:32,128::task::595::TaskManager.Task::(_updateState) Task=`4fd03d51-ec9a-4af2-a369-3ad30ae1d273`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:13:32,129::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:13:32,135::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-17::DEBUG::2014-11-28 15:13:40,426::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-17::DEBUG::2014-11-28 15:13:40,426::task::595::TaskManager.Task::(_updateState) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:13:40,426::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-17::ERROR::2014-11-28 15:13:40,427::task::866::TaskManager.Task::(_setError) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-17::DEBUG::2014-11-28 15:13:40,427::task::885::TaskManager.Task::(_run) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::Task._run: a46ea572-63ca-4405-9168-663b75b27d94 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-17::DEBUG::2014-11-28 15:13:40,427::task::1211::TaskManager.Task::(stop) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::stopping in state preparing (force False)
Thread-17::DEBUG::2014-11-28 15:13:40,427::task::990::TaskManager.Task::(_decref) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::ref 1 aborting True
Thread-17::INFO::2014-11-28 15:13:40,428::task::1168::TaskManager.Task::(prepare) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-17::DEBUG::2014-11-28 15:13:40,428::task::1173::TaskManager.Task::(prepare) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::Prepare: aborted: Unknown pool id, pool not connected
Thread-17::DEBUG::2014-11-28 15:13:40,428::task::990::TaskManager.Task::(_decref) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::ref 0 aborting True
Thread-17::DEBUG::2014-11-28 15:13:40,428::task::925::TaskManager.Task::(_doAbort) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::Task._doAbort: force False
Thread-17::DEBUG::2014-11-28 15:13:40,428::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:13:40,428::task::595::TaskManager.Task::(_updateState) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::moving from state preparing -> state aborting
Thread-17::DEBUG::2014-11-28 15:13:40,429::task::550::TaskManager.Task::(__state_aborting) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::_aborting: recover policy none
Thread-17::DEBUG::2014-11-28 15:13:40,429::task::595::TaskManager.Task::(_updateState) Task=`a46ea572-63ca-4405-9168-663b75b27d94`::moving from state aborting -> state failed
Thread-17::DEBUG::2014-11-28 15:13:40,429::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:13:40,429::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::ERROR::2014-11-28 15:13:40,429::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-17::DEBUG::2014-11-28 15:13:40,451::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-17::DEBUG::2014-11-28 15:13:40,452::task::595::TaskManager.Task::(_updateState) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:13:40,452::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24235, domainsMap=None, options=None)
Thread-17::DEBUG::2014-11-28 15:13:40,452::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`9bce2feb-dd5c-408b-86fa-0e9f1ce40645`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-17::DEBUG::2014-11-28 15:13:40,453::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-17::DEBUG::2014-11-28 15:13:40,453::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-17::DEBUG::2014-11-28 15:13:40,453::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`9bce2feb-dd5c-408b-86fa-0e9f1ce40645`::Granted request
Thread-17::DEBUG::2014-11-28 15:13:40,454::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`9f7f35b9-9aaf-405d-aa62-a851a30f8c3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-17::DEBUG::2014-11-28 15:13:40,454::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-17::DEBUG::2014-11-28 15:13:40,454::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-17::DEBUG::2014-11-28 15:13:40,454::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`9f7f35b9-9aaf-405d-aa62-a851a30f8c3a`::Granted request
Thread-17::INFO::2014-11-28 15:13:40,455::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c041dd0
Thread-17::INFO::2014-11-28 15:13:40,455::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24235)
Thread-17::DEBUG::2014-11-28 15:13:40,455::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,455::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,456::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,456::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,456::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,456::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,457::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-17::DEBUG::2014-11-28 15:13:40,457::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-17::DEBUG::2014-11-28 15:13:40,457::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-17::DEBUG::2014-11-28 15:13:40,457::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-17::DEBUG::2014-11-28 15:13:40,457::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-17::DEBUG::2014-11-28 15:13:40,458::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-17::DEBUG::2014-11-28 15:13:40,469::misc::748::SamplingMethod::(__call__) Returning last result
Thread-17::DEBUG::2014-11-28 15:13:40,472::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-17::DEBUG::2014-11-28 15:13:40,515::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-17::DEBUG::2014-11-28 15:13:40,515::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,516::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,516::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,516::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,516::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,517::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-17::DEBUG::2014-11-28 15:13:40,517::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:14:32,135::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:14:32,136::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:14:32,136::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:14:32,136::task::1185::TaskManager.Task::(prepare) Task=`4fd03d51-ec9a-4af2-a369-3ad30ae1d273`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:14:32,136::task::595::TaskManager.Task::(_updateState) Task=`4fd03d51-ec9a-4af2-a369-3ad30ae1d273`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:14:32,137::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:14:32,137::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:14:32,137::task::990::TaskManager.Task::(_decref) Task=`4fd03d51-ec9a-4af2-a369-3ad30ae1d273`::ref 0 aborting False
Thread-19::WARNING::2014-11-28 15:14:40,519::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::ERROR::2014-11-28 15:14:40,520::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-17::DEBUG::2014-11-28 15:14:40,520::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-17::DEBUG::2014-11-28 15:14:40,520::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-17::DEBUG::2014-11-28 15:14:40,521::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-17::ERROR::2014-11-28 15:14:40,522::task::866::TaskManager.Task::(_setError) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-17::DEBUG::2014-11-28 15:14:40,522::task::885::TaskManager.Task::(_run) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::Task._run: 702c2bee-ac9f-466b-bd0b-2a539b598673 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24235, None) {} failed - stopping task
Thread-17::DEBUG::2014-11-28 15:14:40,522::task::1211::TaskManager.Task::(stop) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::stopping in state preparing (force False)
Thread-17::DEBUG::2014-11-28 15:14:40,522::task::990::TaskManager.Task::(_decref) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::ref 1 aborting True
Thread-17::INFO::2014-11-28 15:14:40,523::task::1168::TaskManager.Task::(prepare) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-17::DEBUG::2014-11-28 15:14:40,523::task::1173::TaskManager.Task::(prepare) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::Prepare: aborted: Cannot find master domain
Thread-17::DEBUG::2014-11-28 15:14:40,523::task::990::TaskManager.Task::(_decref) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::ref 0 aborting True
Thread-17::DEBUG::2014-11-28 15:14:40,523::task::925::TaskManager.Task::(_doAbort) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::Task._doAbort: force False
Thread-17::DEBUG::2014-11-28 15:14:40,523::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:14:40,523::task::595::TaskManager.Task::(_updateState) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::moving from state preparing -> state aborting
Thread-17::DEBUG::2014-11-28 15:14:40,524::task::550::TaskManager.Task::(__state_aborting) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::_aborting: recover policy none
Thread-17::DEBUG::2014-11-28 15:14:40,524::task::595::TaskManager.Task::(_updateState) Task=`702c2bee-ac9f-466b-bd0b-2a539b598673`::moving from state aborting -> state failed
Thread-17::DEBUG::2014-11-28 15:14:40,524::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:14:40,524::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::ERROR::2014-11-28 15:14:40,524::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-17::DEBUG::2014-11-28 15:14:40,549::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-17::DEBUG::2014-11-28 15:14:40,549::task::595::TaskManager.Task::(_updateState) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:14:40,550::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-17::ERROR::2014-11-28 15:14:40,550::task::866::TaskManager.Task::(_setError) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-17::DEBUG::2014-11-28 15:14:40,550::task::885::TaskManager.Task::(_run) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::Task._run: 53a1fac5-62a3-4fed-b3d7-c7d6669219d1 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-17::DEBUG::2014-11-28 15:14:40,550::task::1211::TaskManager.Task::(stop) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::stopping in state preparing (force False)
Thread-17::DEBUG::2014-11-28 15:14:40,550::task::990::TaskManager.Task::(_decref) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::ref 1 aborting True
Thread-17::INFO::2014-11-28 15:14:40,551::task::1168::TaskManager.Task::(prepare) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-17::DEBUG::2014-11-28 15:14:40,551::task::1173::TaskManager.Task::(prepare) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::Prepare: aborted: Unknown pool id, pool not connected
Thread-17::DEBUG::2014-11-28 15:14:40,551::task::990::TaskManager.Task::(_decref) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::ref 0 aborting True
Thread-17::DEBUG::2014-11-28 15:14:40,551::task::925::TaskManager.Task::(_doAbort) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::Task._doAbort: force False
Thread-17::DEBUG::2014-11-28 15:14:40,551::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:14:40,551::task::595::TaskManager.Task::(_updateState) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::moving from state preparing -> state aborting
Thread-17::DEBUG::2014-11-28 15:14:40,552::task::550::TaskManager.Task::(__state_aborting) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::_aborting: recover policy none
Thread-17::DEBUG::2014-11-28 15:14:40,552::task::595::TaskManager.Task::(_updateState) Task=`53a1fac5-62a3-4fed-b3d7-c7d6669219d1`::moving from state aborting -> state failed
Thread-17::DEBUG::2014-11-28 15:14:40,552::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:14:40,552::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::ERROR::2014-11-28 15:14:40,552::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-17::DEBUG::2014-11-28 15:14:44,196::task::595::TaskManager.Task::(_updateState) Task=`22ce519c-c359-4b1b-8148-deae44cf801a`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:14:44,197::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-17::INFO::2014-11-28 15:14:44,197::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-17::DEBUG::2014-11-28 15:14:44,197::task::1185::TaskManager.Task::(prepare) Task=`22ce519c-c359-4b1b-8148-deae44cf801a`::finished: {}
Thread-17::DEBUG::2014-11-28 15:14:44,197::task::595::TaskManager.Task::(_updateState) Task=`22ce519c-c359-4b1b-8148-deae44cf801a`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:14:44,197::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:14:44,198::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:14:44,198::task::990::TaskManager.Task::(_decref) Task=`22ce519c-c359-4b1b-8148-deae44cf801a`::ref 0 aborting False
Thread-17::DEBUG::2014-11-28 15:14:59,927::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5833472]
Thread-17::DEBUG::2014-11-28 15:14:59,928::task::595::TaskManager.Task::(_updateState) Task=`916d2a03-0bee-4c4e-b747-f2ce3a2391a4`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:14:59,928::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-17::DEBUG::2014-11-28 15:14:59,935::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-17::ERROR::2014-11-28 15:14:59,954::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-17::ERROR::2014-11-28 15:14:59,954::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-17::DEBUG::2014-11-28 15:14:59,954::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-17::INFO::2014-11-28 15:14:59,955::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-17::DEBUG::2014-11-28 15:14:59,955::task::1185::TaskManager.Task::(prepare) Task=`916d2a03-0bee-4c4e-b747-f2ce3a2391a4`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-17::DEBUG::2014-11-28 15:14:59,955::task::595::TaskManager.Task::(_updateState) Task=`916d2a03-0bee-4c4e-b747-f2ce3a2391a4`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:14:59,955::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:14:59,955::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:14:59,956::task::990::TaskManager.Task::(_decref) Task=`916d2a03-0bee-4c4e-b747-f2ce3a2391a4`::ref 0 aborting False
Thread-17::DEBUG::2014-11-28 15:15:00,079::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [326dc7b3]
Thread-17::DEBUG::2014-11-28 15:15:00,079::task::595::TaskManager.Task::(_updateState) Task=`602da6e9-cb61-4b63-9700-e8ac34c68e9e`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:15:00,079::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-17::DEBUG::2014-11-28 15:15:00,090::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-13::DEBUG::2014-11-28 15:15:05,792::task::595::TaskManager.Task::(_updateState) Task=`5194754d-0911-42e6-9628-8ecb31308eec`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:15:05,792::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:15:05,793::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:15:05,793::task::1185::TaskManager.Task::(prepare) Task=`5194754d-0911-42e6-9628-8ecb31308eec`::finished: {}
Thread-13::DEBUG::2014-11-28 15:15:05,793::task::595::TaskManager.Task::(_updateState) Task=`5194754d-0911-42e6-9628-8ecb31308eec`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:15:05,793::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:15:05,793::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:15:05,794::task::990::TaskManager.Task::(_decref) Task=`5194754d-0911-42e6-9628-8ecb31308eec`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:15:20,972::task::595::TaskManager.Task::(_updateState) Task=`c57f70d7-4ebd-4932-8d1d-8e6c0034c68b`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:15:20,972::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:15:20,972::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:15:20,972::task::1185::TaskManager.Task::(prepare) Task=`c57f70d7-4ebd-4932-8d1d-8e6c0034c68b`::finished: {}
Thread-13::DEBUG::2014-11-28 15:15:20,972::task::595::TaskManager.Task::(_updateState) Task=`c57f70d7-4ebd-4932-8d1d-8e6c0034c68b`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:15:20,973::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:15:20,973::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:15:20,973::task::990::TaskManager.Task::(_decref) Task=`c57f70d7-4ebd-4932-8d1d-8e6c0034c68b`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:15:36,075::task::595::TaskManager.Task::(_updateState) Task=`7b074aa7-a5df-4286-9069-e86b19d64191`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:15:36,075::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:15:36,076::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:15:36,076::task::1185::TaskManager.Task::(prepare) Task=`7b074aa7-a5df-4286-9069-e86b19d64191`::finished: {}
Thread-13::DEBUG::2014-11-28 15:15:36,076::task::595::TaskManager.Task::(_updateState) Task=`7b074aa7-a5df-4286-9069-e86b19d64191`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:15:36,076::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:15:36,076::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:15:36,076::task::990::TaskManager.Task::(_decref) Task=`7b074aa7-a5df-4286-9069-e86b19d64191`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:15:51,184::task::595::TaskManager.Task::(_updateState) Task=`1368a232-67fd-4ef5-a87c-56a43c64a183`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:15:51,184::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:15:51,185::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:15:51,185::task::1185::TaskManager.Task::(prepare) Task=`1368a232-67fd-4ef5-a87c-56a43c64a183`::finished: {}
Thread-13::DEBUG::2014-11-28 15:15:51,185::task::595::TaskManager.Task::(_updateState) Task=`1368a232-67fd-4ef5-a87c-56a43c64a183`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:15:51,185::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:15:51,185::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:15:51,186::task::990::TaskManager.Task::(_decref) Task=`1368a232-67fd-4ef5-a87c-56a43c64a183`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:16:06,330::task::595::TaskManager.Task::(_updateState) Task=`02bcfcd0-abb9-46c6-a3d3-bc786e30e886`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:16:06,330::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:16:06,330::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:16:06,330::task::1185::TaskManager.Task::(prepare) Task=`02bcfcd0-abb9-46c6-a3d3-bc786e30e886`::finished: {}
Thread-13::DEBUG::2014-11-28 15:16:06,330::task::595::TaskManager.Task::(_updateState) Task=`02bcfcd0-abb9-46c6-a3d3-bc786e30e886`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:16:06,331::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:16:06,331::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:16:06,331::task::990::TaskManager.Task::(_decref) Task=`02bcfcd0-abb9-46c6-a3d3-bc786e30e886`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:16:21,423::task::595::TaskManager.Task::(_updateState) Task=`1c740d90-ea68-4ac5-a4e1-6c586c93a4ff`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:16:21,423::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:16:21,424::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:16:21,424::task::1185::TaskManager.Task::(prepare) Task=`1c740d90-ea68-4ac5-a4e1-6c586c93a4ff`::finished: {}
Thread-13::DEBUG::2014-11-28 15:16:21,424::task::595::TaskManager.Task::(_updateState) Task=`1c740d90-ea68-4ac5-a4e1-6c586c93a4ff`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:16:21,424::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:16:21,424::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:16:21,424::task::990::TaskManager.Task::(_decref) Task=`1c740d90-ea68-4ac5-a4e1-6c586c93a4ff`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:16:36,539::task::595::TaskManager.Task::(_updateState) Task=`56a73601-4d18-49d1-a04a-3aa8238eb186`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:16:36,539::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:16:36,539::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:16:36,540::task::1185::TaskManager.Task::(prepare) Task=`56a73601-4d18-49d1-a04a-3aa8238eb186`::finished: {}
Thread-13::DEBUG::2014-11-28 15:16:36,540::task::595::TaskManager.Task::(_updateState) Task=`56a73601-4d18-49d1-a04a-3aa8238eb186`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:16:36,540::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:16:36,540::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:16:36,540::task::990::TaskManager.Task::(_decref) Task=`56a73601-4d18-49d1-a04a-3aa8238eb186`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:16:41,217::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7f5d50d7]
Thread-13::DEBUG::2014-11-28 15:16:41,217::task::595::TaskManager.Task::(_updateState) Task=`b8f3711c-b62c-4865-8361-99bf517a9fb7`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:16:41,218::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, remove=False, options=None)
Thread-13::WARNING::2014-11-28 15:16:41,218::hsm::1126::Storage.HSM::(disconnectStoragePool) disconnect sp: 45995288-a7d6-4cbf-8b2f-63fa35642bad failed. Known pools {}
Thread-13::INFO::2014-11-28 15:16:41,218::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
Thread-13::DEBUG::2014-11-28 15:16:41,218::task::1185::TaskManager.Task::(prepare) Task=`b8f3711c-b62c-4865-8361-99bf517a9fb7`::finished: None
Thread-13::DEBUG::2014-11-28 15:16:41,218::task::595::TaskManager.Task::(_updateState) Task=`b8f3711c-b62c-4865-8361-99bf517a9fb7`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:16:41,218::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:16:41,219::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:16:41,219::task::990::TaskManager.Task::(_decref) Task=`b8f3711c-b62c-4865-8361-99bf517a9fb7`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:16:41,233::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7f5d50d7]
Thread-13::DEBUG::2014-11-28 15:16:41,234::task::595::TaskManager.Task::(_updateState) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:16:41,234::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', poolName='CTI', masterDom='54b2d645-fc18-4784-99eb-0e17e296eee7', domDict={'1c4abda0-906b-4faf-a2e3-b6683729cc2f': 'active', '474f0809-26c6-4fca-ab8c-72a2e6dcd912': 'active', '54b2d645-fc18-4784-99eb-0e17e296eee7': 'active', '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': 'active', 'e7eaae86-11ea-4c08-8c11-8b8c54373389': 'active'}, masterVersion=24236, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None)
Thread-13::INFO::2014-11-28 15:16:41,234::hsm::1862::Storage.HSM::(reconstructMaster) spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad master=54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::INFO::2014-11-28 15:16:41,234::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c0b0368
Thread-13::DEBUG::2014-11-28 15:16:41,235::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 15:16:41,235::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:16:41,235::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 15:16:41,235::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:16:41,235::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 15:16:41,236::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 15:16:41,247::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:16:41,248::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 15:16:41,294::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:16:41,294::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,295::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,295::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,295::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,296::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,296::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,296::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::ERROR::2014-11-28 15:16:41,296::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::ERROR::2014-11-28 15:16:41,296::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::DEBUG::2014-11-28 15:16:41,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:16:41,298::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-13::DEBUG::2014-11-28 15:16:41,328::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-13::WARNING::2014-11-28 15:16:41,330::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-13::DEBUG::2014-11-28 15:16:41,330::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-17::ERROR::2014-11-28 15:17:11,203::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-17::ERROR::2014-11-28 15:17:11,204::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-17::DEBUG::2014-11-28 15:17:11,204::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-17::INFO::2014-11-28 15:17:11,204::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-17::DEBUG::2014-11-28 15:17:11,204::task::1185::TaskManager.Task::(prepare) Task=`602da6e9-cb61-4b63-9700-e8ac34c68e9e`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-17::DEBUG::2014-11-28 15:17:11,205::task::595::TaskManager.Task::(_updateState) Task=`602da6e9-cb61-4b63-9700-e8ac34c68e9e`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:17:11,205::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:17:11,205::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:17:11,205::task::990::TaskManager.Task::(_decref) Task=`602da6e9-cb61-4b63-9700-e8ac34c68e9e`::ref 0 aborting False
Thread-17::DEBUG::2014-11-28 15:17:11,337::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5b3d3db1]
Thread-17::DEBUG::2014-11-28 15:17:11,337::task::595::TaskManager.Task::(_updateState) Task=`c7773038-ab43-434a-8ac4-1d21246e9ec1`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:17:11,338::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-17::DEBUG::2014-11-28 15:17:11,344::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-17::ERROR::2014-11-28 15:17:11,361::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-17::ERROR::2014-11-28 15:17:11,361::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-17::DEBUG::2014-11-28 15:17:11,362::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-17::INFO::2014-11-28 15:17:11,362::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-17::DEBUG::2014-11-28 15:17:11,362::task::1185::TaskManager.Task::(prepare) Task=`c7773038-ab43-434a-8ac4-1d21246e9ec1`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-17::DEBUG::2014-11-28 15:17:11,362::task::595::TaskManager.Task::(_updateState) Task=`c7773038-ab43-434a-8ac4-1d21246e9ec1`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:17:11,363::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:17:11,363::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:17:11,363::task::990::TaskManager.Task::(_decref) Task=`c7773038-ab43-434a-8ac4-1d21246e9ec1`::ref 0 aborting False
Thread-17::DEBUG::2014-11-28 15:17:11,459::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [40f8b2b2]
Thread-17::DEBUG::2014-11-28 15:17:11,459::task::595::TaskManager.Task::(_updateState) Task=`54d6d8a4-071e-4d91-be05-0c8c6e2c13eb`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:17:11,460::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-17::DEBUG::2014-11-28 15:17:11,466::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-21::WARNING::2014-11-28 15:17:41,343::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::ERROR::2014-11-28 15:17:41,344::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-13::ERROR::2014-11-28 15:17:41,345::task::866::TaskManager.Task::(_setError) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1872, in reconstructMaster
    self.validateSdUUID(masterDom)
  File "/usr/share/vdsm/storage/hsm.py", line 271, in validateSdUUID
    sdDom = sdCache.produce(sdUUID=sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-13::DEBUG::2014-11-28 15:17:41,345::task::885::TaskManager.Task::(_run) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::Task._run: b4b2419b-0407-4f77-8ff1-026c50a02ee8 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 'CTI', '54b2d645-fc18-4784-99eb-0e17e296eee7', {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': 'active', '474f0809-26c6-4fca-ab8c-72a2e6dcd912': 'active', '54b2d645-fc18-4784-99eb-0e17e296eee7': 'active', '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': 'active', 'e7eaae86-11ea-4c08-8c11-8b8c54373389': 'active'}, 24236, None, 5, 60, 10, 3, 1) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:17:41,345::task::1211::TaskManager.Task::(stop) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:17:41,346::task::990::TaskManager.Task::(_decref) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::ref 1 aborting True
Thread-13::INFO::2014-11-28 15:17:41,346::task::1168::TaskManager.Task::(prepare) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
Thread-13::DEBUG::2014-11-28 15:17:41,346::task::1173::TaskManager.Task::(prepare) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::Prepare: aborted: Storage domain does not exist
Thread-13::DEBUG::2014-11-28 15:17:41,346::task::990::TaskManager.Task::(_decref) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:17:41,346::task::925::TaskManager.Task::(_doAbort) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:17:41,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:17:41,347::task::595::TaskManager.Task::(_updateState) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:17:41,347::task::550::TaskManager.Task::(__state_aborting) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:17:41,347::task::595::TaskManager.Task::(_updateState) Task=`b4b2419b-0407-4f77-8ff1-026c50a02ee8`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:17:41,347::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:17:41,347::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:17:41,347::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)", 'code': 358}}
Thread-13::DEBUG::2014-11-28 15:17:47,381::task::595::TaskManager.Task::(_updateState) Task=`b7a020aa-b435-48d8-b052-22db46d6c559`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:17:47,381::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:17:47,381::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:17:47,382::task::1185::TaskManager.Task::(prepare) Task=`b7a020aa-b435-48d8-b052-22db46d6c559`::finished: {}
Thread-13::DEBUG::2014-11-28 15:17:47,382::task::595::TaskManager.Task::(_updateState) Task=`b7a020aa-b435-48d8-b052-22db46d6c559`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:17:47,382::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:17:47,382::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:17:47,382::task::990::TaskManager.Task::(_decref) Task=`b7a020aa-b435-48d8-b052-22db46d6c559`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:17:51,185::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 15:17:51,186::task::595::TaskManager.Task::(_updateState) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:17:51,186::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-13::ERROR::2014-11-28 15:17:51,186::task::866::TaskManager.Task::(_setError) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-13::DEBUG::2014-11-28 15:17:51,187::task::885::TaskManager.Task::(_run) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::Task._run: 8816aef4-78ed-4ae7-9d3e-8ab85d772e93 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:17:51,187::task::1211::TaskManager.Task::(stop) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:17:51,187::task::990::TaskManager.Task::(_decref) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::ref 1 aborting True
Thread-13::INFO::2014-11-28 15:17:51,187::task::1168::TaskManager.Task::(prepare) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-13::DEBUG::2014-11-28 15:17:51,187::task::1173::TaskManager.Task::(prepare) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::Prepare: aborted: Unknown pool id, pool not connected
Thread-13::DEBUG::2014-11-28 15:17:51,188::task::990::TaskManager.Task::(_decref) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:17:51,188::task::925::TaskManager.Task::(_doAbort) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:17:51,188::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:17:51,188::task::595::TaskManager.Task::(_updateState) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:17:51,188::task::550::TaskManager.Task::(__state_aborting) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:17:51,188::task::595::TaskManager.Task::(_updateState) Task=`8816aef4-78ed-4ae7-9d3e-8ab85d772e93`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:17:51,189::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:17:51,189::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:17:51,189::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-13::DEBUG::2014-11-28 15:17:51,202::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 15:17:51,203::task::595::TaskManager.Task::(_updateState) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:17:51,203::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24236, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 15:17:51,203::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a2f7f589-b265-49a7-9c17-ac6cce0d8a39`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:17:51,204::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:17:51,204::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:17:51,204::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a2f7f589-b265-49a7-9c17-ac6cce0d8a39`::Granted request
Thread-13::DEBUG::2014-11-28 15:17:51,205::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`3cc4d818-e704-475e-a670-d73ed7e31f7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 15:17:51,205::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:17:51,205::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:17:51,205::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`3cc4d818-e704-475e-a670-d73ed7e31f7f`::Granted request
Thread-13::INFO::2014-11-28 15:17:51,206::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c198440
Thread-13::INFO::2014-11-28 15:17:51,206::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24236)
Thread-13::DEBUG::2014-11-28 15:17:51,206::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,206::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,207::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,207::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,207::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,207::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,208::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 15:17:51,208::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:17:51,208::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 15:17:51,208::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:17:51,208::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 15:17:51,209::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 15:17:51,220::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:17:51,221::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 15:17:51,259::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:17:51,260::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,260::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,260::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,260::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,261::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,261::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:17:51,261::misc::748::SamplingMethod::(__call__) Returning last result
Thread-17::DEBUG::2014-11-28 15:18:11,471::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-17::DEBUG::2014-11-28 15:18:11,471::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-17::INFO::2014-11-28 15:18:11,471::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-17::DEBUG::2014-11-28 15:18:11,472::task::1185::TaskManager.Task::(prepare) Task=`54d6d8a4-071e-4d91-be05-0c8c6e2c13eb`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-17::DEBUG::2014-11-28 15:18:11,472::task::595::TaskManager.Task::(_updateState) Task=`54d6d8a4-071e-4d91-be05-0c8c6e2c13eb`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:18:11,472::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:18:11,472::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:18:11,472::task::990::TaskManager.Task::(_decref) Task=`54d6d8a4-071e-4d91-be05-0c8c6e2c13eb`::ref 0 aborting False
Thread-17::DEBUG::2014-11-28 15:18:11,535::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3bcae3a0]
Thread-17::DEBUG::2014-11-28 15:18:11,535::task::595::TaskManager.Task::(_updateState) Task=`033b9da9-764f-428c-ac58-6354fd610dd9`::moving from state init -> state preparing
Thread-17::INFO::2014-11-28 15:18:11,535::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-17::DEBUG::2014-11-28 15:18:11,541::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-17::DEBUG::2014-11-28 15:18:11,688::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-17::DEBUG::2014-11-28 15:18:11,689::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-17::INFO::2014-11-28 15:18:11,689::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-17::DEBUG::2014-11-28 15:18:11,689::task::1185::TaskManager.Task::(prepare) Task=`033b9da9-764f-428c-ac58-6354fd610dd9`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-17::DEBUG::2014-11-28 15:18:11,689::task::595::TaskManager.Task::(_updateState) Task=`033b9da9-764f-428c-ac58-6354fd610dd9`::moving from state preparing -> state finished
Thread-17::DEBUG::2014-11-28 15:18:11,689::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-17::DEBUG::2014-11-28 15:18:11,690::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-17::DEBUG::2014-11-28 15:18:11,690::task::990::TaskManager.Task::(_decref) Task=`033b9da9-764f-428c-ac58-6354fd610dd9`::ref 0 aborting False
Thread-24::WARNING::2014-11-28 15:18:51,268::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::ERROR::2014-11-28 15:18:51,269::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 15:18:51,270::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 15:18:51,270::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 15:18:51,270::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:18:51,270::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 15:18:51,271::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-11-28 15:18:51,271::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 15:18:51,271::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:18:51,271::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::ERROR::2014-11-28 15:18:51,271::task::866::TaskManager.Task::(_setError) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-13::DEBUG::2014-11-28 15:18:51,272::task::885::TaskManager.Task::(_run) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::Task._run: 69dea61e-83bd-4be6-b367-ab72955fdc66 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24236, None) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:18:51,272::task::1211::TaskManager.Task::(stop) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:18:51,272::task::990::TaskManager.Task::(_decref) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::ref 1 aborting True
Thread-13::INFO::2014-11-28 15:18:51,272::task::1168::TaskManager.Task::(prepare) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::1173::TaskManager.Task::(prepare) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::Prepare: aborted: Cannot find master domain
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::990::TaskManager.Task::(_decref) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::925::TaskManager.Task::(_doAbort) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:18:51,273::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::595::TaskManager.Task::(_updateState) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::550::TaskManager.Task::(__state_aborting) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:18:51,273::task::595::TaskManager.Task::(_updateState) Task=`69dea61e-83bd-4be6-b367-ab72955fdc66`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:18:51,274::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:18:51,274::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:18:51,274::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-13::DEBUG::2014-11-28 15:18:51,299::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 15:18:51,300::task::595::TaskManager.Task::(_updateState) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:18:51,300::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-13::ERROR::2014-11-28 15:18:51,300::task::866::TaskManager.Task::(_setError) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-13::DEBUG::2014-11-28 15:18:51,300::task::885::TaskManager.Task::(_run) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::Task._run: b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:18:51,300::task::1211::TaskManager.Task::(stop) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:18:51,301::task::990::TaskManager.Task::(_decref) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::ref 1 aborting True
Thread-13::INFO::2014-11-28 15:18:51,301::task::1168::TaskManager.Task::(prepare) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-13::DEBUG::2014-11-28 15:18:51,301::task::1173::TaskManager.Task::(prepare) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::Prepare: aborted: Unknown pool id, pool not connected
Thread-13::DEBUG::2014-11-28 15:18:51,301::task::990::TaskManager.Task::(_decref) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:18:51,301::task::925::TaskManager.Task::(_doAbort) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:18:51,301::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:18:51,301::task::595::TaskManager.Task::(_updateState) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:18:51,302::task::550::TaskManager.Task::(__state_aborting) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:18:51,302::task::595::TaskManager.Task::(_updateState) Task=`b35cb1b3-d5ca-4e84-b9de-61d0ef9ffbf6`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:18:51,302::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:18:51,302::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 15:18:51,302::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-13::DEBUG::2014-11-28 15:18:56,941::task::595::TaskManager.Task::(_updateState) Task=`0ce119ca-bc1f-4843-99d0-f40e5160170e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:18:56,941::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:18:56,941::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:18:56,941::task::1185::TaskManager.Task::(prepare) Task=`0ce119ca-bc1f-4843-99d0-f40e5160170e`::finished: {}
Thread-13::DEBUG::2014-11-28 15:18:56,941::task::595::TaskManager.Task::(_updateState) Task=`0ce119ca-bc1f-4843-99d0-f40e5160170e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:18:56,942::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:18:56,942::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:18:56,942::task::990::TaskManager.Task::(_decref) Task=`0ce119ca-bc1f-4843-99d0-f40e5160170e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:19:12,071::task::595::TaskManager.Task::(_updateState) Task=`90951680-32f2-4eab-a39a-2a7478274a2e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:19:12,071::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:19:12,072::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:19:12,072::task::1185::TaskManager.Task::(prepare) Task=`90951680-32f2-4eab-a39a-2a7478274a2e`::finished: {}
Thread-13::DEBUG::2014-11-28 15:19:12,072::task::595::TaskManager.Task::(_updateState) Task=`90951680-32f2-4eab-a39a-2a7478274a2e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:19:12,072::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:19:12,072::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:19:12,073::task::990::TaskManager.Task::(_decref) Task=`90951680-32f2-4eab-a39a-2a7478274a2e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:19:27,210::task::595::TaskManager.Task::(_updateState) Task=`45cedd2a-ab84-4145-b7cf-b621edb23e66`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:19:27,210::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:19:27,210::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:19:27,210::task::1185::TaskManager.Task::(prepare) Task=`45cedd2a-ab84-4145-b7cf-b621edb23e66`::finished: {}
Thread-13::DEBUG::2014-11-28 15:19:27,210::task::595::TaskManager.Task::(_updateState) Task=`45cedd2a-ab84-4145-b7cf-b621edb23e66`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:19:27,211::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:19:27,211::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:19:27,211::task::990::TaskManager.Task::(_decref) Task=`45cedd2a-ab84-4145-b7cf-b621edb23e66`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:19:42,315::task::595::TaskManager.Task::(_updateState) Task=`d5a123cb-7efc-442b-bc35-1ba530c31c11`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:19:42,316::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:19:42,316::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:19:42,316::task::1185::TaskManager.Task::(prepare) Task=`d5a123cb-7efc-442b-bc35-1ba530c31c11`::finished: {}
Thread-13::DEBUG::2014-11-28 15:19:42,316::task::595::TaskManager.Task::(_updateState) Task=`d5a123cb-7efc-442b-bc35-1ba530c31c11`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:19:42,317::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:19:42,317::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:19:42,317::task::990::TaskManager.Task::(_decref) Task=`d5a123cb-7efc-442b-bc35-1ba530c31c11`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:19:57,411::task::595::TaskManager.Task::(_updateState) Task=`8b4feb42-23a2-4142-a784-e39b36f0a9eb`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:19:57,412::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:19:57,412::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:19:57,412::task::1185::TaskManager.Task::(prepare) Task=`8b4feb42-23a2-4142-a784-e39b36f0a9eb`::finished: {}
Thread-13::DEBUG::2014-11-28 15:19:57,412::task::595::TaskManager.Task::(_updateState) Task=`8b4feb42-23a2-4142-a784-e39b36f0a9eb`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:19:57,412::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:19:57,413::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:19:57,413::task::990::TaskManager.Task::(_decref) Task=`8b4feb42-23a2-4142-a784-e39b36f0a9eb`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:19:59,908::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [6ca1bd52]
Thread-13::DEBUG::2014-11-28 15:19:59,908::task::595::TaskManager.Task::(_updateState) Task=`35c7ffc9-0625-443f-91d8-cde0ac89f29a`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:19:59,909::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:19:59,915::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::ERROR::2014-11-28 15:19:59,934::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::ERROR::2014-11-28 15:19:59,935::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::DEBUG::2014-11-28 15:19:59,935::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:19:59,935::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 15:19:59,935::task::1185::TaskManager.Task::(prepare) Task=`35c7ffc9-0625-443f-91d8-cde0ac89f29a`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 15:19:59,936::task::595::TaskManager.Task::(_updateState) Task=`35c7ffc9-0625-443f-91d8-cde0ac89f29a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:19:59,936::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:19:59,936::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:19:59,936::task::990::TaskManager.Task::(_decref) Task=`35c7ffc9-0625-443f-91d8-cde0ac89f29a`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:20:00,081::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [240bb2d1]
Thread-13::DEBUG::2014-11-28 15:20:00,082::task::595::TaskManager.Task::(_updateState) Task=`27ff3740-f014-473a-b644-7cea608d9f28`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:20:00,082::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:20:00,105::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-25::DEBUG::2014-11-28 15:20:12,553::task::595::TaskManager.Task::(_updateState) Task=`9f3bf93f-4cab-44b6-813b-b26a10910092`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:20:12,554::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:20:12,554::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:20:12,554::task::1185::TaskManager.Task::(prepare) Task=`9f3bf93f-4cab-44b6-813b-b26a10910092`::finished: {}
Thread-25::DEBUG::2014-11-28 15:20:12,554::task::595::TaskManager.Task::(_updateState) Task=`9f3bf93f-4cab-44b6-813b-b26a10910092`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:20:12,555::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:20:12,555::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:20:12,555::task::990::TaskManager.Task::(_decref) Task=`9f3bf93f-4cab-44b6-813b-b26a10910092`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:20:27,697::task::595::TaskManager.Task::(_updateState) Task=`ad6daadc-f73a-49f4-980d-25894b928d69`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:20:27,697::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:20:27,697::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:20:27,697::task::1185::TaskManager.Task::(prepare) Task=`ad6daadc-f73a-49f4-980d-25894b928d69`::finished: {}
Thread-25::DEBUG::2014-11-28 15:20:27,698::task::595::TaskManager.Task::(_updateState) Task=`ad6daadc-f73a-49f4-980d-25894b928d69`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:20:27,698::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:20:27,698::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:20:27,698::task::990::TaskManager.Task::(_decref) Task=`ad6daadc-f73a-49f4-980d-25894b928d69`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:20:42,821::task::595::TaskManager.Task::(_updateState) Task=`d4b50f42-f422-4980-8f75-3935fdb33240`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:20:42,822::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:20:42,822::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:20:42,822::task::1185::TaskManager.Task::(prepare) Task=`d4b50f42-f422-4980-8f75-3935fdb33240`::finished: {}
Thread-25::DEBUG::2014-11-28 15:20:42,822::task::595::TaskManager.Task::(_updateState) Task=`d4b50f42-f422-4980-8f75-3935fdb33240`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:20:42,822::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:20:42,822::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:20:42,823::task::990::TaskManager.Task::(_decref) Task=`d4b50f42-f422-4980-8f75-3935fdb33240`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:21:44,631::task::595::TaskManager.Task::(_updateState) Task=`67f34f19-d379-4047-be80-5bf0e96d5cf8`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:21:44,632::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:21:44,632::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:21:44,632::task::1185::TaskManager.Task::(prepare) Task=`67f34f19-d379-4047-be80-5bf0e96d5cf8`::finished: {}
Thread-25::DEBUG::2014-11-28 15:21:44,632::task::595::TaskManager.Task::(_updateState) Task=`67f34f19-d379-4047-be80-5bf0e96d5cf8`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:21:44,632::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:21:44,633::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:21:44,633::task::990::TaskManager.Task::(_decref) Task=`67f34f19-d379-4047-be80-5bf0e96d5cf8`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:21:59,724::task::595::TaskManager.Task::(_updateState) Task=`76534560-0678-408d-b96f-5a75ae45293c`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:21:59,724::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:21:59,724::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:21:59,724::task::1185::TaskManager.Task::(prepare) Task=`76534560-0678-408d-b96f-5a75ae45293c`::finished: {}
Thread-25::DEBUG::2014-11-28 15:21:59,725::task::595::TaskManager.Task::(_updateState) Task=`76534560-0678-408d-b96f-5a75ae45293c`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:21:59,725::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:21:59,725::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:21:59,725::task::990::TaskManager.Task::(_decref) Task=`76534560-0678-408d-b96f-5a75ae45293c`::ref 0 aborting False
Thread-13::ERROR::2014-11-28 15:22:11,218::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 15:22:11,218::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 15:22:11,219::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:22:11,219::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,219::task::1185::TaskManager.Task::(prepare) Task=`27ff3740-f014-473a-b644-7cea608d9f28`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,219::task::595::TaskManager.Task::(_updateState) Task=`27ff3740-f014-473a-b644-7cea608d9f28`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:11,220::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:11,220::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:11,220::task::990::TaskManager.Task::(_decref) Task=`27ff3740-f014-473a-b644-7cea608d9f28`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,359::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [11965d4b]
Thread-13::DEBUG::2014-11-28 15:22:11,360::task::595::TaskManager.Task::(_updateState) Task=`a492cce9-f863-404a-8116-93dd60323cbd`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,360::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,366::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::ERROR::2014-11-28 15:22:11,381::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::ERROR::2014-11-28 15:22:11,382::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::DEBUG::2014-11-28 15:22:11,382::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:22:11,383::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,383::task::1185::TaskManager.Task::(prepare) Task=`a492cce9-f863-404a-8116-93dd60323cbd`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,383::task::595::TaskManager.Task::(_updateState) Task=`a492cce9-f863-404a-8116-93dd60323cbd`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:11,383::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:11,383::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:11,384::task::990::TaskManager.Task::(_decref) Task=`a492cce9-f863-404a-8116-93dd60323cbd`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,487::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [4c295759]
Thread-13::DEBUG::2014-11-28 15:22:11,488::task::595::TaskManager.Task::(_updateState) Task=`1f5f5e4d-0317-4ccb-b256-2ca7973d0a33`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,488::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,494::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:22:11,495::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:22:11,496::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:22:11,496::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,496::task::1185::TaskManager.Task::(prepare) Task=`1f5f5e4d-0317-4ccb-b256-2ca7973d0a33`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:22:11,496::task::595::TaskManager.Task::(_updateState) Task=`1f5f5e4d-0317-4ccb-b256-2ca7973d0a33`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:22:11,496::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:22:11,497::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:22:11,497::task::990::TaskManager.Task::(_decref) Task=`1f5f5e4d-0317-4ccb-b256-2ca7973d0a33`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:22:11,756::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [75170777]
Thread-13::DEBUG::2014-11-28 15:22:11,756::task::595::TaskManager.Task::(_updateState) Task=`aa0a3c31-2375-4f28-ae02-68c630a61c87`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:22:11,756::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:22:11,762::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-25::DEBUG::2014-11-28 15:22:14,818::task::595::TaskManager.Task::(_updateState) Task=`cc2ebe5d-b66a-4175-a9da-95daa55b8e97`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:22:14,818::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:22:14,818::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:22:14,818::task::1185::TaskManager.Task::(prepare) Task=`cc2ebe5d-b66a-4175-a9da-95daa55b8e97`::finished: {}
Thread-25::DEBUG::2014-11-28 15:22:14,819::task::595::TaskManager.Task::(_updateState) Task=`cc2ebe5d-b66a-4175-a9da-95daa55b8e97`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:22:14,819::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:22:14,819::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:22:14,819::task::990::TaskManager.Task::(_decref) Task=`cc2ebe5d-b66a-4175-a9da-95daa55b8e97`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:22:29,909::task::595::TaskManager.Task::(_updateState) Task=`25258ee5-bb02-4bd2-a8a4-ed91263d5fa3`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:22:29,909::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:22:29,909::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:22:29,909::task::1185::TaskManager.Task::(prepare) Task=`25258ee5-bb02-4bd2-a8a4-ed91263d5fa3`::finished: {}
Thread-25::DEBUG::2014-11-28 15:22:29,910::task::595::TaskManager.Task::(_updateState) Task=`25258ee5-bb02-4bd2-a8a4-ed91263d5fa3`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:22:29,910::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:22:29,911::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:22:29,912::task::990::TaskManager.Task::(_decref) Task=`25258ee5-bb02-4bd2-a8a4-ed91263d5fa3`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:22:45,009::task::595::TaskManager.Task::(_updateState) Task=`01c50130-6968-4bc8-b7af-3701275b07e9`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:22:45,010::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:22:45,010::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:22:45,010::task::1185::TaskManager.Task::(prepare) Task=`01c50130-6968-4bc8-b7af-3701275b07e9`::finished: {}
Thread-25::DEBUG::2014-11-28 15:22:45,010::task::595::TaskManager.Task::(_updateState) Task=`01c50130-6968-4bc8-b7af-3701275b07e9`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:22:45,010::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:22:45,011::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:22:45,011::task::990::TaskManager.Task::(_decref) Task=`01c50130-6968-4bc8-b7af-3701275b07e9`::ref 0 aborting False
Thread-25::DEBUG::2014-11-28 15:23:00,101::task::595::TaskManager.Task::(_updateState) Task=`e9e38dc3-1ca8-4065-8ac6-37dee4b61c9e`::moving from state init -> state preparing
Thread-25::INFO::2014-11-28 15:23:00,102::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-25::INFO::2014-11-28 15:23:00,102::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-25::DEBUG::2014-11-28 15:23:00,102::task::1185::TaskManager.Task::(prepare) Task=`e9e38dc3-1ca8-4065-8ac6-37dee4b61c9e`::finished: {}
Thread-25::DEBUG::2014-11-28 15:23:00,102::task::595::TaskManager.Task::(_updateState) Task=`e9e38dc3-1ca8-4065-8ac6-37dee4b61c9e`::moving from state preparing -> state finished
Thread-25::DEBUG::2014-11-28 15:23:00,102::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-25::DEBUG::2014-11-28 15:23:00,103::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-25::DEBUG::2014-11-28 15:23:00,103::task::990::TaskManager.Task::(_decref) Task=`e9e38dc3-1ca8-4065-8ac6-37dee4b61c9e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:11,763::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:23:11,763::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:23:11,763::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:23:11,764::task::1185::TaskManager.Task::(prepare) Task=`aa0a3c31-2375-4f28-ae02-68c630a61c87`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:23:11,764::task::595::TaskManager.Task::(_updateState) Task=`aa0a3c31-2375-4f28-ae02-68c630a61c87`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:11,764::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:11,764::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:11,764::task::990::TaskManager.Task::(_decref) Task=`aa0a3c31-2375-4f28-ae02-68c630a61c87`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:15,192::task::595::TaskManager.Task::(_updateState) Task=`d04913b3-90ff-458d-9bc9-adfc59e61a81`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:23:15,193::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:23:15,193::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:23:15,193::task::1185::TaskManager.Task::(prepare) Task=`d04913b3-90ff-458d-9bc9-adfc59e61a81`::finished: {}
Thread-13::DEBUG::2014-11-28 15:23:15,193::task::595::TaskManager.Task::(_updateState) Task=`d04913b3-90ff-458d-9bc9-adfc59e61a81`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:15,193::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:15,194::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:15,194::task::990::TaskManager.Task::(_decref) Task=`d04913b3-90ff-458d-9bc9-adfc59e61a81`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:30,299::task::595::TaskManager.Task::(_updateState) Task=`d816ff31-d7ce-4fbc-b356-1254959ca009`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:23:30,300::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:23:30,300::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:23:30,300::task::1185::TaskManager.Task::(prepare) Task=`d816ff31-d7ce-4fbc-b356-1254959ca009`::finished: {}
Thread-13::DEBUG::2014-11-28 15:23:30,300::task::595::TaskManager.Task::(_updateState) Task=`d816ff31-d7ce-4fbc-b356-1254959ca009`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:30,301::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:30,301::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:30,301::task::990::TaskManager.Task::(_decref) Task=`d816ff31-d7ce-4fbc-b356-1254959ca009`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:23:45,387::task::595::TaskManager.Task::(_updateState) Task=`70926a23-d474-4358-9aa9-56d4a82b810f`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:23:45,387::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:23:45,387::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:23:45,388::task::1185::TaskManager.Task::(prepare) Task=`70926a23-d474-4358-9aa9-56d4a82b810f`::finished: {}
Thread-13::DEBUG::2014-11-28 15:23:45,388::task::595::TaskManager.Task::(_updateState) Task=`70926a23-d474-4358-9aa9-56d4a82b810f`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:23:45,388::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:23:45,388::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:23:45,388::task::990::TaskManager.Task::(_decref) Task=`70926a23-d474-4358-9aa9-56d4a82b810f`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:24:00,476::task::595::TaskManager.Task::(_updateState) Task=`6086d8db-d857-40bc-b97c-5f0e5e1eb757`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:24:00,476::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:24:00,477::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:24:00,477::task::1185::TaskManager.Task::(prepare) Task=`6086d8db-d857-40bc-b97c-5f0e5e1eb757`::finished: {}
Thread-13::DEBUG::2014-11-28 15:24:00,477::task::595::TaskManager.Task::(_updateState) Task=`6086d8db-d857-40bc-b97c-5f0e5e1eb757`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:24:00,477::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:24:00,477::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:24:00,478::task::990::TaskManager.Task::(_decref) Task=`6086d8db-d857-40bc-b97c-5f0e5e1eb757`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:24:15,574::task::595::TaskManager.Task::(_updateState) Task=`21f166e8-0a1a-4975-b7e4-6bfad67c1027`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:24:15,574::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:24:15,575::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:24:15,575::task::1185::TaskManager.Task::(prepare) Task=`21f166e8-0a1a-4975-b7e4-6bfad67c1027`::finished: {}
Thread-13::DEBUG::2014-11-28 15:24:15,575::task::595::TaskManager.Task::(_updateState) Task=`21f166e8-0a1a-4975-b7e4-6bfad67c1027`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:24:15,575::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:24:15,575::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:24:15,575::task::990::TaskManager.Task::(_decref) Task=`21f166e8-0a1a-4975-b7e4-6bfad67c1027`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:24:30,666::task::595::TaskManager.Task::(_updateState) Task=`49178440-685e-41d4-b048-a3143fb38994`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:24:30,666::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:24:30,666::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:24:30,666::task::1185::TaskManager.Task::(prepare) Task=`49178440-685e-41d4-b048-a3143fb38994`::finished: {}
Thread-13::DEBUG::2014-11-28 15:24:30,667::task::595::TaskManager.Task::(_updateState) Task=`49178440-685e-41d4-b048-a3143fb38994`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:24:30,667::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:24:30,667::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:24:30,667::task::990::TaskManager.Task::(_decref) Task=`49178440-685e-41d4-b048-a3143fb38994`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:24:45,758::task::595::TaskManager.Task::(_updateState) Task=`ac768d2a-efd1-46da-86bb-2ca7ddab4aed`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:24:45,759::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:24:45,759::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:24:45,759::task::1185::TaskManager.Task::(prepare) Task=`ac768d2a-efd1-46da-86bb-2ca7ddab4aed`::finished: {}
Thread-13::DEBUG::2014-11-28 15:24:45,759::task::595::TaskManager.Task::(_updateState) Task=`ac768d2a-efd1-46da-86bb-2ca7ddab4aed`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:24:45,759::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:24:45,759::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:24:45,760::task::990::TaskManager.Task::(_decref) Task=`ac768d2a-efd1-46da-86bb-2ca7ddab4aed`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:24:59,914::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [456176bb]
Thread-13::DEBUG::2014-11-28 15:24:59,915::task::595::TaskManager.Task::(_updateState) Task=`c1997baa-a93c-451a-90f6-9bfa75c6968e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:24:59,915::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:24:59,921::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::ERROR::2014-11-28 15:24:59,939::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::ERROR::2014-11-28 15:24:59,940::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-13::DEBUG::2014-11-28 15:24:59,940::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:24:59,940::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 15:24:59,940::task::1185::TaskManager.Task::(prepare) Task=`c1997baa-a93c-451a-90f6-9bfa75c6968e`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-13::DEBUG::2014-11-28 15:24:59,940::task::595::TaskManager.Task::(_updateState) Task=`c1997baa-a93c-451a-90f6-9bfa75c6968e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:24:59,941::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:24:59,941::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:24:59,941::task::990::TaskManager.Task::(_decref) Task=`c1997baa-a93c-451a-90f6-9bfa75c6968e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:25:00,110::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7dd50692]
Thread-13::DEBUG::2014-11-28 15:25:00,110::task::595::TaskManager.Task::(_updateState) Task=`17b346ae-5883-4655-b7dd-1bbffd4d412e`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:25:00,111::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:25:00,119::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-26::DEBUG::2014-11-28 15:25:00,848::task::595::TaskManager.Task::(_updateState) Task=`74f6e1c1-3826-411e-9110-07ccb94ebc1b`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:25:00,848::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:25:00,848::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:25:00,848::task::1185::TaskManager.Task::(prepare) Task=`74f6e1c1-3826-411e-9110-07ccb94ebc1b`::finished: {}
Thread-26::DEBUG::2014-11-28 15:25:00,848::task::595::TaskManager.Task::(_updateState) Task=`74f6e1c1-3826-411e-9110-07ccb94ebc1b`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:25:00,849::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:25:00,849::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:25:00,849::task::990::TaskManager.Task::(_decref) Task=`74f6e1c1-3826-411e-9110-07ccb94ebc1b`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:25:15,957::task::595::TaskManager.Task::(_updateState) Task=`c0ff1da7-2345-4e2a-b24a-9de9767dd119`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:25:15,958::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:25:15,958::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:25:15,958::task::1185::TaskManager.Task::(prepare) Task=`c0ff1da7-2345-4e2a-b24a-9de9767dd119`::finished: {}
Thread-26::DEBUG::2014-11-28 15:25:15,958::task::595::TaskManager.Task::(_updateState) Task=`c0ff1da7-2345-4e2a-b24a-9de9767dd119`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:25:15,958::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:25:15,958::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:25:15,959::task::990::TaskManager.Task::(_decref) Task=`c0ff1da7-2345-4e2a-b24a-9de9767dd119`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:25:31,066::task::595::TaskManager.Task::(_updateState) Task=`91c2dd0c-8481-4583-9bee-ca7ea208cca9`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:25:31,067::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:25:31,067::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:25:31,067::task::1185::TaskManager.Task::(prepare) Task=`91c2dd0c-8481-4583-9bee-ca7ea208cca9`::finished: {}
Thread-26::DEBUG::2014-11-28 15:25:31,067::task::595::TaskManager.Task::(_updateState) Task=`91c2dd0c-8481-4583-9bee-ca7ea208cca9`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:25:31,067::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:25:31,068::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:25:31,068::task::990::TaskManager.Task::(_decref) Task=`91c2dd0c-8481-4583-9bee-ca7ea208cca9`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:25:46,154::task::595::TaskManager.Task::(_updateState) Task=`4f575cc1-fba6-462a-9c37-de03fb6c3aad`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:25:46,154::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:25:46,154::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:25:46,154::task::1185::TaskManager.Task::(prepare) Task=`4f575cc1-fba6-462a-9c37-de03fb6c3aad`::finished: {}
Thread-26::DEBUG::2014-11-28 15:25:46,155::task::595::TaskManager.Task::(_updateState) Task=`4f575cc1-fba6-462a-9c37-de03fb6c3aad`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:25:46,155::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:25:46,155::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:25:46,155::task::990::TaskManager.Task::(_decref) Task=`4f575cc1-fba6-462a-9c37-de03fb6c3aad`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:26:01,242::task::595::TaskManager.Task::(_updateState) Task=`48ffc902-952f-47ac-b273-a5ab93febdf1`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:01,242::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:26:01,242::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:26:01,242::task::1185::TaskManager.Task::(prepare) Task=`48ffc902-952f-47ac-b273-a5ab93febdf1`::finished: {}
Thread-26::DEBUG::2014-11-28 15:26:01,243::task::595::TaskManager.Task::(_updateState) Task=`48ffc902-952f-47ac-b273-a5ab93febdf1`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:26:01,243::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:26:01,243::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:26:01,243::task::990::TaskManager.Task::(_decref) Task=`48ffc902-952f-47ac-b273-a5ab93febdf1`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:26:16,332::task::595::TaskManager.Task::(_updateState) Task=`957d2b09-5d42-46e2-899e-603e83ff4d93`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:16,332::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:26:16,332::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:26:16,332::task::1185::TaskManager.Task::(prepare) Task=`957d2b09-5d42-46e2-899e-603e83ff4d93`::finished: {}
Thread-26::DEBUG::2014-11-28 15:26:16,332::task::595::TaskManager.Task::(_updateState) Task=`957d2b09-5d42-46e2-899e-603e83ff4d93`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:26:16,333::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:26:16,333::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:26:16,333::task::990::TaskManager.Task::(_decref) Task=`957d2b09-5d42-46e2-899e-603e83ff4d93`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:26:31,437::task::595::TaskManager.Task::(_updateState) Task=`709e3301-3cb4-4628-91a3-fab5f5d9530f`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:31,438::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:26:31,438::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:26:31,438::task::1185::TaskManager.Task::(prepare) Task=`709e3301-3cb4-4628-91a3-fab5f5d9530f`::finished: {}
Thread-26::DEBUG::2014-11-28 15:26:31,438::task::595::TaskManager.Task::(_updateState) Task=`709e3301-3cb4-4628-91a3-fab5f5d9530f`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:26:31,438::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:26:31,439::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:26:31,439::task::990::TaskManager.Task::(_decref) Task=`709e3301-3cb4-4628-91a3-fab5f5d9530f`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:26:46,525::task::595::TaskManager.Task::(_updateState) Task=`00b235a2-df3f-4386-bb78-4937888a6f98`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:46,526::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:26:46,526::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:26:46,526::task::1185::TaskManager.Task::(prepare) Task=`00b235a2-df3f-4386-bb78-4937888a6f98`::finished: {}
Thread-26::DEBUG::2014-11-28 15:26:46,526::task::595::TaskManager.Task::(_updateState) Task=`00b235a2-df3f-4386-bb78-4937888a6f98`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:26:46,526::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:26:46,526::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:26:46,527::task::990::TaskManager.Task::(_decref) Task=`00b235a2-df3f-4386-bb78-4937888a6f98`::ref 0 aborting False
Thread-26::DEBUG::2014-11-28 15:26:53,550::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-26::DEBUG::2014-11-28 15:26:53,551::task::595::TaskManager.Task::(_updateState) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:53,551::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-26::ERROR::2014-11-28 15:26:53,551::task::866::TaskManager.Task::(_setError) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-26::DEBUG::2014-11-28 15:26:53,552::task::885::TaskManager.Task::(_run) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::Task._run: 709c079f-5886-479a-9c90-20cd7d6c8483 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-26::DEBUG::2014-11-28 15:26:53,552::task::1211::TaskManager.Task::(stop) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::stopping in state preparing (force False)
Thread-26::DEBUG::2014-11-28 15:26:53,552::task::990::TaskManager.Task::(_decref) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::ref 1 aborting True
Thread-26::INFO::2014-11-28 15:26:53,552::task::1168::TaskManager.Task::(prepare) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-26::DEBUG::2014-11-28 15:26:53,552::task::1173::TaskManager.Task::(prepare) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::Prepare: aborted: Unknown pool id, pool not connected
Thread-26::DEBUG::2014-11-28 15:26:53,552::task::990::TaskManager.Task::(_decref) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::ref 0 aborting True
Thread-26::DEBUG::2014-11-28 15:26:53,553::task::925::TaskManager.Task::(_doAbort) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::Task._doAbort: force False
Thread-26::DEBUG::2014-11-28 15:26:53,553::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:26:53,553::task::595::TaskManager.Task::(_updateState) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::moving from state preparing -> state aborting
Thread-26::DEBUG::2014-11-28 15:26:53,553::task::550::TaskManager.Task::(__state_aborting) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::_aborting: recover policy none
Thread-26::DEBUG::2014-11-28 15:26:53,553::task::595::TaskManager.Task::(_updateState) Task=`709c079f-5886-479a-9c90-20cd7d6c8483`::moving from state aborting -> state failed
Thread-26::DEBUG::2014-11-28 15:26:53,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:26:53,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::ERROR::2014-11-28 15:26:53,554::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-26::DEBUG::2014-11-28 15:26:53,569::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-26::DEBUG::2014-11-28 15:26:53,569::task::595::TaskManager.Task::(_updateState) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:26:53,570::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24237, domainsMap=None, options=None)
Thread-26::DEBUG::2014-11-28 15:26:53,570::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e90b852b-c8e9-4c09-894b-8ee0968d3c72`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-26::DEBUG::2014-11-28 15:26:53,570::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-26::DEBUG::2014-11-28 15:26:53,571::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-26::DEBUG::2014-11-28 15:26:53,571::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`e90b852b-c8e9-4c09-894b-8ee0968d3c72`::Granted request
Thread-26::DEBUG::2014-11-28 15:26:53,571::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`5860540d-f5e5-44c9-a684-d015c02a5a27`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-26::DEBUG::2014-11-28 15:26:53,571::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-26::DEBUG::2014-11-28 15:26:53,572::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-26::DEBUG::2014-11-28 15:26:53,572::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`5860540d-f5e5-44c9-a684-d015c02a5a27`::Granted request
Thread-26::INFO::2014-11-28 15:26:53,572::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c702482d8
Thread-26::INFO::2014-11-28 15:26:53,573::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24237)
Thread-26::DEBUG::2014-11-28 15:26:53,573::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,573::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,573::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,574::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,574::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,574::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,574::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-26::DEBUG::2014-11-28 15:26:53,574::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-26::DEBUG::2014-11-28 15:26:53,575::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-26::DEBUG::2014-11-28 15:26:53,575::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-26::DEBUG::2014-11-28 15:26:53,575::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-26::DEBUG::2014-11-28 15:26:53,575::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-26::DEBUG::2014-11-28 15:26:53,588::misc::748::SamplingMethod::(__call__) Returning last result
Thread-26::DEBUG::2014-11-28 15:26:53,590::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-26::DEBUG::2014-11-28 15:26:53,642::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-26::DEBUG::2014-11-28 15:26:53,642::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,643::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,643::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,643::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,643::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,644::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-26::DEBUG::2014-11-28 15:26:53,644::misc::748::SamplingMethod::(__call__) Returning last result
Thread-26::DEBUG::2014-11-28 15:26:56,808::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-26::DEBUG::2014-11-28 15:26:56,955::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-26::DEBUG::2014-11-28 15:26:56,958::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-13::ERROR::2014-11-28 15:27:11,230::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 15:27:11,231::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 15:27:11,231::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:27:11,231::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 15:27:11,231::task::1185::TaskManager.Task::(prepare) Task=`17b346ae-5883-4655-b7dd-1bbffd4d412e`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-13::DEBUG::2014-11-28 15:27:11,232::task::595::TaskManager.Task::(_updateState) Task=`17b346ae-5883-4655-b7dd-1bbffd4d412e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:27:11,232::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:27:11,232::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:27:11,232::task::990::TaskManager.Task::(_decref) Task=`17b346ae-5883-4655-b7dd-1bbffd4d412e`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:27:11,397::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [57f87563]
Thread-13::DEBUG::2014-11-28 15:27:11,398::task::595::TaskManager.Task::(_updateState) Task=`782706c5-a53d-4e03-a1c3-cde28926efa5`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:27:11,398::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:27:11,404::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 15:27:11,551::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 15:27:11,552::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:27:11,552::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:27:11,552::task::1185::TaskManager.Task::(prepare) Task=`782706c5-a53d-4e03-a1c3-cde28926efa5`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 15:27:11,552::task::595::TaskManager.Task::(_updateState) Task=`782706c5-a53d-4e03-a1c3-cde28926efa5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:27:11,553::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:27:11,553::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:27:11,553::task::990::TaskManager.Task::(_decref) Task=`782706c5-a53d-4e03-a1c3-cde28926efa5`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:27:11,638::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [15a720c0]
Thread-13::DEBUG::2014-11-28 15:27:11,638::task::595::TaskManager.Task::(_updateState) Task=`a111fbca-168b-4330-95be-ac955edc4054`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:27:11,639::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:27:11,645::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-26::DEBUG::2014-11-28 15:27:56,960::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-26::DEBUG::2014-11-28 15:27:56,960::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-26::DEBUG::2014-11-28 15:27:56,960::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-26::DEBUG::2014-11-28 15:27:56,960::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-26::DEBUG::2014-11-28 15:27:56,961::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-26::DEBUG::2014-11-28 15:27:56,961::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-26::DEBUG::2014-11-28 15:27:56,961::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-26::DEBUG::2014-11-28 15:27:56,961::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-26::ERROR::2014-11-28 15:27:56,961::task::866::TaskManager.Task::(_setError) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-26::DEBUG::2014-11-28 15:27:56,962::task::885::TaskManager.Task::(_run) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::Task._run: 1a307c74-c14b-42f1-9b14-6f2bb302c371 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24237, None) {} failed - stopping task
Thread-26::DEBUG::2014-11-28 15:27:56,962::task::1211::TaskManager.Task::(stop) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::stopping in state preparing (force False)
Thread-26::DEBUG::2014-11-28 15:27:56,963::task::990::TaskManager.Task::(_decref) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::ref 1 aborting True
Thread-26::DEBUG::2014-11-28 15:27:56,963::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-26::INFO::2014-11-28 15:27:56,963::task::1168::TaskManager.Task::(prepare) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::aborting: Task is aborted: u'' - code 100
Thread-26::DEBUG::2014-11-28 15:27:56,963::task::1173::TaskManager.Task::(prepare) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::Prepare: aborted: 
Thread-26::DEBUG::2014-11-28 15:27:56,964::task::990::TaskManager.Task::(_decref) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::ref 0 aborting True
Thread-26::DEBUG::2014-11-28 15:27:56,964::task::925::TaskManager.Task::(_doAbort) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::Task._doAbort: force False
Thread-26::DEBUG::2014-11-28 15:27:56,964::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:27:56,964::task::595::TaskManager.Task::(_updateState) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::moving from state preparing -> state aborting
Thread-26::DEBUG::2014-11-28 15:27:56,964::task::550::TaskManager.Task::(__state_aborting) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::_aborting: recover policy none
Thread-26::DEBUG::2014-11-28 15:27:56,964::task::595::TaskManager.Task::(_updateState) Task=`1a307c74-c14b-42f1-9b14-6f2bb302c371`::moving from state aborting -> state failed
Thread-26::DEBUG::2014-11-28 15:27:56,965::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:27:56,965::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::ERROR::2014-11-28 15:27:56,965::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-26::DEBUG::2014-11-28 15:27:56,988::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-26::DEBUG::2014-11-28 15:27:56,988::task::595::TaskManager.Task::(_updateState) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:27:56,988::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-26::ERROR::2014-11-28 15:27:56,989::task::866::TaskManager.Task::(_setError) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-26::DEBUG::2014-11-28 15:27:56,989::task::885::TaskManager.Task::(_run) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::Task._run: c10b11fc-0313-47dd-abc2-6618e9132681 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-26::DEBUG::2014-11-28 15:27:56,989::task::1211::TaskManager.Task::(stop) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::stopping in state preparing (force False)
Thread-26::DEBUG::2014-11-28 15:27:56,989::task::990::TaskManager.Task::(_decref) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::ref 1 aborting True
Thread-26::INFO::2014-11-28 15:27:56,989::task::1168::TaskManager.Task::(prepare) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-26::DEBUG::2014-11-28 15:27:56,990::task::1173::TaskManager.Task::(prepare) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::Prepare: aborted: Unknown pool id, pool not connected
Thread-26::DEBUG::2014-11-28 15:27:56,990::task::990::TaskManager.Task::(_decref) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::ref 0 aborting True
Thread-26::DEBUG::2014-11-28 15:27:56,990::task::925::TaskManager.Task::(_doAbort) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::Task._doAbort: force False
Thread-26::DEBUG::2014-11-28 15:27:56,990::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:27:56,990::task::595::TaskManager.Task::(_updateState) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::moving from state preparing -> state aborting
Thread-26::DEBUG::2014-11-28 15:27:56,991::task::550::TaskManager.Task::(__state_aborting) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::_aborting: recover policy none
Thread-26::DEBUG::2014-11-28 15:27:56,991::task::595::TaskManager.Task::(_updateState) Task=`c10b11fc-0313-47dd-abc2-6618e9132681`::moving from state aborting -> state failed
Thread-26::DEBUG::2014-11-28 15:27:56,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:27:56,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::ERROR::2014-11-28 15:27:56,991::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-26::DEBUG::2014-11-28 15:27:59,982::task::595::TaskManager.Task::(_updateState) Task=`97b93c07-68c7-4df7-921d-67d81d7a9a41`::moving from state init -> state preparing
Thread-26::INFO::2014-11-28 15:27:59,983::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-26::INFO::2014-11-28 15:27:59,983::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-26::DEBUG::2014-11-28 15:27:59,983::task::1185::TaskManager.Task::(prepare) Task=`97b93c07-68c7-4df7-921d-67d81d7a9a41`::finished: {}
Thread-26::DEBUG::2014-11-28 15:27:59,983::task::595::TaskManager.Task::(_updateState) Task=`97b93c07-68c7-4df7-921d-67d81d7a9a41`::moving from state preparing -> state finished
Thread-26::DEBUG::2014-11-28 15:27:59,983::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-26::DEBUG::2014-11-28 15:27:59,983::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-26::DEBUG::2014-11-28 15:27:59,984::task::990::TaskManager.Task::(_decref) Task=`97b93c07-68c7-4df7-921d-67d81d7a9a41`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:28:11,646::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::DEBUG::2014-11-28 15:28:11,646::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:28:11,646::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:28:11,647::task::1185::TaskManager.Task::(prepare) Task=`a111fbca-168b-4330-95be-ac955edc4054`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 15:28:11,647::task::595::TaskManager.Task::(_updateState) Task=`a111fbca-168b-4330-95be-ac955edc4054`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:28:11,647::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:28:11,647::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:28:11,647::task::990::TaskManager.Task::(_decref) Task=`a111fbca-168b-4330-95be-ac955edc4054`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:28:11,739::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [20138430]
Thread-13::DEBUG::2014-11-28 15:28:11,740::task::595::TaskManager.Task::(_updateState) Task=`830fd230-19b4-4ee7-b288-401650a2a2fc`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:28:11,740::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 15:28:11,746::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::ERROR::2014-11-28 15:28:11,763::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::ERROR::2014-11-28 15:28:11,764::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::DEBUG::2014-11-28 15:28:11,764::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 15:28:11,764::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:28:11,765::task::1185::TaskManager.Task::(prepare) Task=`830fd230-19b4-4ee7-b288-401650a2a2fc`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 15:28:11,765::task::595::TaskManager.Task::(_updateState) Task=`830fd230-19b4-4ee7-b288-401650a2a2fc`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:28:11,765::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:28:11,765::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:28:11,765::task::990::TaskManager.Task::(_decref) Task=`830fd230-19b4-4ee7-b288-401650a2a2fc`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:28:15,066::task::595::TaskManager.Task::(_updateState) Task=`d84ac4bc-f4b6-4eb1-806d-3f2a9c7f9289`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:28:15,066::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:28:15,067::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:28:15,067::task::1185::TaskManager.Task::(prepare) Task=`d84ac4bc-f4b6-4eb1-806d-3f2a9c7f9289`::finished: {}
Thread-13::DEBUG::2014-11-28 15:28:15,067::task::595::TaskManager.Task::(_updateState) Task=`d84ac4bc-f4b6-4eb1-806d-3f2a9c7f9289`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:28:15,067::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:28:15,067::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:28:15,068::task::990::TaskManager.Task::(_decref) Task=`d84ac4bc-f4b6-4eb1-806d-3f2a9c7f9289`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:28:30,155::task::595::TaskManager.Task::(_updateState) Task=`17ece54b-42aa-428e-b207-8f522ee8ef1f`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:28:30,155::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:28:30,155::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:28:30,156::task::1185::TaskManager.Task::(prepare) Task=`17ece54b-42aa-428e-b207-8f522ee8ef1f`::finished: {}
Thread-13::DEBUG::2014-11-28 15:28:30,156::task::595::TaskManager.Task::(_updateState) Task=`17ece54b-42aa-428e-b207-8f522ee8ef1f`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:28:30,156::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:28:30,156::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:28:30,156::task::990::TaskManager.Task::(_decref) Task=`17ece54b-42aa-428e-b207-8f522ee8ef1f`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:28:45,267::task::595::TaskManager.Task::(_updateState) Task=`76360465-3ad0-4a29-b8fc-6c7c84cec639`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:28:45,267::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:28:45,267::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:28:45,268::task::1185::TaskManager.Task::(prepare) Task=`76360465-3ad0-4a29-b8fc-6c7c84cec639`::finished: {}
Thread-13::DEBUG::2014-11-28 15:28:45,268::task::595::TaskManager.Task::(_updateState) Task=`76360465-3ad0-4a29-b8fc-6c7c84cec639`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:28:45,268::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:28:45,268::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:28:45,268::task::990::TaskManager.Task::(_decref) Task=`76360465-3ad0-4a29-b8fc-6c7c84cec639`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:00,356::task::595::TaskManager.Task::(_updateState) Task=`63109484-8585-47f0-be39-7e222277496f`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:00,356::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:29:00,356::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:29:00,357::task::1185::TaskManager.Task::(prepare) Task=`63109484-8585-47f0-be39-7e222277496f`::finished: {}
Thread-13::DEBUG::2014-11-28 15:29:00,357::task::595::TaskManager.Task::(_updateState) Task=`63109484-8585-47f0-be39-7e222277496f`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:29:00,357::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:29:00,357::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:29:00,357::task::990::TaskManager.Task::(_decref) Task=`63109484-8585-47f0-be39-7e222277496f`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:15,444::task::595::TaskManager.Task::(_updateState) Task=`7ed0a5f2-8287-49c2-88d6-d7817b3038e6`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:15,445::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:29:15,445::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:29:15,445::task::1185::TaskManager.Task::(prepare) Task=`7ed0a5f2-8287-49c2-88d6-d7817b3038e6`::finished: {}
Thread-13::DEBUG::2014-11-28 15:29:15,445::task::595::TaskManager.Task::(_updateState) Task=`7ed0a5f2-8287-49c2-88d6-d7817b3038e6`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:29:15,445::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:29:15,446::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:29:15,446::task::990::TaskManager.Task::(_decref) Task=`7ed0a5f2-8287-49c2-88d6-d7817b3038e6`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:30,530::task::595::TaskManager.Task::(_updateState) Task=`ba2d4702-8fe2-424e-9cea-6886a896cc22`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:30,531::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:29:30,531::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:29:30,531::task::1185::TaskManager.Task::(prepare) Task=`ba2d4702-8fe2-424e-9cea-6886a896cc22`::finished: {}
Thread-13::DEBUG::2014-11-28 15:29:30,531::task::595::TaskManager.Task::(_updateState) Task=`ba2d4702-8fe2-424e-9cea-6886a896cc22`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:29:30,531::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:29:30,532::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:29:30,532::task::990::TaskManager.Task::(_decref) Task=`ba2d4702-8fe2-424e-9cea-6886a896cc22`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:45,612::task::595::TaskManager.Task::(_updateState) Task=`24463d0e-c53a-4386-989e-ff302698068a`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:45,612::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 15:29:45,613::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 15:29:45,613::task::1185::TaskManager.Task::(prepare) Task=`24463d0e-c53a-4386-989e-ff302698068a`::finished: {}
Thread-13::DEBUG::2014-11-28 15:29:45,613::task::595::TaskManager.Task::(_updateState) Task=`24463d0e-c53a-4386-989e-ff302698068a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:29:45,613::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:29:45,613::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:29:45,614::task::990::TaskManager.Task::(_decref) Task=`24463d0e-c53a-4386-989e-ff302698068a`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:57,421::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1e30deed]
Thread-13::DEBUG::2014-11-28 15:29:57,421::task::595::TaskManager.Task::(_updateState) Task=`11813894-998a-4fdd-8031-314a635228dd`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:57,422::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, remove=False, options=None)
Thread-13::WARNING::2014-11-28 15:29:57,422::hsm::1126::Storage.HSM::(disconnectStoragePool) disconnect sp: 45995288-a7d6-4cbf-8b2f-63fa35642bad failed. Known pools {}
Thread-13::INFO::2014-11-28 15:29:57,422::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
Thread-13::DEBUG::2014-11-28 15:29:57,422::task::1185::TaskManager.Task::(prepare) Task=`11813894-998a-4fdd-8031-314a635228dd`::finished: None
Thread-13::DEBUG::2014-11-28 15:29:57,422::task::595::TaskManager.Task::(_updateState) Task=`11813894-998a-4fdd-8031-314a635228dd`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 15:29:57,423::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 15:29:57,423::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:29:57,423::task::990::TaskManager.Task::(_decref) Task=`11813894-998a-4fdd-8031-314a635228dd`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 15:29:57,438::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1e30deed]
Thread-13::DEBUG::2014-11-28 15:29:57,438::task::595::TaskManager.Task::(_updateState) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 15:29:57,438::logUtils::44::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', poolName='CTI', masterDom='e7eaae86-11ea-4c08-8c11-8b8c54373389', domDict={'1c4abda0-906b-4faf-a2e3-b6683729cc2f': 'active', '474f0809-26c6-4fca-ab8c-72a2e6dcd912': 'active', '54b2d645-fc18-4784-99eb-0e17e296eee7': 'active', '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': 'active', 'e7eaae86-11ea-4c08-8c11-8b8c54373389': 'active'}, masterVersion=24238, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=1, options=None)
Thread-13::INFO::2014-11-28 15:29:57,438::hsm::1862::Storage.HSM::(reconstructMaster) spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad master=e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::INFO::2014-11-28 15:29:57,439::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c01cab8
Thread-13::DEBUG::2014-11-28 15:29:57,439::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 15:29:57,439::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:29:57,439::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 15:29:57,439::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 15:29:57,440::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 15:29:57,440::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 15:29:57,452::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:29:57,453::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 15:29:57,498::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 15:29:57,498::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,498::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,499::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,499::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,499::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,499::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 15:29:57,499::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 15:29:57,508::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 15:29:57,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 15:29:57,658::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1349', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=235c792ce1a47fff85e9d8ec532bb5477b310d1f']
Thread-13::DEBUG::2014-11-28 15:29:57,660::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::DEBUG::2014-11-28 15:29:57,660::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_imageNS'
Thread-13::DEBUG::2014-11-28 15:29:57,661::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_volumeNS'
Thread-13::INFO::2014-11-28 15:29:57,661::fileSD::350::Storage.StorageDomain::(validate) sdUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 15:29:57,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1349', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=235c792ce1a47fff85e9d8ec532bb5477b310d1f']
Thread-13::DEBUG::2014-11-28 15:29:57,664::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`a432d52f-bdd5-4184-8194-9edd296cdfa8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1877' at 'reconstructMaster'
Thread-13::DEBUG::2014-11-28 15:29:57,664::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 15:29:57,664::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 15:29:57,665::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`a432d52f-bdd5-4184-8194-9edd296cdfa8`::Granted request
Thread-13::DEBUG::2014-11-28 15:29:57,665::task::827::TaskManager.Task::(resourceAcquired) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::_resourcesAcquired: Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad (exclusive)
Thread-13::DEBUG::2014-11-28 15:29:57,665::task::990::TaskManager.Task::(_decref) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::ref 1 aborting False
Thread-13::INFO::2014-11-28 15:29:57,665::sp::684::Storage.StoragePool::(reconstructMaster) spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad hostId=1 poolName=CTI msdUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389 domDict={'1c4abda0-906b-4faf-a2e3-b6683729cc2f': 'active', '474f0809-26c6-4fca-ab8c-72a2e6dcd912': 'active', '54b2d645-fc18-4784-99eb-0e17e296eee7': 'active', '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': 'active', 'e7eaae86-11ea-4c08-8c11-8b8c54373389': 'active'} masterVersion=24238 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5})
Thread-13::INFO::2014-11-28 15:29:57,666::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 (id: 1)
Thread-32::DEBUG::2014-11-28 15:29:59,969::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [219918c1]
Thread-32::DEBUG::2014-11-28 15:29:59,970::task::595::TaskManager.Task::(_updateState) Task=`072c70a6-4907-4817-a342-79c84257e213`::moving from state init -> state preparing
Thread-32::INFO::2014-11-28 15:29:59,970::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-32::DEBUG::2014-11-28 15:29:59,976::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-32::ERROR::2014-11-28 15:29:59,996::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-32::ERROR::2014-11-28 15:29:59,997::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-32::DEBUG::2014-11-28 15:29:59,997::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-32::INFO::2014-11-28 15:29:59,997::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-32::DEBUG::2014-11-28 15:29:59,997::task::1185::TaskManager.Task::(prepare) Task=`072c70a6-4907-4817-a342-79c84257e213`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-32::DEBUG::2014-11-28 15:29:59,997::task::595::TaskManager.Task::(_updateState) Task=`072c70a6-4907-4817-a342-79c84257e213`::moving from state preparing -> state finished
Thread-32::DEBUG::2014-11-28 15:29:59,998::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-32::DEBUG::2014-11-28 15:29:59,998::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::DEBUG::2014-11-28 15:29:59,998::task::990::TaskManager.Task::(_decref) Task=`072c70a6-4907-4817-a342-79c84257e213`::ref 0 aborting False
Thread-32::DEBUG::2014-11-28 15:30:00,117::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5c85512]
Thread-32::DEBUG::2014-11-28 15:30:00,117::task::595::TaskManager.Task::(_updateState) Task=`3afd752f-b3e1-40fb-8f8d-1fe0f5ed6509`::moving from state init -> state preparing
Thread-32::INFO::2014-11-28 15:30:00,117::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-32::DEBUG::2014-11-28 15:30:00,125::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-31::WARNING::2014-11-28 15:30:57,501::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-32::ERROR::2014-11-28 15:32:11,237::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-32::ERROR::2014-11-28 15:32:11,238::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-32::DEBUG::2014-11-28 15:32:11,238::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-32::INFO::2014-11-28 15:32:11,238::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-32::DEBUG::2014-11-28 15:32:11,239::task::1185::TaskManager.Task::(prepare) Task=`3afd752f-b3e1-40fb-8f8d-1fe0f5ed6509`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-32::DEBUG::2014-11-28 15:32:11,239::task::595::TaskManager.Task::(_updateState) Task=`3afd752f-b3e1-40fb-8f8d-1fe0f5ed6509`::moving from state preparing -> state finished
Thread-32::DEBUG::2014-11-28 15:32:11,239::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-32::DEBUG::2014-11-28 15:32:11,239::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::DEBUG::2014-11-28 15:32:11,239::task::990::TaskManager.Task::(_decref) Task=`3afd752f-b3e1-40fb-8f8d-1fe0f5ed6509`::ref 0 aborting False
Thread-32::DEBUG::2014-11-28 15:32:11,251::task::595::TaskManager.Task::(_updateState) Task=`3216f667-c6af-4029-bbac-27e7d6912e44`::moving from state init -> state preparing
Thread-32::INFO::2014-11-28 15:32:11,251::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-32::INFO::2014-11-28 15:32:11,252::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-32::DEBUG::2014-11-28 15:32:11,252::task::1185::TaskManager.Task::(prepare) Task=`3216f667-c6af-4029-bbac-27e7d6912e44`::finished: {}
Thread-32::DEBUG::2014-11-28 15:32:11,252::task::595::TaskManager.Task::(_updateState) Task=`3216f667-c6af-4029-bbac-27e7d6912e44`::moving from state preparing -> state finished
Thread-32::DEBUG::2014-11-28 15:32:11,252::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-32::DEBUG::2014-11-28 15:32:11,252::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::DEBUG::2014-11-28 15:32:11,252::task::990::TaskManager.Task::(_decref) Task=`3216f667-c6af-4029-bbac-27e7d6912e44`::ref 0 aborting False
Thread-32::DEBUG::2014-11-28 15:32:11,265::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-32::DEBUG::2014-11-28 15:32:11,265::task::595::TaskManager.Task::(_updateState) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::moving from state init -> state preparing
Thread-32::INFO::2014-11-28 15:32:11,266::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-32::ERROR::2014-11-28 15:32:11,266::task::866::TaskManager.Task::(_setError) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-32::DEBUG::2014-11-28 15:32:11,266::task::885::TaskManager.Task::(_run) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::Task._run: f383d2c0-6922-49e0-b488-ac921d9d0b99 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-32::DEBUG::2014-11-28 15:32:11,266::task::1211::TaskManager.Task::(stop) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::stopping in state preparing (force False)
Thread-32::DEBUG::2014-11-28 15:32:11,266::task::990::TaskManager.Task::(_decref) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::ref 1 aborting True
Thread-32::INFO::2014-11-28 15:32:11,267::task::1168::TaskManager.Task::(prepare) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-32::DEBUG::2014-11-28 15:32:11,267::task::1173::TaskManager.Task::(prepare) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::Prepare: aborted: Unknown pool id, pool not connected
Thread-32::DEBUG::2014-11-28 15:32:11,267::task::990::TaskManager.Task::(_decref) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::ref 0 aborting True
Thread-32::DEBUG::2014-11-28 15:32:11,267::task::925::TaskManager.Task::(_doAbort) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::Task._doAbort: force False
Thread-32::DEBUG::2014-11-28 15:32:11,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::DEBUG::2014-11-28 15:32:11,268::task::595::TaskManager.Task::(_updateState) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::moving from state preparing -> state aborting
Thread-32::DEBUG::2014-11-28 15:32:11,268::task::550::TaskManager.Task::(__state_aborting) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::_aborting: recover policy none
Thread-32::DEBUG::2014-11-28 15:32:11,268::task::595::TaskManager.Task::(_updateState) Task=`f383d2c0-6922-49e0-b488-ac921d9d0b99`::moving from state aborting -> state failed
Thread-32::DEBUG::2014-11-28 15:32:11,268::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-32::DEBUG::2014-11-28 15:32:11,268::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::ERROR::2014-11-28 15:32:11,268::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-32::DEBUG::2014-11-28 15:32:11,299::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-32::DEBUG::2014-11-28 15:32:11,299::task::595::TaskManager.Task::(_updateState) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::moving from state init -> state preparing
Thread-32::INFO::2014-11-28 15:32:11,299::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='e7eaae86-11ea-4c08-8c11-8b8c54373389', masterVersion=24238, domainsMap=None, options=None)
Thread-32::DEBUG::2014-11-28 15:32:11,300::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`87c82fd8-29f4-40db-8274-30f3bcee04de`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-32::DEBUG::2014-11-28 15:32:11,300::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-32::DEBUG::2014-11-28 15:32:11,300::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-32::DEBUG::2014-11-28 15:32:11,300::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`87c82fd8-29f4-40db-8274-30f3bcee04de`::Granted request
Thread-32::DEBUG::2014-11-28 15:32:11,301::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`f8992864-a7ab-4322-8d38-052a66e9c4ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-32::DEBUG::2014-11-28 15:32:11,301::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-32::DEBUG::2014-11-28 15:32:11,301::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is currently locked, Entering queue (1 in queue)
Thread-13::DEBUG::2014-11-28 15:32:38,698::clusterlock::202::SANLock::(acquireHostId) Host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 successfully acquired (id: 1)
Thread-13::INFO::2014-11-28 15:32:38,699::clusterlock::235::SANLock::(acquire) Acquiring cluster lock for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 (id: 1)
Thread-33::DEBUG::2014-11-28 15:32:57,541::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [14f70d4a]
Thread-33::DEBUG::2014-11-28 15:32:57,542::task::595::TaskManager.Task::(_updateState) Task=`a84e7d78-7931-4019-94fd-7af9d55709b5`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:32:57,542::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-33::DEBUG::2014-11-28 15:32:57,549::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-33::ERROR::2014-11-28 15:32:57,565::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-33::ERROR::2014-11-28 15:32:57,566::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-33::DEBUG::2014-11-28 15:32:57,566::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-33::INFO::2014-11-28 15:32:57,566::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-33::DEBUG::2014-11-28 15:32:57,566::task::1185::TaskManager.Task::(prepare) Task=`a84e7d78-7931-4019-94fd-7af9d55709b5`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-33::DEBUG::2014-11-28 15:32:57,567::task::595::TaskManager.Task::(_updateState) Task=`a84e7d78-7931-4019-94fd-7af9d55709b5`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:32:57,567::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:32:57,567::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:32:57,567::task::990::TaskManager.Task::(_decref) Task=`a84e7d78-7931-4019-94fd-7af9d55709b5`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:33:09,625::task::595::TaskManager.Task::(_updateState) Task=`b583dfd0-e98e-4cf2-853e-b5e6f1b26baf`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:33:09,626::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:33:09,626::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:33:09,626::task::1185::TaskManager.Task::(prepare) Task=`b583dfd0-e98e-4cf2-853e-b5e6f1b26baf`::finished: {}
Thread-33::DEBUG::2014-11-28 15:33:09,626::task::595::TaskManager.Task::(_updateState) Task=`b583dfd0-e98e-4cf2-853e-b5e6f1b26baf`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:33:09,627::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:33:09,627::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:33:09,627::task::990::TaskManager.Task::(_decref) Task=`b583dfd0-e98e-4cf2-853e-b5e6f1b26baf`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:33:11,508::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [4d9513e3]
Thread-33::DEBUG::2014-11-28 15:33:11,508::task::595::TaskManager.Task::(_updateState) Task=`a84dbdb4-9bb6-4ecd-bf49-56601261a10e`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:33:11,509::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-33::DEBUG::2014-11-28 15:33:11,514::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-33::DEBUG::2014-11-28 15:34:11,519::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-33::DEBUG::2014-11-28 15:34:11,520::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-33::INFO::2014-11-28 15:34:11,520::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-33::DEBUG::2014-11-28 15:34:11,520::task::1185::TaskManager.Task::(prepare) Task=`a84dbdb4-9bb6-4ecd-bf49-56601261a10e`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-33::DEBUG::2014-11-28 15:34:11,520::task::595::TaskManager.Task::(_updateState) Task=`a84dbdb4-9bb6-4ecd-bf49-56601261a10e`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:34:11,521::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:34:11,521::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:34:11,521::task::990::TaskManager.Task::(_decref) Task=`a84dbdb4-9bb6-4ecd-bf49-56601261a10e`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:34:11,608::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5b906b47]
Thread-33::DEBUG::2014-11-28 15:34:11,609::task::595::TaskManager.Task::(_updateState) Task=`b3bf17d5-6692-478d-aa8c-e3ce8073ffc0`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:34:11,609::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-33::DEBUG::2014-11-28 15:34:11,615::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-34::DEBUG::2014-11-28 15:35:11,434::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5c8b9768]
Thread-34::DEBUG::2014-11-28 15:35:11,436::task::595::TaskManager.Task::(_updateState) Task=`cb0adf0e-93a5-4971-a2ed-4b736a7ef379`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:35:11,438::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:35:11,468::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-34::ERROR::2014-11-28 15:35:11,486::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-34::ERROR::2014-11-28 15:35:11,488::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-34::DEBUG::2014-11-28 15:35:11,489::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:35:11,490::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-34::DEBUG::2014-11-28 15:35:11,491::task::1185::TaskManager.Task::(prepare) Task=`cb0adf0e-93a5-4971-a2ed-4b736a7ef379`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-34::DEBUG::2014-11-28 15:35:11,492::task::595::TaskManager.Task::(_updateState) Task=`cb0adf0e-93a5-4971-a2ed-4b736a7ef379`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:35:11,494::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:35:11,495::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:35:11,496::task::990::TaskManager.Task::(_decref) Task=`cb0adf0e-93a5-4971-a2ed-4b736a7ef379`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:35:11,503::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-34::DEBUG::2014-11-28 15:35:11,505::task::595::TaskManager.Task::(_updateState) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:35:11,506::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-34::ERROR::2014-11-28 15:35:11,507::task::866::TaskManager.Task::(_setError) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-34::DEBUG::2014-11-28 15:35:11,509::task::885::TaskManager.Task::(_run) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::Task._run: c4cf1f50-4dda-49d6-a722-1f3b68d4498e ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-34::DEBUG::2014-11-28 15:35:11,510::task::1211::TaskManager.Task::(stop) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::stopping in state preparing (force False)
Thread-34::DEBUG::2014-11-28 15:35:11,512::task::990::TaskManager.Task::(_decref) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::ref 1 aborting True
Thread-34::INFO::2014-11-28 15:35:11,513::task::1168::TaskManager.Task::(prepare) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-34::DEBUG::2014-11-28 15:35:11,514::task::1173::TaskManager.Task::(prepare) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::Prepare: aborted: Unknown pool id, pool not connected
Thread-34::DEBUG::2014-11-28 15:35:11,515::task::990::TaskManager.Task::(_decref) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::ref 0 aborting True
Thread-34::DEBUG::2014-11-28 15:35:11,516::task::925::TaskManager.Task::(_doAbort) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::Task._doAbort: force False
Thread-34::DEBUG::2014-11-28 15:35:11,517::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:35:11,518::task::595::TaskManager.Task::(_updateState) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::moving from state preparing -> state aborting
Thread-34::DEBUG::2014-11-28 15:35:11,520::task::550::TaskManager.Task::(__state_aborting) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::_aborting: recover policy none
Thread-34::DEBUG::2014-11-28 15:35:11,521::task::595::TaskManager.Task::(_updateState) Task=`c4cf1f50-4dda-49d6-a722-1f3b68d4498e`::moving from state aborting -> state failed
Thread-34::DEBUG::2014-11-28 15:35:11,522::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:35:11,523::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::ERROR::2014-11-28 15:35:11,524::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-34::DEBUG::2014-11-28 15:35:11,601::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [470d359e]
Thread-34::DEBUG::2014-11-28 15:35:11,603::task::595::TaskManager.Task::(_updateState) Task=`c97ef1e5-a091-4217-b45e-29667b3e76ce`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:35:11,606::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-33::DEBUG::2014-11-28 15:35:11,622::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-33::DEBUG::2014-11-28 15:35:11,623::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-33::DEBUG::2014-11-28 15:35:11,624::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-33::INFO::2014-11-28 15:35:11,625::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-33::DEBUG::2014-11-28 15:35:11,626::task::1185::TaskManager.Task::(prepare) Task=`b3bf17d5-6692-478d-aa8c-e3ce8073ffc0`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-33::DEBUG::2014-11-28 15:35:11,626::task::595::TaskManager.Task::(_updateState) Task=`b3bf17d5-6692-478d-aa8c-e3ce8073ffc0`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:35:11,627::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:35:11,628::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:35:11,628::task::990::TaskManager.Task::(_decref) Task=`b3bf17d5-6692-478d-aa8c-e3ce8073ffc0`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:35:11,642::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-33::DEBUG::2014-11-28 15:35:20,474::task::595::TaskManager.Task::(_updateState) Task=`a7b4ce6e-7bf7-43fc-a127-c9e96e38bccf`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:35:20,475::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:35:20,475::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:35:20,475::task::1185::TaskManager.Task::(prepare) Task=`a7b4ce6e-7bf7-43fc-a127-c9e96e38bccf`::finished: {}
Thread-33::DEBUG::2014-11-28 15:35:20,475::task::595::TaskManager.Task::(_updateState) Task=`a7b4ce6e-7bf7-43fc-a127-c9e96e38bccf`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:35:20,475::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:35:20,476::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:35:20,476::task::990::TaskManager.Task::(_decref) Task=`a7b4ce6e-7bf7-43fc-a127-c9e96e38bccf`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:35:35,562::task::595::TaskManager.Task::(_updateState) Task=`bd987ba8-cc22-4e3e-bba1-37d01cfe91a5`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:35:35,563::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:35:35,563::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:35:35,563::task::1185::TaskManager.Task::(prepare) Task=`bd987ba8-cc22-4e3e-bba1-37d01cfe91a5`::finished: {}
Thread-33::DEBUG::2014-11-28 15:35:35,563::task::595::TaskManager.Task::(_updateState) Task=`bd987ba8-cc22-4e3e-bba1-37d01cfe91a5`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:35:35,563::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:35:35,563::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:35:35,564::task::990::TaskManager.Task::(_decref) Task=`bd987ba8-cc22-4e3e-bba1-37d01cfe91a5`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:35:50,649::task::595::TaskManager.Task::(_updateState) Task=`678eb090-5016-4945-a07a-baf0ebb5eb6b`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:35:50,650::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:35:50,650::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:35:50,650::task::1185::TaskManager.Task::(prepare) Task=`678eb090-5016-4945-a07a-baf0ebb5eb6b`::finished: {}
Thread-33::DEBUG::2014-11-28 15:35:50,650::task::595::TaskManager.Task::(_updateState) Task=`678eb090-5016-4945-a07a-baf0ebb5eb6b`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:35:50,650::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:35:50,651::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:35:50,651::task::990::TaskManager.Task::(_decref) Task=`678eb090-5016-4945-a07a-baf0ebb5eb6b`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:36:05,743::task::595::TaskManager.Task::(_updateState) Task=`32f20d14-2ecf-42c9-9274-663fdf026880`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:36:05,743::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:36:05,744::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:36:05,744::task::1185::TaskManager.Task::(prepare) Task=`32f20d14-2ecf-42c9-9274-663fdf026880`::finished: {}
Thread-33::DEBUG::2014-11-28 15:36:05,744::task::595::TaskManager.Task::(_updateState) Task=`32f20d14-2ecf-42c9-9274-663fdf026880`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:36:05,744::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:36:05,744::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:36:05,745::task::990::TaskManager.Task::(_decref) Task=`32f20d14-2ecf-42c9-9274-663fdf026880`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:36:20,864::task::595::TaskManager.Task::(_updateState) Task=`38627f15-a62c-43fd-a26c-55a7299fbf6d`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:36:20,865::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:36:20,865::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:36:20,865::task::1185::TaskManager.Task::(prepare) Task=`38627f15-a62c-43fd-a26c-55a7299fbf6d`::finished: {}
Thread-33::DEBUG::2014-11-28 15:36:20,865::task::595::TaskManager.Task::(_updateState) Task=`38627f15-a62c-43fd-a26c-55a7299fbf6d`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:36:20,865::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:36:20,865::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:36:20,866::task::990::TaskManager.Task::(_decref) Task=`38627f15-a62c-43fd-a26c-55a7299fbf6d`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:36:35,985::task::595::TaskManager.Task::(_updateState) Task=`4bf3cc61-7afa-430a-98e3-80d45e92360c`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:36:35,986::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:36:35,986::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:36:35,986::task::1185::TaskManager.Task::(prepare) Task=`4bf3cc61-7afa-430a-98e3-80d45e92360c`::finished: {}
Thread-33::DEBUG::2014-11-28 15:36:35,986::task::595::TaskManager.Task::(_updateState) Task=`4bf3cc61-7afa-430a-98e3-80d45e92360c`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:36:35,986::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:36:35,987::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:36:35,987::task::990::TaskManager.Task::(_decref) Task=`4bf3cc61-7afa-430a-98e3-80d45e92360c`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:36:51,079::task::595::TaskManager.Task::(_updateState) Task=`f576c071-2868-43e3-85ff-f0c75e9a999e`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:36:51,080::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:36:51,080::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:36:51,080::task::1185::TaskManager.Task::(prepare) Task=`f576c071-2868-43e3-85ff-f0c75e9a999e`::finished: {}
Thread-33::DEBUG::2014-11-28 15:36:51,080::task::595::TaskManager.Task::(_updateState) Task=`f576c071-2868-43e3-85ff-f0c75e9a999e`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:36:51,081::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:36:51,081::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:36:51,081::task::990::TaskManager.Task::(_decref) Task=`f576c071-2868-43e3-85ff-f0c75e9a999e`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:37:06,173::task::595::TaskManager.Task::(_updateState) Task=`348a46d6-1a43-4cbb-9dc0-469c09f6453d`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:37:06,174::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:37:06,174::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:37:06,174::task::1185::TaskManager.Task::(prepare) Task=`348a46d6-1a43-4cbb-9dc0-469c09f6453d`::finished: {}
Thread-33::DEBUG::2014-11-28 15:37:06,174::task::595::TaskManager.Task::(_updateState) Task=`348a46d6-1a43-4cbb-9dc0-469c09f6453d`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:37:06,174::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:37:06,175::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:37:06,175::task::990::TaskManager.Task::(_decref) Task=`348a46d6-1a43-4cbb-9dc0-469c09f6453d`::ref 0 aborting False
Thread-33::DEBUG::2014-11-28 15:37:21,276::task::595::TaskManager.Task::(_updateState) Task=`177027d9-e7be-4fc2-b83e-0cf2573a315d`::moving from state init -> state preparing
Thread-33::INFO::2014-11-28 15:37:21,277::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-33::INFO::2014-11-28 15:37:21,277::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-33::DEBUG::2014-11-28 15:37:21,277::task::1185::TaskManager.Task::(prepare) Task=`177027d9-e7be-4fc2-b83e-0cf2573a315d`::finished: {}
Thread-33::DEBUG::2014-11-28 15:37:21,277::task::595::TaskManager.Task::(_updateState) Task=`177027d9-e7be-4fc2-b83e-0cf2573a315d`::moving from state preparing -> state finished
Thread-33::DEBUG::2014-11-28 15:37:21,277::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-33::DEBUG::2014-11-28 15:37:21,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-33::DEBUG::2014-11-28 15:37:21,278::task::990::TaskManager.Task::(_decref) Task=`177027d9-e7be-4fc2-b83e-0cf2573a315d`::ref 0 aborting False
Thread-34::ERROR::2014-11-28 15:37:22,754::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-34::ERROR::2014-11-28 15:37:22,755::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-34::DEBUG::2014-11-28 15:37:22,755::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:37:22,755::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-34::DEBUG::2014-11-28 15:37:22,755::task::1185::TaskManager.Task::(prepare) Task=`c97ef1e5-a091-4217-b45e-29667b3e76ce`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-34::DEBUG::2014-11-28 15:37:22,755::task::595::TaskManager.Task::(_updateState) Task=`c97ef1e5-a091-4217-b45e-29667b3e76ce`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:37:22,756::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:37:22,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:37:22,756::task::990::TaskManager.Task::(_decref) Task=`c97ef1e5-a091-4217-b45e-29667b3e76ce`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:37:22,899::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [461736a6]
Thread-34::DEBUG::2014-11-28 15:37:22,900::task::595::TaskManager.Task::(_updateState) Task=`192b5ac3-1179-4362-8cbe-0f3e79368dd2`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:37:22,900::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:37:22,906::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-34::ERROR::2014-11-28 15:37:22,922::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-34::ERROR::2014-11-28 15:37:22,922::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-34::DEBUG::2014-11-28 15:37:22,923::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:37:22,923::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-34::DEBUG::2014-11-28 15:37:22,923::task::1185::TaskManager.Task::(prepare) Task=`192b5ac3-1179-4362-8cbe-0f3e79368dd2`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-34::DEBUG::2014-11-28 15:37:22,923::task::595::TaskManager.Task::(_updateState) Task=`192b5ac3-1179-4362-8cbe-0f3e79368dd2`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:37:22,923::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:37:22,924::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:37:22,924::task::990::TaskManager.Task::(_decref) Task=`192b5ac3-1179-4362-8cbe-0f3e79368dd2`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:37:36,371::task::595::TaskManager.Task::(_updateState) Task=`c9fc1dbd-76e2-4fe4-8e0e-d4d5c78130e1`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:37:36,371::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:37:36,372::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:37:36,372::task::1185::TaskManager.Task::(prepare) Task=`c9fc1dbd-76e2-4fe4-8e0e-d4d5c78130e1`::finished: {}
Thread-34::DEBUG::2014-11-28 15:37:36,372::task::595::TaskManager.Task::(_updateState) Task=`c9fc1dbd-76e2-4fe4-8e0e-d4d5c78130e1`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:37:36,372::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:37:36,372::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:37:36,372::task::990::TaskManager.Task::(_decref) Task=`c9fc1dbd-76e2-4fe4-8e0e-d4d5c78130e1`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:37:51,486::task::595::TaskManager.Task::(_updateState) Task=`9ec23db7-aadd-4995-8663-753380a6910b`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:37:51,486::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:37:51,486::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:37:51,486::task::1185::TaskManager.Task::(prepare) Task=`9ec23db7-aadd-4995-8663-753380a6910b`::finished: {}
Thread-34::DEBUG::2014-11-28 15:37:51,486::task::595::TaskManager.Task::(_updateState) Task=`9ec23db7-aadd-4995-8663-753380a6910b`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:37:51,487::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:37:51,487::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:37:51,487::task::990::TaskManager.Task::(_decref) Task=`9ec23db7-aadd-4995-8663-753380a6910b`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:38:06,570::task::595::TaskManager.Task::(_updateState) Task=`cc3314ca-09e5-4f74-8a6d-04dac285f183`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:38:06,570::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:38:06,570::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:38:06,570::task::1185::TaskManager.Task::(prepare) Task=`cc3314ca-09e5-4f74-8a6d-04dac285f183`::finished: {}
Thread-34::DEBUG::2014-11-28 15:38:06,571::task::595::TaskManager.Task::(_updateState) Task=`cc3314ca-09e5-4f74-8a6d-04dac285f183`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:38:06,571::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:38:06,571::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:38:06,571::task::990::TaskManager.Task::(_decref) Task=`cc3314ca-09e5-4f74-8a6d-04dac285f183`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:38:21,652::task::595::TaskManager.Task::(_updateState) Task=`f09034d5-50df-4e91-b773-fd5351fc9507`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:38:21,652::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:38:21,652::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:38:21,652::task::1185::TaskManager.Task::(prepare) Task=`f09034d5-50df-4e91-b773-fd5351fc9507`::finished: {}
Thread-34::DEBUG::2014-11-28 15:38:21,653::task::595::TaskManager.Task::(_updateState) Task=`f09034d5-50df-4e91-b773-fd5351fc9507`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:38:21,653::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:38:21,653::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:38:21,653::task::990::TaskManager.Task::(_decref) Task=`f09034d5-50df-4e91-b773-fd5351fc9507`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:38:23,016::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3df609df]
Thread-34::DEBUG::2014-11-28 15:38:23,016::task::595::TaskManager.Task::(_updateState) Task=`77646c4f-4e0a-48c8-aa1f-d6c8b21fd8e8`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:38:23,017::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:38:23,023::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-35::DEBUG::2014-11-28 15:38:36,783::task::595::TaskManager.Task::(_updateState) Task=`86f6e1ba-d32a-4ff3-86f7-5de4210cac35`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:38:36,783::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:38:36,783::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:38:36,783::task::1185::TaskManager.Task::(prepare) Task=`86f6e1ba-d32a-4ff3-86f7-5de4210cac35`::finished: {}
Thread-35::DEBUG::2014-11-28 15:38:36,784::task::595::TaskManager.Task::(_updateState) Task=`86f6e1ba-d32a-4ff3-86f7-5de4210cac35`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:38:36,784::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:38:36,784::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:38:36,784::task::990::TaskManager.Task::(_decref) Task=`86f6e1ba-d32a-4ff3-86f7-5de4210cac35`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:38:51,873::task::595::TaskManager.Task::(_updateState) Task=`db1de5f2-4be2-495d-ad2a-72248c4b4f05`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:38:51,873::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:38:51,874::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:38:51,874::task::1185::TaskManager.Task::(prepare) Task=`db1de5f2-4be2-495d-ad2a-72248c4b4f05`::finished: {}
Thread-35::DEBUG::2014-11-28 15:38:51,874::task::595::TaskManager.Task::(_updateState) Task=`db1de5f2-4be2-495d-ad2a-72248c4b4f05`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:38:51,874::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:38:51,874::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:38:51,874::task::990::TaskManager.Task::(_decref) Task=`db1de5f2-4be2-495d-ad2a-72248c4b4f05`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:39:06,966::task::595::TaskManager.Task::(_updateState) Task=`205da601-2c77-4b83-847f-268092688f5f`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:39:06,967::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:39:06,967::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:39:06,967::task::1185::TaskManager.Task::(prepare) Task=`205da601-2c77-4b83-847f-268092688f5f`::finished: {}
Thread-35::DEBUG::2014-11-28 15:39:06,967::task::595::TaskManager.Task::(_updateState) Task=`205da601-2c77-4b83-847f-268092688f5f`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:39:06,967::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:39:06,967::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:39:06,968::task::990::TaskManager.Task::(_decref) Task=`205da601-2c77-4b83-847f-268092688f5f`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:39:22,100::task::595::TaskManager.Task::(_updateState) Task=`a66ca0f8-6137-4369-8086-b7f120a24c2c`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:39:22,101::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:39:22,102::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:39:22,104::task::1185::TaskManager.Task::(prepare) Task=`a66ca0f8-6137-4369-8086-b7f120a24c2c`::finished: {}
Thread-35::DEBUG::2014-11-28 15:39:22,105::task::595::TaskManager.Task::(_updateState) Task=`a66ca0f8-6137-4369-8086-b7f120a24c2c`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:39:22,106::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:39:22,107::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:39:22,108::task::990::TaskManager.Task::(_decref) Task=`a66ca0f8-6137-4369-8086-b7f120a24c2c`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:39:23,027::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-34::DEBUG::2014-11-28 15:39:23,028::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:39:23,028::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-34::DEBUG::2014-11-28 15:39:23,028::task::1185::TaskManager.Task::(prepare) Task=`77646c4f-4e0a-48c8-aa1f-d6c8b21fd8e8`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-34::DEBUG::2014-11-28 15:39:23,028::task::595::TaskManager.Task::(_updateState) Task=`77646c4f-4e0a-48c8-aa1f-d6c8b21fd8e8`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:39:23,028::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:39:23,029::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:39:23,029::task::990::TaskManager.Task::(_decref) Task=`77646c4f-4e0a-48c8-aa1f-d6c8b21fd8e8`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:39:23,139::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [453314d1]
Thread-34::DEBUG::2014-11-28 15:39:23,139::task::595::TaskManager.Task::(_updateState) Task=`f2b4adcb-b295-41c7-bdd2-fc50a072ef95`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:39:23,139::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:39:23,145::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-35::DEBUG::2014-11-28 15:39:37,215::task::595::TaskManager.Task::(_updateState) Task=`63b3ca36-8efe-4e3f-a061-594882549126`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:39:37,215::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:39:37,216::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:39:37,216::task::1185::TaskManager.Task::(prepare) Task=`63b3ca36-8efe-4e3f-a061-594882549126`::finished: {}
Thread-35::DEBUG::2014-11-28 15:39:37,216::task::595::TaskManager.Task::(_updateState) Task=`63b3ca36-8efe-4e3f-a061-594882549126`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:39:37,216::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:39:37,216::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:39:37,216::task::990::TaskManager.Task::(_decref) Task=`63b3ca36-8efe-4e3f-a061-594882549126`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:39:52,304::task::595::TaskManager.Task::(_updateState) Task=`bbc8972e-46df-426e-95b3-9c045c631c8b`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:39:52,305::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:39:52,305::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:39:52,305::task::1185::TaskManager.Task::(prepare) Task=`bbc8972e-46df-426e-95b3-9c045c631c8b`::finished: {}
Thread-35::DEBUG::2014-11-28 15:39:52,305::task::595::TaskManager.Task::(_updateState) Task=`bbc8972e-46df-426e-95b3-9c045c631c8b`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:39:52,305::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:39:52,306::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:39:52,306::task::990::TaskManager.Task::(_decref) Task=`bbc8972e-46df-426e-95b3-9c045c631c8b`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:39:59,918::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [139a26d9]
Thread-35::DEBUG::2014-11-28 15:39:59,918::task::595::TaskManager.Task::(_updateState) Task=`145758e4-6fd7-4a33-999c-869ddf57660f`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:39:59,919::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:39:59,925::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-35::ERROR::2014-11-28 15:39:59,958::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::ERROR::2014-11-28 15:39:59,959::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::DEBUG::2014-11-28 15:39:59,959::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:39:59,960::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:39:59,960::task::1185::TaskManager.Task::(prepare) Task=`145758e4-6fd7-4a33-999c-869ddf57660f`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:39:59,960::task::595::TaskManager.Task::(_updateState) Task=`145758e4-6fd7-4a33-999c-869ddf57660f`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:39:59,960::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:39:59,961::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:39:59,961::task::990::TaskManager.Task::(_decref) Task=`145758e4-6fd7-4a33-999c-869ddf57660f`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:40:00,097::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [9b79908]
Thread-35::DEBUG::2014-11-28 15:40:00,098::task::595::TaskManager.Task::(_updateState) Task=`993fa654-29ff-497d-a362-304c101c23d2`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:40:00,098::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:40:00,111::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-34::DEBUG::2014-11-28 15:40:23,150::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-34::DEBUG::2014-11-28 15:40:23,151::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:40:23,151::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-34::DEBUG::2014-11-28 15:40:23,151::task::1185::TaskManager.Task::(prepare) Task=`f2b4adcb-b295-41c7-bdd2-fc50a072ef95`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-34::DEBUG::2014-11-28 15:40:23,151::task::595::TaskManager.Task::(_updateState) Task=`f2b4adcb-b295-41c7-bdd2-fc50a072ef95`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:40:23,151::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:40:23,152::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:40:23,152::task::990::TaskManager.Task::(_decref) Task=`f2b4adcb-b295-41c7-bdd2-fc50a072ef95`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:40:29,182::task::595::TaskManager.Task::(_updateState) Task=`e30aa6e2-ab6e-4161-9d8b-6d8a78a05920`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:40:29,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:40:29,183::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:40:29,183::task::1185::TaskManager.Task::(prepare) Task=`e30aa6e2-ab6e-4161-9d8b-6d8a78a05920`::finished: {}
Thread-34::DEBUG::2014-11-28 15:40:29,183::task::595::TaskManager.Task::(_updateState) Task=`e30aa6e2-ab6e-4161-9d8b-6d8a78a05920`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:40:29,183::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:40:29,184::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:40:29,184::task::990::TaskManager.Task::(_decref) Task=`e30aa6e2-ab6e-4161-9d8b-6d8a78a05920`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:40:44,274::task::595::TaskManager.Task::(_updateState) Task=`9def5d8b-a2f9-4e93-b270-f1c856960fa8`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:40:44,275::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:40:44,275::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:40:44,275::task::1185::TaskManager.Task::(prepare) Task=`9def5d8b-a2f9-4e93-b270-f1c856960fa8`::finished: {}
Thread-34::DEBUG::2014-11-28 15:40:44,275::task::595::TaskManager.Task::(_updateState) Task=`9def5d8b-a2f9-4e93-b270-f1c856960fa8`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:40:44,275::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:40:44,276::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:40:44,276::task::990::TaskManager.Task::(_decref) Task=`9def5d8b-a2f9-4e93-b270-f1c856960fa8`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:40:59,375::task::595::TaskManager.Task::(_updateState) Task=`06b43d7d-db0d-4b26-8fe2-a866f4b9d9b8`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:40:59,376::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:40:59,376::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:40:59,376::task::1185::TaskManager.Task::(prepare) Task=`06b43d7d-db0d-4b26-8fe2-a866f4b9d9b8`::finished: {}
Thread-34::DEBUG::2014-11-28 15:40:59,376::task::595::TaskManager.Task::(_updateState) Task=`06b43d7d-db0d-4b26-8fe2-a866f4b9d9b8`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:40:59,376::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:40:59,377::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:40:59,377::task::990::TaskManager.Task::(_decref) Task=`06b43d7d-db0d-4b26-8fe2-a866f4b9d9b8`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:41:14,463::task::595::TaskManager.Task::(_updateState) Task=`6e47b9c6-94f8-4143-97f1-817b5484f18a`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:41:14,464::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:41:14,464::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:41:14,464::task::1185::TaskManager.Task::(prepare) Task=`6e47b9c6-94f8-4143-97f1-817b5484f18a`::finished: {}
Thread-34::DEBUG::2014-11-28 15:41:14,464::task::595::TaskManager.Task::(_updateState) Task=`6e47b9c6-94f8-4143-97f1-817b5484f18a`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:41:14,464::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:41:14,464::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:41:14,465::task::990::TaskManager.Task::(_decref) Task=`6e47b9c6-94f8-4143-97f1-817b5484f18a`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:41:29,546::task::595::TaskManager.Task::(_updateState) Task=`29664dbc-d6d7-4165-ab59-396110ae5487`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:41:29,546::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:41:29,546::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:41:29,546::task::1185::TaskManager.Task::(prepare) Task=`29664dbc-d6d7-4165-ab59-396110ae5487`::finished: {}
Thread-34::DEBUG::2014-11-28 15:41:29,547::task::595::TaskManager.Task::(_updateState) Task=`29664dbc-d6d7-4165-ab59-396110ae5487`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:41:29,547::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:41:29,547::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:41:29,547::task::990::TaskManager.Task::(_decref) Task=`29664dbc-d6d7-4165-ab59-396110ae5487`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:41:44,631::task::595::TaskManager.Task::(_updateState) Task=`8a2d3663-6d5f-4414-aa92-9b2cc9706798`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:41:44,631::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:41:44,631::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:41:44,631::task::1185::TaskManager.Task::(prepare) Task=`8a2d3663-6d5f-4414-aa92-9b2cc9706798`::finished: {}
Thread-34::DEBUG::2014-11-28 15:41:44,632::task::595::TaskManager.Task::(_updateState) Task=`8a2d3663-6d5f-4414-aa92-9b2cc9706798`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:41:44,632::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:41:44,632::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:41:44,632::task::990::TaskManager.Task::(_decref) Task=`8a2d3663-6d5f-4414-aa92-9b2cc9706798`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:41:59,713::task::595::TaskManager.Task::(_updateState) Task=`66d756c9-2345-4aab-8a6b-aafcd9d89115`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:41:59,714::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:41:59,714::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:41:59,714::task::1185::TaskManager.Task::(prepare) Task=`66d756c9-2345-4aab-8a6b-aafcd9d89115`::finished: {}
Thread-34::DEBUG::2014-11-28 15:41:59,714::task::595::TaskManager.Task::(_updateState) Task=`66d756c9-2345-4aab-8a6b-aafcd9d89115`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:41:59,714::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:41:59,714::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:41:59,715::task::990::TaskManager.Task::(_decref) Task=`66d756c9-2345-4aab-8a6b-aafcd9d89115`::ref 0 aborting False
Thread-35::ERROR::2014-11-28 15:42:11,224::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::ERROR::2014-11-28 15:42:11,225::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::DEBUG::2014-11-28 15:42:11,225::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:42:11,225::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:42:11,226::task::1185::TaskManager.Task::(prepare) Task=`993fa654-29ff-497d-a362-304c101c23d2`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:42:11,226::task::595::TaskManager.Task::(_updateState) Task=`993fa654-29ff-497d-a362-304c101c23d2`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:42:11,226::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:42:11,226::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:42:11,226::task::990::TaskManager.Task::(_decref) Task=`993fa654-29ff-497d-a362-304c101c23d2`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:42:11,382::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [c2e4e49]
Thread-35::DEBUG::2014-11-28 15:42:11,382::task::595::TaskManager.Task::(_updateState) Task=`c00b0811-5d5f-400d-a7e9-1a19eb58ba44`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:42:11,383::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:42:11,389::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-35::ERROR::2014-11-28 15:42:11,407::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::ERROR::2014-11-28 15:42:11,408::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::DEBUG::2014-11-28 15:42:11,408::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:42:11,408::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:42:11,409::task::1185::TaskManager.Task::(prepare) Task=`c00b0811-5d5f-400d-a7e9-1a19eb58ba44`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:42:11,409::task::595::TaskManager.Task::(_updateState) Task=`c00b0811-5d5f-400d-a7e9-1a19eb58ba44`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:42:11,409::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:42:11,409::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:42:11,409::task::990::TaskManager.Task::(_decref) Task=`c00b0811-5d5f-400d-a7e9-1a19eb58ba44`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:42:14,798::task::595::TaskManager.Task::(_updateState) Task=`b10cd70d-fcbe-404c-b30c-addbe3ad504a`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:42:14,798::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:42:14,799::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:42:14,799::task::1185::TaskManager.Task::(prepare) Task=`b10cd70d-fcbe-404c-b30c-addbe3ad504a`::finished: {}
Thread-35::DEBUG::2014-11-28 15:42:14,799::task::595::TaskManager.Task::(_updateState) Task=`b10cd70d-fcbe-404c-b30c-addbe3ad504a`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:42:14,799::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:42:14,799::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:42:14,800::task::990::TaskManager.Task::(_decref) Task=`b10cd70d-fcbe-404c-b30c-addbe3ad504a`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:42:24,307::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:42:24,308::task::595::TaskManager.Task::(_updateState) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:42:24,308::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-35::ERROR::2014-11-28 15:42:24,309::task::866::TaskManager.Task::(_setError) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-35::DEBUG::2014-11-28 15:42:24,309::task::885::TaskManager.Task::(_run) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::Task._run: 7cdfefd1-f30e-49c2-9f07-2db1cf0889df ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:42:24,309::task::1211::TaskManager.Task::(stop) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:42:24,309::task::990::TaskManager.Task::(_decref) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::ref 1 aborting True
Thread-35::INFO::2014-11-28 15:42:24,310::task::1168::TaskManager.Task::(prepare) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-35::DEBUG::2014-11-28 15:42:24,310::task::1173::TaskManager.Task::(prepare) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::Prepare: aborted: Unknown pool id, pool not connected
Thread-35::DEBUG::2014-11-28 15:42:24,310::task::990::TaskManager.Task::(_decref) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:42:24,310::task::925::TaskManager.Task::(_doAbort) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:42:24,310::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:42:24,310::task::595::TaskManager.Task::(_updateState) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:42:24,311::task::550::TaskManager.Task::(__state_aborting) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:42:24,311::task::595::TaskManager.Task::(_updateState) Task=`7cdfefd1-f30e-49c2-9f07-2db1cf0889df`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:42:24,311::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:42:24,311::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:42:24,311::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-35::DEBUG::2014-11-28 15:42:24,327::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:42:24,327::task::595::TaskManager.Task::(_updateState) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:42:24,327::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24238, domainsMap=None, options=None)
Thread-35::DEBUG::2014-11-28 15:42:24,328::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`48e0901b-0a21-40a0-9d59-c123bae18f3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-35::DEBUG::2014-11-28 15:42:24,328::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-35::DEBUG::2014-11-28 15:42:24,328::resourceManager::577::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is currently locked, Entering queue (1 in queue)
Thread-34::DEBUG::2014-11-28 15:42:29,902::task::595::TaskManager.Task::(_updateState) Task=`cc8838b3-a76b-4d67-b60a-a9b88853ad78`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:42:29,903::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:42:29,903::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:42:29,903::task::1185::TaskManager.Task::(prepare) Task=`cc8838b3-a76b-4d67-b60a-a9b88853ad78`::finished: {}
Thread-34::DEBUG::2014-11-28 15:42:29,903::task::595::TaskManager.Task::(_updateState) Task=`cc8838b3-a76b-4d67-b60a-a9b88853ad78`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:42:29,903::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:42:29,904::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:42:29,904::task::990::TaskManager.Task::(_decref) Task=`cc8838b3-a76b-4d67-b60a-a9b88853ad78`::ref 0 aborting False
Thread-13::ERROR::2014-11-28 15:42:38,936::task::866::TaskManager.Task::(_setError) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1887, in reconstructMaster
    masterVersion, leaseParams)
  File "/usr/share/vdsm/storage/sp.py", line 700, in reconstructMaster
    futureMaster.acquireClusterLock(hostId)
  File "/usr/share/vdsm/storage/sd.py", line 496, in acquireClusterLock
    self._clusterLock.acquire(hostID)
  File "/usr/share/vdsm/storage/clusterlock.py", line 254, in acquire
    "Cannot acquire cluster lock", str(e))
AcquireLockFailure: Cannot obtain lock: "id=e7eaae86-11ea-4c08-8c11-8b8c54373389, rc=5, out=Cannot acquire cluster lock, err=(5, 'Sanlock resource not acquired', 'Input/output error')"
Thread-13::DEBUG::2014-11-28 15:42:38,937::task::885::TaskManager.Task::(_run) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::Task._run: f1e5964f-050f-44ae-abb3-a04c6db3559a ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 'CTI', 'e7eaae86-11ea-4c08-8c11-8b8c54373389', {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': 'active', '474f0809-26c6-4fca-ab8c-72a2e6dcd912': 'active', '54b2d645-fc18-4784-99eb-0e17e296eee7': 'active', '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': 'active', 'e7eaae86-11ea-4c08-8c11-8b8c54373389': 'active'}, 24238, None, 5, 60, 10, 3, 1) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 15:42:38,937::task::1211::TaskManager.Task::(stop) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 15:42:38,938::task::990::TaskManager.Task::(_decref) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::ref 1 aborting True
Thread-13::INFO::2014-11-28 15:42:38,938::task::1168::TaskManager.Task::(prepare) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::aborting: Task is aborted: 'Cannot obtain lock' - code 651
Thread-13::DEBUG::2014-11-28 15:42:38,938::task::1173::TaskManager.Task::(prepare) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::Prepare: aborted: Cannot obtain lock
Thread-13::DEBUG::2014-11-28 15:42:38,938::task::990::TaskManager.Task::(_decref) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 15:42:38,938::task::925::TaskManager.Task::(_doAbort) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 15:42:38,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:42:38,939::task::595::TaskManager.Task::(_updateState) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 15:42:38,939::task::550::TaskManager.Task::(__state_aborting) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 15:42:38,939::task::595::TaskManager.Task::(_updateState) Task=`f1e5964f-050f-44ae-abb3-a04c6db3559a`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 15:42:38,939::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad': < ResourceRef 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', isValid: 'True' obj: 'None'>}
Thread-13::DEBUG::2014-11-28 15:42:38,940::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 15:42:38,940::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 15:42:38,940::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 15:42:38,940::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 15:42:38,941::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' has 1 requests in queue. Handling top request.
Thread-13::DEBUG::2014-11-28 15:42:38,941::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`f8992864-a7ab-4322-8d38-052a66e9c4ea`::Granted request
Thread-13::DEBUG::2014-11-28 15:42:38,941::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad - exclusive: granted' was granted
Thread-13::ERROR::2014-11-28 15:42:38,942::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Cannot obtain lock: "id=e7eaae86-11ea-4c08-8c11-8b8c54373389, rc=5, out=Cannot acquire cluster lock, err=(5, \'Sanlock resource not acquired\', \'Input/output error\')"', 'code': 651}}
Thread-32::INFO::2014-11-28 15:42:38,942::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c0c1050
Thread-32::INFO::2014-11-28 15:42:38,943::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: e7eaae86-11ea-4c08-8c11-8b8c54373389 (ver = 24238)
Thread-32::DEBUG::2014-11-28 15:42:38,944::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,944::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,944::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,945::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,945::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,945::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:38,945::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-32::DEBUG::2014-11-28 15:42:38,946::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-32::DEBUG::2014-11-28 15:42:38,946::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-32::DEBUG::2014-11-28 15:42:38,946::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-32::DEBUG::2014-11-28 15:42:38,946::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-32::DEBUG::2014-11-28 15:42:38,946::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-32::DEBUG::2014-11-28 15:42:38,958::misc::748::SamplingMethod::(__call__) Returning last result
Thread-32::DEBUG::2014-11-28 15:42:38,960::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-32::DEBUG::2014-11-28 15:42:39,010::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-32::DEBUG::2014-11-28 15:42:39,011::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,011::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,012::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,012::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,012::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,012::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:42:39,012::misc::748::SamplingMethod::(__call__) Returning last result
Thread-34::DEBUG::2014-11-28 15:42:44,990::task::595::TaskManager.Task::(_updateState) Task=`1a0eaf9f-a92b-48f8-9dcd-c49dccee9f10`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:42:44,991::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:42:44,991::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:42:44,991::task::1185::TaskManager.Task::(prepare) Task=`1a0eaf9f-a92b-48f8-9dcd-c49dccee9f10`::finished: {}
Thread-34::DEBUG::2014-11-28 15:42:44,991::task::595::TaskManager.Task::(_updateState) Task=`1a0eaf9f-a92b-48f8-9dcd-c49dccee9f10`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:42:44,991::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:42:44,991::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:42:44,992::task::990::TaskManager.Task::(_decref) Task=`1a0eaf9f-a92b-48f8-9dcd-c49dccee9f10`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:43:00,076::task::595::TaskManager.Task::(_updateState) Task=`7264955f-6604-4c07-b7ce-af2991986017`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:43:00,076::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:43:00,076::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:43:00,076::task::1185::TaskManager.Task::(prepare) Task=`7264955f-6604-4c07-b7ce-af2991986017`::finished: {}
Thread-34::DEBUG::2014-11-28 15:43:00,077::task::595::TaskManager.Task::(_updateState) Task=`7264955f-6604-4c07-b7ce-af2991986017`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:43:00,077::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:43:00,077::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:43:00,077::task::990::TaskManager.Task::(_decref) Task=`7264955f-6604-4c07-b7ce-af2991986017`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:43:11,529::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [17957049]
Thread-34::DEBUG::2014-11-28 15:43:11,530::task::595::TaskManager.Task::(_updateState) Task=`5cad7c86-5311-4f1a-8b35-2b0a93edceff`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:43:11,530::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:43:11,538::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-34::DEBUG::2014-11-28 15:43:11,687::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-34::DEBUG::2014-11-28 15:43:11,688::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:43:11,688::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-34::DEBUG::2014-11-28 15:43:11,688::task::1185::TaskManager.Task::(prepare) Task=`5cad7c86-5311-4f1a-8b35-2b0a93edceff`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-34::DEBUG::2014-11-28 15:43:11,688::task::595::TaskManager.Task::(_updateState) Task=`5cad7c86-5311-4f1a-8b35-2b0a93edceff`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:43:11,688::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:43:11,689::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:43:11,689::task::990::TaskManager.Task::(_decref) Task=`5cad7c86-5311-4f1a-8b35-2b0a93edceff`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:43:15,164::task::595::TaskManager.Task::(_updateState) Task=`f2427638-78a1-4e2e-88a1-4672a9545d3d`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:43:15,164::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:43:15,164::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:43:15,164::task::1185::TaskManager.Task::(prepare) Task=`f2427638-78a1-4e2e-88a1-4672a9545d3d`::finished: {}
Thread-34::DEBUG::2014-11-28 15:43:15,165::task::595::TaskManager.Task::(_updateState) Task=`f2427638-78a1-4e2e-88a1-4672a9545d3d`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:43:15,165::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:43:15,165::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:43:15,165::task::990::TaskManager.Task::(_decref) Task=`f2427638-78a1-4e2e-88a1-4672a9545d3d`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:43:30,265::task::595::TaskManager.Task::(_updateState) Task=`f71d97e4-7eec-452d-b6ff-1c99e54f7f55`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:43:30,266::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:43:30,266::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:43:30,266::task::1185::TaskManager.Task::(prepare) Task=`f71d97e4-7eec-452d-b6ff-1c99e54f7f55`::finished: {}
Thread-34::DEBUG::2014-11-28 15:43:30,266::task::595::TaskManager.Task::(_updateState) Task=`f71d97e4-7eec-452d-b6ff-1c99e54f7f55`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:43:30,266::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:43:30,267::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:43:30,267::task::990::TaskManager.Task::(_decref) Task=`f71d97e4-7eec-452d-b6ff-1c99e54f7f55`::ref 0 aborting False
Thread-36::WARNING::2014-11-28 15:43:39,015::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.225.2:_data_bs02store timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-32::ERROR::2014-11-28 15:43:39,017::sdc::143::Storage.StorageDomainCache::(_findDomain) domain e7eaae86-11ea-4c08-8c11-8b8c54373389 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-32::DEBUG::2014-11-28 15:43:39,017::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-32::DEBUG::2014-11-28 15:43:39,018::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-32::DEBUG::2014-11-28 15:43:39,018::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-32::DEBUG::2014-11-28 15:43:39,018::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-32::DEBUG::2014-11-28 15:43:39,018::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-32::DEBUG::2014-11-28 15:43:39,018::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-32::DEBUG::2014-11-28 15:43:39,019::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-32::DEBUG::2014-11-28 15:43:39,019::resourceManager::654::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' has 1 requests in queue. Handling top request.
Thread-32::DEBUG::2014-11-28 15:43:39,019::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`48e0901b-0a21-40a0-9d59-c123bae18f3a`::Granted request
Thread-32::DEBUG::2014-11-28 15:43:39,019::resourceManager::686::ResourceManager::(releaseResource) Request 'Request for Storage.HsmDomainMonitorLock - exclusive: granted' was granted
Thread-35::DEBUG::2014-11-28 15:43:39,020::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`2d88d1be-af18-41cb-932d-fd7ec1f9e9fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-32::ERROR::2014-11-28 15:43:39,020::task::866::TaskManager.Task::(_setError) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389'
Thread-35::DEBUG::2014-11-28 15:43:39,021::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-32::DEBUG::2014-11-28 15:43:39,021::task::885::TaskManager.Task::(_run) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::Task._run: 836df563-6251-4213-ac78-5e3a99ad9eb7 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, 'e7eaae86-11ea-4c08-8c11-8b8c54373389', 24238, None) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:43:39,021::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-32::DEBUG::2014-11-28 15:43:39,021::task::1211::TaskManager.Task::(stop) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:43:39,021::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`2d88d1be-af18-41cb-932d-fd7ec1f9e9fd`::Granted request
Thread-32::DEBUG::2014-11-28 15:43:39,022::task::990::TaskManager.Task::(_decref) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::ref 1 aborting True
Thread-32::INFO::2014-11-28 15:43:39,022::task::1168::TaskManager.Task::(prepare) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-32::DEBUG::2014-11-28 15:43:39,022::task::1173::TaskManager.Task::(prepare) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::Prepare: aborted: Cannot find master domain
Thread-32::DEBUG::2014-11-28 15:43:39,023::task::990::TaskManager.Task::(_decref) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::ref 0 aborting True
Thread-35::INFO::2014-11-28 15:43:39,023::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c022ab8
Thread-32::DEBUG::2014-11-28 15:43:39,023::task::925::TaskManager.Task::(_doAbort) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::Task._doAbort: force False
Thread-35::INFO::2014-11-28 15:43:39,023::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24238)
Thread-32::DEBUG::2014-11-28 15:43:39,024::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:43:39,024::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:43:39,024::task::595::TaskManager.Task::(_updateState) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:43:39,024::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:43:39,024::task::550::TaskManager.Task::(__state_aborting) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:43:39,025::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:43:39,025::task::595::TaskManager.Task::(_updateState) Task=`836df563-6251-4213-ac78-5e3a99ad9eb7`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:43:39,025::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2014-11-28 15:43:39,025::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:43:39,026::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2014-11-28 15:43:39,026::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:43:39,026::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::ERROR::2014-11-28 15:43:39,026::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389'", 'code': 304}}
Thread-35::DEBUG::2014-11-28 15:43:39,026::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-35::DEBUG::2014-11-28 15:43:39,027::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:43:39,027::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-35::DEBUG::2014-11-28 15:43:39,027::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:43:39,028::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-35::DEBUG::2014-11-28 15:43:39,028::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-35::DEBUG::2014-11-28 15:43:39,043::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::DEBUG::2014-11-28 15:43:39,045::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-35::DEBUG::2014-11-28 15:43:39,111::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-35::DEBUG::2014-11-28 15:43:39,111::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,112::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,112::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,112::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,113::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,113::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:43:39,113::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::DEBUG::2014-11-28 15:43:39,127::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-35::DEBUG::2014-11-28 15:43:39,275::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-35::DEBUG::2014-11-28 15:43:39,278::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-34::DEBUG::2014-11-28 15:43:45,350::task::595::TaskManager.Task::(_updateState) Task=`02ad3390-2adf-43a6-8e72-c05a755163eb`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:43:45,350::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:43:45,350::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:43:45,350::task::1185::TaskManager.Task::(prepare) Task=`02ad3390-2adf-43a6-8e72-c05a755163eb`::finished: {}
Thread-34::DEBUG::2014-11-28 15:43:45,351::task::595::TaskManager.Task::(_updateState) Task=`02ad3390-2adf-43a6-8e72-c05a755163eb`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:43:45,351::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:43:45,351::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:43:45,351::task::990::TaskManager.Task::(_decref) Task=`02ad3390-2adf-43a6-8e72-c05a755163eb`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:44:00,434::task::595::TaskManager.Task::(_updateState) Task=`e51ee326-e108-4238-8a00-89a80c5b2cec`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:44:00,435::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:44:00,435::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:44:00,435::task::1185::TaskManager.Task::(prepare) Task=`e51ee326-e108-4238-8a00-89a80c5b2cec`::finished: {}
Thread-34::DEBUG::2014-11-28 15:44:00,435::task::595::TaskManager.Task::(_updateState) Task=`e51ee326-e108-4238-8a00-89a80c5b2cec`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:44:00,435::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:44:00,436::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:44:00,436::task::990::TaskManager.Task::(_decref) Task=`e51ee326-e108-4238-8a00-89a80c5b2cec`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:44:11,617::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [48dfa288]
Thread-34::DEBUG::2014-11-28 15:44:11,618::task::595::TaskManager.Task::(_updateState) Task=`39cf2833-ff3d-4a7f-bb0a-a7a6eb7c99f4`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:44:11,618::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-34::DEBUG::2014-11-28 15:44:11,624::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-34::DEBUG::2014-11-28 15:44:11,625::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-34::DEBUG::2014-11-28 15:44:11,625::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-34::INFO::2014-11-28 15:44:11,625::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-34::DEBUG::2014-11-28 15:44:11,626::task::1185::TaskManager.Task::(prepare) Task=`39cf2833-ff3d-4a7f-bb0a-a7a6eb7c99f4`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-34::DEBUG::2014-11-28 15:44:11,626::task::595::TaskManager.Task::(_updateState) Task=`39cf2833-ff3d-4a7f-bb0a-a7a6eb7c99f4`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:44:11,626::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:44:11,626::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:44:11,626::task::990::TaskManager.Task::(_decref) Task=`39cf2833-ff3d-4a7f-bb0a-a7a6eb7c99f4`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:44:15,521::task::595::TaskManager.Task::(_updateState) Task=`b1011993-9b0f-4cb0-b844-5dfdef308101`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:44:15,522::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:44:15,522::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:44:15,522::task::1185::TaskManager.Task::(prepare) Task=`b1011993-9b0f-4cb0-b844-5dfdef308101`::finished: {}
Thread-34::DEBUG::2014-11-28 15:44:15,522::task::595::TaskManager.Task::(_updateState) Task=`b1011993-9b0f-4cb0-b844-5dfdef308101`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:44:15,522::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:44:15,523::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:44:15,523::task::990::TaskManager.Task::(_decref) Task=`b1011993-9b0f-4cb0-b844-5dfdef308101`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:44:30,625::task::595::TaskManager.Task::(_updateState) Task=`650b11fa-2c08-4cc5-93a7-7e87cede0886`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:44:30,625::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:44:30,626::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:44:30,626::task::1185::TaskManager.Task::(prepare) Task=`650b11fa-2c08-4cc5-93a7-7e87cede0886`::finished: {}
Thread-34::DEBUG::2014-11-28 15:44:30,626::task::595::TaskManager.Task::(_updateState) Task=`650b11fa-2c08-4cc5-93a7-7e87cede0886`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:44:30,626::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:44:30,626::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:44:30,626::task::990::TaskManager.Task::(_decref) Task=`650b11fa-2c08-4cc5-93a7-7e87cede0886`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:44:39,279::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-35::DEBUG::2014-11-28 15:44:39,280::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-35::DEBUG::2014-11-28 15:44:39,280::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-35::DEBUG::2014-11-28 15:44:39,280::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-35::DEBUG::2014-11-28 15:44:39,281::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-35::DEBUG::2014-11-28 15:44:39,281::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-35::DEBUG::2014-11-28 15:44:39,281::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-35::DEBUG::2014-11-28 15:44:39,281::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-35::ERROR::2014-11-28 15:44:39,281::task::866::TaskManager.Task::(_setError) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-35::DEBUG::2014-11-28 15:44:39,283::task::885::TaskManager.Task::(_run) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::Task._run: 86cad46f-e001-4bd2-a1e2-8dbe706a4918 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24238, None) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:44:39,283::task::1211::TaskManager.Task::(stop) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:44:39,283::task::990::TaskManager.Task::(_decref) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::ref 1 aborting True
Thread-35::DEBUG::2014-11-28 15:44:39,283::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-35::INFO::2014-11-28 15:44:39,284::task::1168::TaskManager.Task::(prepare) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::aborting: Task is aborted: u'' - code 100
Thread-35::DEBUG::2014-11-28 15:44:39,284::task::1173::TaskManager.Task::(prepare) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::Prepare: aborted: 
Thread-35::DEBUG::2014-11-28 15:44:39,284::task::990::TaskManager.Task::(_decref) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:44:39,284::task::925::TaskManager.Task::(_doAbort) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:44:39,284::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:44:39,285::task::595::TaskManager.Task::(_updateState) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:44:39,285::task::550::TaskManager.Task::(__state_aborting) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:44:39,285::task::595::TaskManager.Task::(_updateState) Task=`86cad46f-e001-4bd2-a1e2-8dbe706a4918`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:44:39,285::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:44:39,285::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:44:39,286::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-35::DEBUG::2014-11-28 15:44:39,298::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:44:39,298::task::595::TaskManager.Task::(_updateState) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:44:39,298::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-35::ERROR::2014-11-28 15:44:39,299::task::866::TaskManager.Task::(_setError) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-35::DEBUG::2014-11-28 15:44:39,299::task::885::TaskManager.Task::(_run) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::Task._run: 1f056bb7-c71a-40a7-b961-d7345ca45849 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:44:39,299::task::1211::TaskManager.Task::(stop) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:44:39,299::task::990::TaskManager.Task::(_decref) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::ref 1 aborting True
Thread-35::INFO::2014-11-28 15:44:39,299::task::1168::TaskManager.Task::(prepare) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-35::DEBUG::2014-11-28 15:44:39,300::task::1173::TaskManager.Task::(prepare) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::Prepare: aborted: Unknown pool id, pool not connected
Thread-35::DEBUG::2014-11-28 15:44:39,300::task::990::TaskManager.Task::(_decref) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:44:39,300::task::925::TaskManager.Task::(_doAbort) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:44:39,300::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:44:39,300::task::595::TaskManager.Task::(_updateState) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:44:39,301::task::550::TaskManager.Task::(__state_aborting) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:44:39,301::task::595::TaskManager.Task::(_updateState) Task=`1f056bb7-c71a-40a7-b961-d7345ca45849`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:44:39,301::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:44:39,301::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:44:39,301::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-35::DEBUG::2014-11-28 15:44:45,707::task::595::TaskManager.Task::(_updateState) Task=`c8940f62-f73f-479c-98df-0ec7875e8cef`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:44:45,707::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:44:45,707::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:44:45,707::task::1185::TaskManager.Task::(prepare) Task=`c8940f62-f73f-479c-98df-0ec7875e8cef`::finished: {}
Thread-35::DEBUG::2014-11-28 15:44:45,707::task::595::TaskManager.Task::(_updateState) Task=`c8940f62-f73f-479c-98df-0ec7875e8cef`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:44:45,708::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:44:45,708::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:44:45,708::task::990::TaskManager.Task::(_decref) Task=`c8940f62-f73f-479c-98df-0ec7875e8cef`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:44:59,934::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [6e425900]
Thread-35::DEBUG::2014-11-28 15:44:59,935::task::595::TaskManager.Task::(_updateState) Task=`171346d9-a855-4994-afc6-d5f3b152d627`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:44:59,935::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:44:59,941::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-35::ERROR::2014-11-28 15:44:59,967::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::ERROR::2014-11-28 15:44:59,968::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::DEBUG::2014-11-28 15:44:59,968::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:44:59,968::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:44:59,968::task::1185::TaskManager.Task::(prepare) Task=`171346d9-a855-4994-afc6-d5f3b152d627`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:44:59,969::task::595::TaskManager.Task::(_updateState) Task=`171346d9-a855-4994-afc6-d5f3b152d627`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:44:59,969::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:44:59,969::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:44:59,969::task::990::TaskManager.Task::(_decref) Task=`171346d9-a855-4994-afc6-d5f3b152d627`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:45:00,090::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5f9464ca]
Thread-35::DEBUG::2014-11-28 15:45:00,091::task::595::TaskManager.Task::(_updateState) Task=`d7417e89-4ef5-4415-9824-8419adbcab25`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:45:00,091::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:45:00,102::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-34::DEBUG::2014-11-28 15:45:00,789::task::595::TaskManager.Task::(_updateState) Task=`f73a50b5-22a9-4bbe-8ed4-d8eb50675db7`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:45:00,790::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:45:00,790::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:45:00,790::task::1185::TaskManager.Task::(prepare) Task=`f73a50b5-22a9-4bbe-8ed4-d8eb50675db7`::finished: {}
Thread-34::DEBUG::2014-11-28 15:45:00,790::task::595::TaskManager.Task::(_updateState) Task=`f73a50b5-22a9-4bbe-8ed4-d8eb50675db7`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:45:00,790::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:45:00,791::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:45:00,791::task::990::TaskManager.Task::(_decref) Task=`f73a50b5-22a9-4bbe-8ed4-d8eb50675db7`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:45:15,873::task::595::TaskManager.Task::(_updateState) Task=`61b5dad3-0553-4962-a53a-e966475a5088`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:45:15,874::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:45:15,874::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:45:15,874::task::1185::TaskManager.Task::(prepare) Task=`61b5dad3-0553-4962-a53a-e966475a5088`::finished: {}
Thread-34::DEBUG::2014-11-28 15:45:15,874::task::595::TaskManager.Task::(_updateState) Task=`61b5dad3-0553-4962-a53a-e966475a5088`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:45:15,874::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:45:15,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:45:15,875::task::990::TaskManager.Task::(_decref) Task=`61b5dad3-0553-4962-a53a-e966475a5088`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:45:30,957::task::595::TaskManager.Task::(_updateState) Task=`9a1d45d4-9c2b-49a7-9624-456454658e34`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:45:30,957::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:45:30,957::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:45:30,957::task::1185::TaskManager.Task::(prepare) Task=`9a1d45d4-9c2b-49a7-9624-456454658e34`::finished: {}
Thread-34::DEBUG::2014-11-28 15:45:30,957::task::595::TaskManager.Task::(_updateState) Task=`9a1d45d4-9c2b-49a7-9624-456454658e34`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:45:30,958::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:45:30,958::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:45:30,958::task::990::TaskManager.Task::(_decref) Task=`9a1d45d4-9c2b-49a7-9624-456454658e34`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:45:46,058::task::595::TaskManager.Task::(_updateState) Task=`8c57efe3-2c2c-47eb-acef-9cd7cc7f5689`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:45:46,058::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:45:46,059::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:45:46,059::task::1185::TaskManager.Task::(prepare) Task=`8c57efe3-2c2c-47eb-acef-9cd7cc7f5689`::finished: {}
Thread-34::DEBUG::2014-11-28 15:45:46,059::task::595::TaskManager.Task::(_updateState) Task=`8c57efe3-2c2c-47eb-acef-9cd7cc7f5689`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:45:46,059::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:45:46,059::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:45:46,060::task::990::TaskManager.Task::(_decref) Task=`8c57efe3-2c2c-47eb-acef-9cd7cc7f5689`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:46:01,144::task::595::TaskManager.Task::(_updateState) Task=`eaa55821-f629-4ca6-9553-3dfd65b97797`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:46:01,145::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:46:01,145::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:46:01,145::task::1185::TaskManager.Task::(prepare) Task=`eaa55821-f629-4ca6-9553-3dfd65b97797`::finished: {}
Thread-34::DEBUG::2014-11-28 15:46:01,145::task::595::TaskManager.Task::(_updateState) Task=`eaa55821-f629-4ca6-9553-3dfd65b97797`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:46:01,145::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:46:01,145::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:46:01,146::task::990::TaskManager.Task::(_decref) Task=`eaa55821-f629-4ca6-9553-3dfd65b97797`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:46:16,229::task::595::TaskManager.Task::(_updateState) Task=`3451b1db-73f2-4969-8ef7-db04cacfec71`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:46:16,229::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:46:16,230::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:46:16,230::task::1185::TaskManager.Task::(prepare) Task=`3451b1db-73f2-4969-8ef7-db04cacfec71`::finished: {}
Thread-34::DEBUG::2014-11-28 15:46:16,230::task::595::TaskManager.Task::(_updateState) Task=`3451b1db-73f2-4969-8ef7-db04cacfec71`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:46:16,230::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:46:16,230::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:46:16,230::task::990::TaskManager.Task::(_decref) Task=`3451b1db-73f2-4969-8ef7-db04cacfec71`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:46:31,337::task::595::TaskManager.Task::(_updateState) Task=`235ff83f-6789-484a-9e93-4fbf179241cc`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:46:31,337::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:46:31,337::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:46:31,337::task::1185::TaskManager.Task::(prepare) Task=`235ff83f-6789-484a-9e93-4fbf179241cc`::finished: {}
Thread-34::DEBUG::2014-11-28 15:46:31,338::task::595::TaskManager.Task::(_updateState) Task=`235ff83f-6789-484a-9e93-4fbf179241cc`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:46:31,338::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:46:31,338::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:46:31,338::task::990::TaskManager.Task::(_decref) Task=`235ff83f-6789-484a-9e93-4fbf179241cc`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:46:46,422::task::595::TaskManager.Task::(_updateState) Task=`65629072-c6b9-416c-a00a-991e3c0db13c`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:46:46,422::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:46:46,422::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:46:46,422::task::1185::TaskManager.Task::(prepare) Task=`65629072-c6b9-416c-a00a-991e3c0db13c`::finished: {}
Thread-34::DEBUG::2014-11-28 15:46:46,423::task::595::TaskManager.Task::(_updateState) Task=`65629072-c6b9-416c-a00a-991e3c0db13c`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:46:46,423::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:46:46,423::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:46:46,423::task::990::TaskManager.Task::(_decref) Task=`65629072-c6b9-416c-a00a-991e3c0db13c`::ref 0 aborting False
Thread-34::DEBUG::2014-11-28 15:47:01,505::task::595::TaskManager.Task::(_updateState) Task=`7afb3da5-4451-40a4-bc71-5d5aa227090c`::moving from state init -> state preparing
Thread-34::INFO::2014-11-28 15:47:01,505::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2014-11-28 15:47:01,505::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-34::DEBUG::2014-11-28 15:47:01,505::task::1185::TaskManager.Task::(prepare) Task=`7afb3da5-4451-40a4-bc71-5d5aa227090c`::finished: {}
Thread-34::DEBUG::2014-11-28 15:47:01,506::task::595::TaskManager.Task::(_updateState) Task=`7afb3da5-4451-40a4-bc71-5d5aa227090c`::moving from state preparing -> state finished
Thread-34::DEBUG::2014-11-28 15:47:01,506::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-34::DEBUG::2014-11-28 15:47:01,506::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-34::DEBUG::2014-11-28 15:47:01,506::task::990::TaskManager.Task::(_decref) Task=`7afb3da5-4451-40a4-bc71-5d5aa227090c`::ref 0 aborting False
Thread-35::ERROR::2014-11-28 15:47:11,214::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::ERROR::2014-11-28 15:47:11,214::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::DEBUG::2014-11-28 15:47:11,215::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:47:11,215::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:47:11,215::task::1185::TaskManager.Task::(prepare) Task=`d7417e89-4ef5-4415-9824-8419adbcab25`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:47:11,215::task::595::TaskManager.Task::(_updateState) Task=`d7417e89-4ef5-4415-9824-8419adbcab25`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:47:11,215::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:47:11,216::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:47:11,216::task::990::TaskManager.Task::(_decref) Task=`d7417e89-4ef5-4415-9824-8419adbcab25`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:47:11,360::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [140fcf95]
Thread-35::DEBUG::2014-11-28 15:47:11,361::task::595::TaskManager.Task::(_updateState) Task=`79bfe6d3-2ea0-403c-a573-055a285f1b77`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:47:11,361::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:47:11,367::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-35::ERROR::2014-11-28 15:47:11,382::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::ERROR::2014-11-28 15:47:11,383::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::DEBUG::2014-11-28 15:47:11,383::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:47:11,383::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:47:11,383::task::1185::TaskManager.Task::(prepare) Task=`79bfe6d3-2ea0-403c-a573-055a285f1b77`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:47:11,384::task::595::TaskManager.Task::(_updateState) Task=`79bfe6d3-2ea0-403c-a573-055a285f1b77`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:47:11,384::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:47:11,384::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:47:11,384::task::990::TaskManager.Task::(_decref) Task=`79bfe6d3-2ea0-403c-a573-055a285f1b77`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:47:16,639::task::595::TaskManager.Task::(_updateState) Task=`9ac914e1-5199-4c93-bad5-6f64f95d57e4`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:47:16,639::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:47:16,639::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:47:16,639::task::1185::TaskManager.Task::(prepare) Task=`9ac914e1-5199-4c93-bad5-6f64f95d57e4`::finished: {}
Thread-35::DEBUG::2014-11-28 15:47:16,640::task::595::TaskManager.Task::(_updateState) Task=`9ac914e1-5199-4c93-bad5-6f64f95d57e4`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:47:16,640::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:47:16,640::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:47:16,640::task::990::TaskManager.Task::(_decref) Task=`9ac914e1-5199-4c93-bad5-6f64f95d57e4`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:47:31,771::task::595::TaskManager.Task::(_updateState) Task=`e70da126-777c-4bef-a7d2-9ebe212cdd3a`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:47:31,772::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:47:31,772::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:47:31,772::task::1185::TaskManager.Task::(prepare) Task=`e70da126-777c-4bef-a7d2-9ebe212cdd3a`::finished: {}
Thread-35::DEBUG::2014-11-28 15:47:31,772::task::595::TaskManager.Task::(_updateState) Task=`e70da126-777c-4bef-a7d2-9ebe212cdd3a`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:47:31,772::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:47:31,773::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:47:31,773::task::990::TaskManager.Task::(_decref) Task=`e70da126-777c-4bef-a7d2-9ebe212cdd3a`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:47:46,854::task::595::TaskManager.Task::(_updateState) Task=`ffe577c5-2111-4140-bc74-1d1e11ec4989`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:47:46,855::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:47:46,855::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:47:46,855::task::1185::TaskManager.Task::(prepare) Task=`ffe577c5-2111-4140-bc74-1d1e11ec4989`::finished: {}
Thread-35::DEBUG::2014-11-28 15:47:46,855::task::595::TaskManager.Task::(_updateState) Task=`ffe577c5-2111-4140-bc74-1d1e11ec4989`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:47:46,855::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:47:46,856::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:47:46,856::task::990::TaskManager.Task::(_decref) Task=`ffe577c5-2111-4140-bc74-1d1e11ec4989`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:48:01,940::task::595::TaskManager.Task::(_updateState) Task=`09c31786-052e-4f28-9b2b-09785a112f90`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:48:01,940::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:48:01,940::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:48:01,941::task::1185::TaskManager.Task::(prepare) Task=`09c31786-052e-4f28-9b2b-09785a112f90`::finished: {}
Thread-35::DEBUG::2014-11-28 15:48:01,941::task::595::TaskManager.Task::(_updateState) Task=`09c31786-052e-4f28-9b2b-09785a112f90`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:48:01,941::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:48:01,941::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:48:01,941::task::990::TaskManager.Task::(_decref) Task=`09c31786-052e-4f28-9b2b-09785a112f90`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:48:11,476::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [ff474c1]
Thread-35::DEBUG::2014-11-28 15:48:11,477::task::595::TaskManager.Task::(_updateState) Task=`fba72529-feb4-4b3e-ab96-7df6153f1031`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:48:11,477::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:48:11,483::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-40::DEBUG::2014-11-28 15:48:17,051::task::595::TaskManager.Task::(_updateState) Task=`b95b6764-d3ef-48fc-877c-71740951eecd`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:48:17,051::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:48:17,051::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:48:17,051::task::1185::TaskManager.Task::(prepare) Task=`b95b6764-d3ef-48fc-877c-71740951eecd`::finished: {}
Thread-40::DEBUG::2014-11-28 15:48:17,051::task::595::TaskManager.Task::(_updateState) Task=`b95b6764-d3ef-48fc-877c-71740951eecd`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:48:17,052::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:48:17,052::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:48:17,052::task::990::TaskManager.Task::(_decref) Task=`b95b6764-d3ef-48fc-877c-71740951eecd`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:48:32,152::task::595::TaskManager.Task::(_updateState) Task=`d43d2ec5-5e78-49c6-9047-a1d47af21f80`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:48:32,152::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:48:32,152::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:48:32,153::task::1185::TaskManager.Task::(prepare) Task=`d43d2ec5-5e78-49c6-9047-a1d47af21f80`::finished: {}
Thread-40::DEBUG::2014-11-28 15:48:32,153::task::595::TaskManager.Task::(_updateState) Task=`d43d2ec5-5e78-49c6-9047-a1d47af21f80`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:48:32,153::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:48:32,153::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:48:32,153::task::990::TaskManager.Task::(_decref) Task=`d43d2ec5-5e78-49c6-9047-a1d47af21f80`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:48:47,266::task::595::TaskManager.Task::(_updateState) Task=`0c8299c5-cb3d-422e-9bb7-a9154b890824`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:48:47,267::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:48:47,267::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:48:47,267::task::1185::TaskManager.Task::(prepare) Task=`0c8299c5-cb3d-422e-9bb7-a9154b890824`::finished: {}
Thread-40::DEBUG::2014-11-28 15:48:47,267::task::595::TaskManager.Task::(_updateState) Task=`0c8299c5-cb3d-422e-9bb7-a9154b890824`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:48:47,267::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:48:47,268::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:48:47,268::task::990::TaskManager.Task::(_decref) Task=`0c8299c5-cb3d-422e-9bb7-a9154b890824`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:49:02,351::task::595::TaskManager.Task::(_updateState) Task=`cc862dd0-8f62-43a1-82a1-094ac793a2d2`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:49:02,351::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:49:02,351::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:49:02,352::task::1185::TaskManager.Task::(prepare) Task=`cc862dd0-8f62-43a1-82a1-094ac793a2d2`::finished: {}
Thread-40::DEBUG::2014-11-28 15:49:02,352::task::595::TaskManager.Task::(_updateState) Task=`cc862dd0-8f62-43a1-82a1-094ac793a2d2`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:49:02,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:49:02,352::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:49:02,352::task::990::TaskManager.Task::(_decref) Task=`cc862dd0-8f62-43a1-82a1-094ac793a2d2`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:49:11,483::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-35::DEBUG::2014-11-28 15:49:11,484::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:49:11,484::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-35::DEBUG::2014-11-28 15:49:11,484::task::1185::TaskManager.Task::(prepare) Task=`fba72529-feb4-4b3e-ab96-7df6153f1031`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-35::DEBUG::2014-11-28 15:49:11,485::task::595::TaskManager.Task::(_updateState) Task=`fba72529-feb4-4b3e-ab96-7df6153f1031`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:11,485::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:11,485::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:11,485::task::990::TaskManager.Task::(_decref) Task=`fba72529-feb4-4b3e-ab96-7df6153f1031`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:49:17,440::task::595::TaskManager.Task::(_updateState) Task=`2f1313a0-031b-4ba5-8b0c-128215385120`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:49:17,440::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:49:17,440::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:49:17,440::task::1185::TaskManager.Task::(prepare) Task=`2f1313a0-031b-4ba5-8b0c-128215385120`::finished: {}
Thread-35::DEBUG::2014-11-28 15:49:17,441::task::595::TaskManager.Task::(_updateState) Task=`2f1313a0-031b-4ba5-8b0c-128215385120`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:17,441::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:17,441::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:17,441::task::990::TaskManager.Task::(_decref) Task=`2f1313a0-031b-4ba5-8b0c-128215385120`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:49:32,544::task::595::TaskManager.Task::(_updateState) Task=`9efbd7b9-92e2-4a03-9696-d4bdff21ddf5`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:49:32,544::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:49:32,544::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:49:32,544::task::1185::TaskManager.Task::(prepare) Task=`9efbd7b9-92e2-4a03-9696-d4bdff21ddf5`::finished: {}
Thread-35::DEBUG::2014-11-28 15:49:32,544::task::595::TaskManager.Task::(_updateState) Task=`9efbd7b9-92e2-4a03-9696-d4bdff21ddf5`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:32,545::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:32,545::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:32,545::task::990::TaskManager.Task::(_decref) Task=`9efbd7b9-92e2-4a03-9696-d4bdff21ddf5`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:49:40,370::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [1b0763e4]
Thread-35::DEBUG::2014-11-28 15:49:40,371::task::595::TaskManager.Task::(_updateState) Task=`2513902b-dde7-4a61-aa41-6522e68331ee`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:49:40,371::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:49:40,377::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-40::DEBUG::2014-11-28 15:49:40,462::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-40::DEBUG::2014-11-28 15:49:40,463::task::595::TaskManager.Task::(_updateState) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:49:40,463::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-40::ERROR::2014-11-28 15:49:40,463::task::866::TaskManager.Task::(_setError) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-40::DEBUG::2014-11-28 15:49:40,464::task::885::TaskManager.Task::(_run) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::Task._run: bbec043f-fb8c-4639-bd8d-dd72c6de2a86 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-40::DEBUG::2014-11-28 15:49:40,464::task::1211::TaskManager.Task::(stop) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::stopping in state preparing (force False)
Thread-40::DEBUG::2014-11-28 15:49:40,464::task::990::TaskManager.Task::(_decref) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::ref 1 aborting True
Thread-40::INFO::2014-11-28 15:49:40,464::task::1168::TaskManager.Task::(prepare) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-40::DEBUG::2014-11-28 15:49:40,465::task::1173::TaskManager.Task::(prepare) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::Prepare: aborted: Unknown pool id, pool not connected
Thread-40::DEBUG::2014-11-28 15:49:40,465::task::990::TaskManager.Task::(_decref) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::ref 0 aborting True
Thread-40::DEBUG::2014-11-28 15:49:40,465::task::925::TaskManager.Task::(_doAbort) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::Task._doAbort: force False
Thread-40::DEBUG::2014-11-28 15:49:40,465::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:49:40,465::task::595::TaskManager.Task::(_updateState) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::moving from state preparing -> state aborting
Thread-40::DEBUG::2014-11-28 15:49:40,465::task::550::TaskManager.Task::(__state_aborting) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::_aborting: recover policy none
Thread-40::DEBUG::2014-11-28 15:49:40,466::task::595::TaskManager.Task::(_updateState) Task=`bbec043f-fb8c-4639-bd8d-dd72c6de2a86`::moving from state aborting -> state failed
Thread-40::DEBUG::2014-11-28 15:49:40,466::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:49:40,466::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::ERROR::2014-11-28 15:49:40,466::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-40::DEBUG::2014-11-28 15:49:40,482::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-40::DEBUG::2014-11-28 15:49:40,484::task::595::TaskManager.Task::(_updateState) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:49:40,484::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24239, domainsMap=None, options=None)
Thread-40::DEBUG::2014-11-28 15:49:40,485::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bae9f1e5-d7f9-4dc7-aaec-54c51eb6bd9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-40::DEBUG::2014-11-28 15:49:40,485::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-40::DEBUG::2014-11-28 15:49:40,485::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-40::DEBUG::2014-11-28 15:49:40,485::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`bae9f1e5-d7f9-4dc7-aaec-54c51eb6bd9f`::Granted request
Thread-40::DEBUG::2014-11-28 15:49:40,486::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`fab55600-3526-4e79-b4d1-e56b89ad35d6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-40::DEBUG::2014-11-28 15:49:40,486::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-40::DEBUG::2014-11-28 15:49:40,487::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-40::DEBUG::2014-11-28 15:49:40,487::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`fab55600-3526-4e79-b4d1-e56b89ad35d6`::Granted request
Thread-40::INFO::2014-11-28 15:49:40,487::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c6c114320
Thread-40::INFO::2014-11-28 15:49:40,487::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24239)
Thread-40::DEBUG::2014-11-28 15:49:40,488::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,488::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,488::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,488::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,489::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,489::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,489::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-40::DEBUG::2014-11-28 15:49:40,489::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-40::DEBUG::2014-11-28 15:49:40,489::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-40::DEBUG::2014-11-28 15:49:40,490::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-40::DEBUG::2014-11-28 15:49:40,490::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-40::DEBUG::2014-11-28 15:49:40,490::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-40::DEBUG::2014-11-28 15:49:40,502::misc::748::SamplingMethod::(__call__) Returning last result
Thread-40::DEBUG::2014-11-28 15:49:40,504::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-35::DEBUG::2014-11-28 15:49:40,527::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-35::DEBUG::2014-11-28 15:49:40,527::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:49:40,528::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-35::DEBUG::2014-11-28 15:49:40,528::task::1185::TaskManager.Task::(prepare) Task=`2513902b-dde7-4a61-aa41-6522e68331ee`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-35::DEBUG::2014-11-28 15:49:40,528::task::595::TaskManager.Task::(_updateState) Task=`2513902b-dde7-4a61-aa41-6522e68331ee`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:40,528::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:40,528::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:40,529::task::990::TaskManager.Task::(_decref) Task=`2513902b-dde7-4a61-aa41-6522e68331ee`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:49:40,545::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-40::DEBUG::2014-11-28 15:49:40,546::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,546::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,546::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,547::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,547::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,547::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-40::DEBUG::2014-11-28 15:49:40,547::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::DEBUG::2014-11-28 15:49:47,647::task::595::TaskManager.Task::(_updateState) Task=`2e3af2b4-0bef-4190-8ac3-d2eea05c1fbd`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:49:47,647::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:49:47,647::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:49:47,648::task::1185::TaskManager.Task::(prepare) Task=`2e3af2b4-0bef-4190-8ac3-d2eea05c1fbd`::finished: {}
Thread-35::DEBUG::2014-11-28 15:49:47,648::task::595::TaskManager.Task::(_updateState) Task=`2e3af2b4-0bef-4190-8ac3-d2eea05c1fbd`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:47,648::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:47,648::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:47,648::task::990::TaskManager.Task::(_decref) Task=`2e3af2b4-0bef-4190-8ac3-d2eea05c1fbd`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:49:59,935::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [5a42c442]
Thread-35::DEBUG::2014-11-28 15:49:59,936::task::595::TaskManager.Task::(_updateState) Task=`a809417c-d296-4188-ba98-fa6dc6cbc718`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:49:59,936::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:49:59,943::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-35::ERROR::2014-11-28 15:49:59,973::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::ERROR::2014-11-28 15:49:59,974::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::DEBUG::2014-11-28 15:49:59,974::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:49:59,974::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:49:59,974::task::1185::TaskManager.Task::(prepare) Task=`a809417c-d296-4188-ba98-fa6dc6cbc718`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:49:59,975::task::595::TaskManager.Task::(_updateState) Task=`a809417c-d296-4188-ba98-fa6dc6cbc718`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:49:59,975::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:49:59,975::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:49:59,975::task::990::TaskManager.Task::(_decref) Task=`a809417c-d296-4188-ba98-fa6dc6cbc718`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:50:00,095::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [90bcaa7]
Thread-35::DEBUG::2014-11-28 15:50:00,095::task::595::TaskManager.Task::(_updateState) Task=`8adf9f42-30fd-4c5d-ac4b-a96653d05d2d`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:50:00,096::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:50:00,116::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-42::WARNING::2014-11-28 15:50:40,549::fileSD::673::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/172.16.241.99:_mnt_storage timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 662, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-40::ERROR::2014-11-28 15:50:40,550::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomainPath
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-40::DEBUG::2014-11-28 15:50:40,551::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-40::DEBUG::2014-11-28 15:50:40,551::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-40::DEBUG::2014-11-28 15:50:40,551::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-40::DEBUG::2014-11-28 15:50:40,551::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-40::DEBUG::2014-11-28 15:50:40,551::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-40::DEBUG::2014-11-28 15:50:40,552::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-40::DEBUG::2014-11-28 15:50:40,552::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-40::DEBUG::2014-11-28 15:50:40,552::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-40::ERROR::2014-11-28 15:50:40,552::task::866::TaskManager.Task::(_setError) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1360, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'
Thread-40::DEBUG::2014-11-28 15:50:40,553::task::885::TaskManager.Task::(_run) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::Task._run: 42f6a798-0c8e-4129-855f-728b7aedb859 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24239, None) {} failed - stopping task
Thread-40::DEBUG::2014-11-28 15:50:40,553::task::1211::TaskManager.Task::(stop) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::stopping in state preparing (force False)
Thread-40::DEBUG::2014-11-28 15:50:40,553::task::990::TaskManager.Task::(_decref) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::ref 1 aborting True
Thread-40::INFO::2014-11-28 15:50:40,553::task::1168::TaskManager.Task::(prepare) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-40::DEBUG::2014-11-28 15:50:40,553::task::1173::TaskManager.Task::(prepare) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::Prepare: aborted: Cannot find master domain
Thread-40::DEBUG::2014-11-28 15:50:40,553::task::990::TaskManager.Task::(_decref) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::ref 0 aborting True
Thread-40::DEBUG::2014-11-28 15:50:40,554::task::925::TaskManager.Task::(_doAbort) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::Task._doAbort: force False
Thread-40::DEBUG::2014-11-28 15:50:40,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:50:40,554::task::595::TaskManager.Task::(_updateState) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::moving from state preparing -> state aborting
Thread-40::DEBUG::2014-11-28 15:50:40,554::task::550::TaskManager.Task::(__state_aborting) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::_aborting: recover policy none
Thread-40::DEBUG::2014-11-28 15:50:40,554::task::595::TaskManager.Task::(_updateState) Task=`42f6a798-0c8e-4129-855f-728b7aedb859`::moving from state aborting -> state failed
Thread-40::DEBUG::2014-11-28 15:50:40,554::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:50:40,554::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::ERROR::2014-11-28 15:50:40,555::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=45995288-a7d6-4cbf-8b2f-63fa35642bad, msdUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584'", 'code': 304}}
Thread-40::DEBUG::2014-11-28 15:50:40,558::task::595::TaskManager.Task::(_updateState) Task=`d8ec7072-71dd-4ea7-9552-d75682c1f548`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:50:40,558::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:50:40,558::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:50:40,559::task::1185::TaskManager.Task::(prepare) Task=`d8ec7072-71dd-4ea7-9552-d75682c1f548`::finished: {}
Thread-40::DEBUG::2014-11-28 15:50:40,559::task::595::TaskManager.Task::(_updateState) Task=`d8ec7072-71dd-4ea7-9552-d75682c1f548`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:50:40,559::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:50:40,559::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:50:40,559::task::990::TaskManager.Task::(_decref) Task=`d8ec7072-71dd-4ea7-9552-d75682c1f548`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:50:40,585::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-40::DEBUG::2014-11-28 15:50:40,585::task::595::TaskManager.Task::(_updateState) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:50:40,585::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-40::ERROR::2014-11-28 15:50:40,586::task::866::TaskManager.Task::(_setError) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-40::DEBUG::2014-11-28 15:50:40,586::task::885::TaskManager.Task::(_run) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::Task._run: 0d77c9eb-74b2-4f3d-bd3b-a6003761eeff ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-40::DEBUG::2014-11-28 15:50:40,586::task::1211::TaskManager.Task::(stop) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::stopping in state preparing (force False)
Thread-40::DEBUG::2014-11-28 15:50:40,586::task::990::TaskManager.Task::(_decref) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::ref 1 aborting True
Thread-40::INFO::2014-11-28 15:50:40,586::task::1168::TaskManager.Task::(prepare) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::1173::TaskManager.Task::(prepare) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::Prepare: aborted: Unknown pool id, pool not connected
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::990::TaskManager.Task::(_decref) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::ref 0 aborting True
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::925::TaskManager.Task::(_doAbort) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::Task._doAbort: force False
Thread-40::DEBUG::2014-11-28 15:50:40,587::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::595::TaskManager.Task::(_updateState) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::moving from state preparing -> state aborting
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::550::TaskManager.Task::(__state_aborting) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::_aborting: recover policy none
Thread-40::DEBUG::2014-11-28 15:50:40,587::task::595::TaskManager.Task::(_updateState) Task=`0d77c9eb-74b2-4f3d-bd3b-a6003761eeff`::moving from state aborting -> state failed
Thread-40::DEBUG::2014-11-28 15:50:40,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:50:40,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::ERROR::2014-11-28 15:50:40,588::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-40::DEBUG::2014-11-28 15:50:55,645::task::595::TaskManager.Task::(_updateState) Task=`2d573f7e-9847-4a73-a570-4a13717da30a`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:50:55,645::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:50:55,645::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:50:55,645::task::1185::TaskManager.Task::(prepare) Task=`2d573f7e-9847-4a73-a570-4a13717da30a`::finished: {}
Thread-40::DEBUG::2014-11-28 15:50:55,645::task::595::TaskManager.Task::(_updateState) Task=`2d573f7e-9847-4a73-a570-4a13717da30a`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:50:55,646::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:50:55,646::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:50:55,646::task::990::TaskManager.Task::(_decref) Task=`2d573f7e-9847-4a73-a570-4a13717da30a`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:51:10,729::task::595::TaskManager.Task::(_updateState) Task=`d05b0355-1481-4b04-8db9-5e5279847d31`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:51:10,729::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:51:10,729::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:51:10,729::task::1185::TaskManager.Task::(prepare) Task=`d05b0355-1481-4b04-8db9-5e5279847d31`::finished: {}
Thread-40::DEBUG::2014-11-28 15:51:10,730::task::595::TaskManager.Task::(_updateState) Task=`d05b0355-1481-4b04-8db9-5e5279847d31`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:51:10,730::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:51:10,730::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:51:10,730::task::990::TaskManager.Task::(_decref) Task=`d05b0355-1481-4b04-8db9-5e5279847d31`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:51:25,815::task::595::TaskManager.Task::(_updateState) Task=`e8260524-cedc-451a-aa52-775b4e456e07`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:51:25,816::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:51:25,816::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:51:25,816::task::1185::TaskManager.Task::(prepare) Task=`e8260524-cedc-451a-aa52-775b4e456e07`::finished: {}
Thread-40::DEBUG::2014-11-28 15:51:25,816::task::595::TaskManager.Task::(_updateState) Task=`e8260524-cedc-451a-aa52-775b4e456e07`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:51:25,816::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:51:25,817::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:51:25,817::task::990::TaskManager.Task::(_decref) Task=`e8260524-cedc-451a-aa52-775b4e456e07`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:51:40,905::task::595::TaskManager.Task::(_updateState) Task=`a42bf62e-e05b-4deb-814a-a641e4d8f650`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:51:40,905::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:51:40,906::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:51:40,906::task::1185::TaskManager.Task::(prepare) Task=`a42bf62e-e05b-4deb-814a-a641e4d8f650`::finished: {}
Thread-40::DEBUG::2014-11-28 15:51:40,906::task::595::TaskManager.Task::(_updateState) Task=`a42bf62e-e05b-4deb-814a-a641e4d8f650`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:51:40,906::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:51:40,906::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:51:40,906::task::990::TaskManager.Task::(_decref) Task=`a42bf62e-e05b-4deb-814a-a641e4d8f650`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:51:55,992::task::595::TaskManager.Task::(_updateState) Task=`4298cebd-4463-4565-8758-85b39e8b39c0`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:51:55,992::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:51:55,992::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:51:55,993::task::1185::TaskManager.Task::(prepare) Task=`4298cebd-4463-4565-8758-85b39e8b39c0`::finished: {}
Thread-40::DEBUG::2014-11-28 15:51:55,993::task::595::TaskManager.Task::(_updateState) Task=`4298cebd-4463-4565-8758-85b39e8b39c0`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:51:55,993::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:51:55,993::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:51:55,993::task::990::TaskManager.Task::(_decref) Task=`4298cebd-4463-4565-8758-85b39e8b39c0`::ref 0 aborting False
Thread-40::DEBUG::2014-11-28 15:52:11,078::task::595::TaskManager.Task::(_updateState) Task=`7a429b89-0d25-4eb1-a3ac-24c75c9342c8`::moving from state init -> state preparing
Thread-40::INFO::2014-11-28 15:52:11,079::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-40::INFO::2014-11-28 15:52:11,079::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-40::DEBUG::2014-11-28 15:52:11,079::task::1185::TaskManager.Task::(prepare) Task=`7a429b89-0d25-4eb1-a3ac-24c75c9342c8`::finished: {}
Thread-40::DEBUG::2014-11-28 15:52:11,079::task::595::TaskManager.Task::(_updateState) Task=`7a429b89-0d25-4eb1-a3ac-24c75c9342c8`::moving from state preparing -> state finished
Thread-40::DEBUG::2014-11-28 15:52:11,080::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-40::DEBUG::2014-11-28 15:52:11,080::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-40::DEBUG::2014-11-28 15:52:11,080::task::990::TaskManager.Task::(_decref) Task=`7a429b89-0d25-4eb1-a3ac-24c75c9342c8`::ref 0 aborting False
Thread-35::ERROR::2014-11-28 15:52:11,230::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::ERROR::2014-11-28 15:52:11,230::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::DEBUG::2014-11-28 15:52:11,231::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:52:11,231::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:52:11,231::task::1185::TaskManager.Task::(prepare) Task=`8adf9f42-30fd-4c5d-ac4b-a96653d05d2d`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:52:11,231::task::595::TaskManager.Task::(_updateState) Task=`8adf9f42-30fd-4c5d-ac4b-a96653d05d2d`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:52:11,231::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:52:11,232::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:52:11,232::task::990::TaskManager.Task::(_decref) Task=`8adf9f42-30fd-4c5d-ac4b-a96653d05d2d`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:52:11,366::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3d0f27da]
Thread-35::DEBUG::2014-11-28 15:52:11,366::task::595::TaskManager.Task::(_updateState) Task=`45c41a5a-a221-43c2-b8c4-49d9ad2bcaf7`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:52:11,366::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:52:11,372::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-35::ERROR::2014-11-28 15:52:11,388::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::ERROR::2014-11-28 15:52:11,389::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::DEBUG::2014-11-28 15:52:11,389::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:52:11,389::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:52:11,389::task::1185::TaskManager.Task::(prepare) Task=`45c41a5a-a221-43c2-b8c4-49d9ad2bcaf7`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:52:11,390::task::595::TaskManager.Task::(_updateState) Task=`45c41a5a-a221-43c2-b8c4-49d9ad2bcaf7`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:52:11,390::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:52:11,390::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:52:11,390::task::990::TaskManager.Task::(_decref) Task=`45c41a5a-a221-43c2-b8c4-49d9ad2bcaf7`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:52:26,257::task::595::TaskManager.Task::(_updateState) Task=`fad79d98-9e80-43e4-9fd4-a7e8b057cc29`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:52:26,257::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:52:26,258::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:52:26,258::task::1185::TaskManager.Task::(prepare) Task=`fad79d98-9e80-43e4-9fd4-a7e8b057cc29`::finished: {}
Thread-35::DEBUG::2014-11-28 15:52:26,258::task::595::TaskManager.Task::(_updateState) Task=`fad79d98-9e80-43e4-9fd4-a7e8b057cc29`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:52:26,258::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:52:26,258::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:52:26,258::task::990::TaskManager.Task::(_decref) Task=`fad79d98-9e80-43e4-9fd4-a7e8b057cc29`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:52:41,354::task::595::TaskManager.Task::(_updateState) Task=`84bc2192-f7f5-4712-ac6f-6094558c0364`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:52:41,354::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:52:41,355::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:52:41,355::task::1185::TaskManager.Task::(prepare) Task=`84bc2192-f7f5-4712-ac6f-6094558c0364`::finished: {}
Thread-35::DEBUG::2014-11-28 15:52:41,355::task::595::TaskManager.Task::(_updateState) Task=`84bc2192-f7f5-4712-ac6f-6094558c0364`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:52:41,355::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:52:41,355::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:52:41,356::task::990::TaskManager.Task::(_decref) Task=`84bc2192-f7f5-4712-ac6f-6094558c0364`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:52:56,450::task::595::TaskManager.Task::(_updateState) Task=`06d848dd-8b9b-4365-9851-9b29eb608371`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:52:56,450::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:52:56,450::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:52:56,451::task::1185::TaskManager.Task::(prepare) Task=`06d848dd-8b9b-4365-9851-9b29eb608371`::finished: {}
Thread-35::DEBUG::2014-11-28 15:52:56,451::task::595::TaskManager.Task::(_updateState) Task=`06d848dd-8b9b-4365-9851-9b29eb608371`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:52:56,451::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:52:56,451::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:52:56,451::task::990::TaskManager.Task::(_decref) Task=`06d848dd-8b9b-4365-9851-9b29eb608371`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:53:11,451::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [6dd78ef0]
Thread-35::DEBUG::2014-11-28 15:53:11,452::task::595::TaskManager.Task::(_updateState) Task=`b7dd947e-df1c-4402-8522-077784e325b0`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:53:11,452::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:53:11,458::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-35::DEBUG::2014-11-28 15:53:11,460::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-35::DEBUG::2014-11-28 15:53:11,460::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:53:11,460::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-35::DEBUG::2014-11-28 15:53:11,461::task::1185::TaskManager.Task::(prepare) Task=`b7dd947e-df1c-4402-8522-077784e325b0`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-35::DEBUG::2014-11-28 15:53:11,461::task::595::TaskManager.Task::(_updateState) Task=`b7dd947e-df1c-4402-8522-077784e325b0`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:53:11,461::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:53:11,461::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:53:11,461::task::990::TaskManager.Task::(_decref) Task=`b7dd947e-df1c-4402-8522-077784e325b0`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:53:11,544::task::595::TaskManager.Task::(_updateState) Task=`9f9a8aac-cc22-4e5a-b43f-3ce8ac5b121f`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:53:11,544::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:53:11,544::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:53:11,544::task::1185::TaskManager.Task::(prepare) Task=`9f9a8aac-cc22-4e5a-b43f-3ce8ac5b121f`::finished: {}
Thread-35::DEBUG::2014-11-28 15:53:11,544::task::595::TaskManager.Task::(_updateState) Task=`9f9a8aac-cc22-4e5a-b43f-3ce8ac5b121f`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:53:11,545::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:53:11,545::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:53:11,545::task::990::TaskManager.Task::(_decref) Task=`9f9a8aac-cc22-4e5a-b43f-3ce8ac5b121f`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:53:11,698::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [7dcbaff0]
Thread-35::DEBUG::2014-11-28 15:53:11,698::task::595::TaskManager.Task::(_updateState) Task=`766a3d33-354e-42c0-b815-d25b454cf02f`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:53:11,698::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:53:11,704::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-43::DEBUG::2014-11-28 15:53:26,658::task::595::TaskManager.Task::(_updateState) Task=`55a46c58-5ff5-4817-a1b5-5f6a8219ea85`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:53:26,658::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:53:26,659::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:53:26,659::task::1185::TaskManager.Task::(prepare) Task=`55a46c58-5ff5-4817-a1b5-5f6a8219ea85`::finished: {}
Thread-43::DEBUG::2014-11-28 15:53:26,659::task::595::TaskManager.Task::(_updateState) Task=`55a46c58-5ff5-4817-a1b5-5f6a8219ea85`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:53:26,659::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:53:26,659::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:53:26,660::task::990::TaskManager.Task::(_decref) Task=`55a46c58-5ff5-4817-a1b5-5f6a8219ea85`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:53:41,742::task::595::TaskManager.Task::(_updateState) Task=`5c0f86d5-a9f2-4fd1-882c-ba7523bf7f18`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:53:41,742::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:53:41,743::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:53:41,743::task::1185::TaskManager.Task::(prepare) Task=`5c0f86d5-a9f2-4fd1-882c-ba7523bf7f18`::finished: {}
Thread-43::DEBUG::2014-11-28 15:53:41,743::task::595::TaskManager.Task::(_updateState) Task=`5c0f86d5-a9f2-4fd1-882c-ba7523bf7f18`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:53:41,743::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:53:41,743::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:53:41,743::task::990::TaskManager.Task::(_decref) Task=`5c0f86d5-a9f2-4fd1-882c-ba7523bf7f18`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:53:56,836::task::595::TaskManager.Task::(_updateState) Task=`4a0c332c-f854-4ace-9baf-423dee531d6e`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:53:56,836::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:53:56,836::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:53:56,836::task::1185::TaskManager.Task::(prepare) Task=`4a0c332c-f854-4ace-9baf-423dee531d6e`::finished: {}
Thread-43::DEBUG::2014-11-28 15:53:56,837::task::595::TaskManager.Task::(_updateState) Task=`4a0c332c-f854-4ace-9baf-423dee531d6e`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:53:56,837::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:53:56,837::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:53:56,837::task::990::TaskManager.Task::(_decref) Task=`4a0c332c-f854-4ace-9baf-423dee531d6e`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:11,704::hsm::2397::Storage.HSM::(connectStorageServer) prefetch failed: {'e7eaae86-11ea-4c08-8c11-8b8c54373389': <function findDomain at 0x26df230>, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': <function findDomain at 0x26df230>}
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2394, in connectStorageServer
    doms = self.__prefetchDomains(domType, conObj)
  File "/usr/share/vdsm/storage/hsm.py", line 2331, in __prefetchDomains
    goop.glob.glob(os.path.join(lPath, uuidPatern)))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-35::DEBUG::2014-11-28 15:54:11,705::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:54:11,705::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-35::DEBUG::2014-11-28 15:54:11,705::task::1185::TaskManager.Task::(prepare) Task=`766a3d33-354e-42c0-b815-d25b454cf02f`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-35::DEBUG::2014-11-28 15:54:11,706::task::595::TaskManager.Task::(_updateState) Task=`766a3d33-354e-42c0-b815-d25b454cf02f`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:11,706::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:11,706::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:11,706::task::990::TaskManager.Task::(_decref) Task=`766a3d33-354e-42c0-b815-d25b454cf02f`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:11,924::task::595::TaskManager.Task::(_updateState) Task=`e9a94849-21dc-4236-9af7-10764fc15795`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:54:11,924::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:54:11,925::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:54:11,925::task::1185::TaskManager.Task::(prepare) Task=`e9a94849-21dc-4236-9af7-10764fc15795`::finished: {}
Thread-35::DEBUG::2014-11-28 15:54:11,925::task::595::TaskManager.Task::(_updateState) Task=`e9a94849-21dc-4236-9af7-10764fc15795`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:11,925::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:11,925::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:11,925::task::990::TaskManager.Task::(_decref) Task=`e9a94849-21dc-4236-9af7-10764fc15795`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:27,023::task::595::TaskManager.Task::(_updateState) Task=`b3fb5e75-f8a5-4202-8a69-7bb78d95fafe`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:54:27,023::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:54:27,023::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:54:27,024::task::1185::TaskManager.Task::(prepare) Task=`b3fb5e75-f8a5-4202-8a69-7bb78d95fafe`::finished: {}
Thread-35::DEBUG::2014-11-28 15:54:27,024::task::595::TaskManager.Task::(_updateState) Task=`b3fb5e75-f8a5-4202-8a69-7bb78d95fafe`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:27,024::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:27,024::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:27,024::task::990::TaskManager.Task::(_decref) Task=`b3fb5e75-f8a5-4202-8a69-7bb78d95fafe`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:42,102::task::595::TaskManager.Task::(_updateState) Task=`0af021cb-d1a8-4fa1-b158-e09feaa53473`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:54:42,102::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:54:42,102::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:54:42,102::task::1185::TaskManager.Task::(prepare) Task=`0af021cb-d1a8-4fa1-b158-e09feaa53473`::finished: {}
Thread-35::DEBUG::2014-11-28 15:54:42,102::task::595::TaskManager.Task::(_updateState) Task=`0af021cb-d1a8-4fa1-b158-e09feaa53473`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:42,103::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:42,103::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:42,103::task::990::TaskManager.Task::(_decref) Task=`0af021cb-d1a8-4fa1-b158-e09feaa53473`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:57,192::task::595::TaskManager.Task::(_updateState) Task=`e2e3b5cc-6adb-42c6-ab12-b02c45b19480`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:54:57,193::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:54:57,193::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:54:57,193::task::1185::TaskManager.Task::(prepare) Task=`e2e3b5cc-6adb-42c6-ab12-b02c45b19480`::finished: {}
Thread-35::DEBUG::2014-11-28 15:54:57,193::task::595::TaskManager.Task::(_updateState) Task=`e2e3b5cc-6adb-42c6-ab12-b02c45b19480`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:57,193::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:57,194::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:57,194::task::990::TaskManager.Task::(_decref) Task=`e2e3b5cc-6adb-42c6-ab12-b02c45b19480`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:54:59,908::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [2b3c8feb]
Thread-35::DEBUG::2014-11-28 15:54:59,909::task::595::TaskManager.Task::(_updateState) Task=`d0e2daf0-e224-46ab-8fe8-c786ea5fa12d`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:54:59,909::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:54:59,915::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-35::ERROR::2014-11-28 15:54:59,933::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::ERROR::2014-11-28 15:54:59,933::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: access denied by server while mounting 172.16.241.100:/var/lib/exports/iso\n')
Thread-35::DEBUG::2014-11-28 15:54:59,934::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:54:59,934::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:54:59,934::task::1185::TaskManager.Task::(prepare) Task=`d0e2daf0-e224-46ab-8fe8-c786ea5fa12d`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}]}
Thread-35::DEBUG::2014-11-28 15:54:59,934::task::595::TaskManager.Task::(_updateState) Task=`d0e2daf0-e224-46ab-8fe8-c786ea5fa12d`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:54:59,935::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:54:59,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:54:59,935::task::990::TaskManager.Task::(_decref) Task=`d0e2daf0-e224-46ab-8fe8-c786ea5fa12d`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:55:00,097::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [65b71295]
Thread-35::DEBUG::2014-11-28 15:55:00,097::task::595::TaskManager.Task::(_updateState) Task=`2cef81da-19c9-44cf-be43-05f2c69c4629`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:55:00,097::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:55:00,113::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-43::DEBUG::2014-11-28 15:55:12,273::task::595::TaskManager.Task::(_updateState) Task=`2ef768a4-b41a-47d1-9a52-f081ded69ce6`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:55:12,274::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:55:12,274::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:55:12,274::task::1185::TaskManager.Task::(prepare) Task=`2ef768a4-b41a-47d1-9a52-f081ded69ce6`::finished: {}
Thread-43::DEBUG::2014-11-28 15:55:12,274::task::595::TaskManager.Task::(_updateState) Task=`2ef768a4-b41a-47d1-9a52-f081ded69ce6`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:55:12,274::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:55:12,274::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:55:12,275::task::990::TaskManager.Task::(_decref) Task=`2ef768a4-b41a-47d1-9a52-f081ded69ce6`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:55:27,362::task::595::TaskManager.Task::(_updateState) Task=`f3c16e51-60a2-41de-b081-c6adc520e814`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:55:27,363::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:55:27,363::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:55:27,363::task::1185::TaskManager.Task::(prepare) Task=`f3c16e51-60a2-41de-b081-c6adc520e814`::finished: {}
Thread-43::DEBUG::2014-11-28 15:55:27,363::task::595::TaskManager.Task::(_updateState) Task=`f3c16e51-60a2-41de-b081-c6adc520e814`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:55:27,363::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:55:27,364::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:55:27,364::task::990::TaskManager.Task::(_decref) Task=`f3c16e51-60a2-41de-b081-c6adc520e814`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:55:42,447::task::595::TaskManager.Task::(_updateState) Task=`3872033c-c020-4c3c-8978-6dfc84283c40`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:55:42,447::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:55:42,447::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:55:42,447::task::1185::TaskManager.Task::(prepare) Task=`3872033c-c020-4c3c-8978-6dfc84283c40`::finished: {}
Thread-43::DEBUG::2014-11-28 15:55:42,447::task::595::TaskManager.Task::(_updateState) Task=`3872033c-c020-4c3c-8978-6dfc84283c40`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:55:42,448::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:55:42,448::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:55:42,448::task::990::TaskManager.Task::(_decref) Task=`3872033c-c020-4c3c-8978-6dfc84283c40`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:55:57,530::task::595::TaskManager.Task::(_updateState) Task=`2162a089-1d7f-45a9-b61f-a6d7332dca62`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:55:57,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:55:57,530::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:55:57,530::task::1185::TaskManager.Task::(prepare) Task=`2162a089-1d7f-45a9-b61f-a6d7332dca62`::finished: {}
Thread-43::DEBUG::2014-11-28 15:55:57,531::task::595::TaskManager.Task::(_updateState) Task=`2162a089-1d7f-45a9-b61f-a6d7332dca62`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:55:57,531::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:55:57,531::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:55:57,531::task::990::TaskManager.Task::(_decref) Task=`2162a089-1d7f-45a9-b61f-a6d7332dca62`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:56:12,636::task::595::TaskManager.Task::(_updateState) Task=`0183df8f-0279-4ca2-b253-135b611ea9ed`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:56:12,636::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:56:12,636::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:56:12,636::task::1185::TaskManager.Task::(prepare) Task=`0183df8f-0279-4ca2-b253-135b611ea9ed`::finished: {}
Thread-43::DEBUG::2014-11-28 15:56:12,636::task::595::TaskManager.Task::(_updateState) Task=`0183df8f-0279-4ca2-b253-135b611ea9ed`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:56:12,637::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:56:12,637::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:56:12,637::task::990::TaskManager.Task::(_decref) Task=`0183df8f-0279-4ca2-b253-135b611ea9ed`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:56:27,722::task::595::TaskManager.Task::(_updateState) Task=`60ccbdf1-7a03-4372-89bd-c5986fbfa621`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:56:27,723::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:56:27,723::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:56:27,723::task::1185::TaskManager.Task::(prepare) Task=`60ccbdf1-7a03-4372-89bd-c5986fbfa621`::finished: {}
Thread-43::DEBUG::2014-11-28 15:56:27,723::task::595::TaskManager.Task::(_updateState) Task=`60ccbdf1-7a03-4372-89bd-c5986fbfa621`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:56:27,724::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:56:27,724::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:56:27,724::task::990::TaskManager.Task::(_decref) Task=`60ccbdf1-7a03-4372-89bd-c5986fbfa621`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:56:42,808::task::595::TaskManager.Task::(_updateState) Task=`fdcca3aa-e4ac-4b4f-8d7e-9e1a1769d0d4`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:56:42,809::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:56:42,809::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:56:42,809::task::1185::TaskManager.Task::(prepare) Task=`fdcca3aa-e4ac-4b4f-8d7e-9e1a1769d0d4`::finished: {}
Thread-43::DEBUG::2014-11-28 15:56:42,809::task::595::TaskManager.Task::(_updateState) Task=`fdcca3aa-e4ac-4b4f-8d7e-9e1a1769d0d4`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:56:42,809::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:56:42,810::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:56:42,810::task::990::TaskManager.Task::(_decref) Task=`fdcca3aa-e4ac-4b4f-8d7e-9e1a1769d0d4`::ref 0 aborting False
Thread-43::DEBUG::2014-11-28 15:56:57,895::task::595::TaskManager.Task::(_updateState) Task=`7735fe20-1a6b-48f5-9342-e671cea4b23d`::moving from state init -> state preparing
Thread-43::INFO::2014-11-28 15:56:57,896::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-43::INFO::2014-11-28 15:56:57,896::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-43::DEBUG::2014-11-28 15:56:57,896::task::1185::TaskManager.Task::(prepare) Task=`7735fe20-1a6b-48f5-9342-e671cea4b23d`::finished: {}
Thread-43::DEBUG::2014-11-28 15:56:57,896::task::595::TaskManager.Task::(_updateState) Task=`7735fe20-1a6b-48f5-9342-e671cea4b23d`::moving from state preparing -> state finished
Thread-43::DEBUG::2014-11-28 15:56:57,896::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-43::DEBUG::2014-11-28 15:56:57,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-43::DEBUG::2014-11-28 15:56:57,897::task::990::TaskManager.Task::(_decref) Task=`7735fe20-1a6b-48f5-9342-e671cea4b23d`::ref 0 aborting False
Thread-35::ERROR::2014-11-28 15:57:11,227::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::ERROR::2014-11-28 15:57:11,228::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-35::DEBUG::2014-11-28 15:57:11,228::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:57:11,228::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:57:11,228::task::1185::TaskManager.Task::(prepare) Task=`2cef81da-19c9-44cf-be43-05f2c69c4629`::finished: {'statuslist': [{'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}]}
Thread-35::DEBUG::2014-11-28 15:57:11,228::task::595::TaskManager.Task::(_updateState) Task=`2cef81da-19c9-44cf-be43-05f2c69c4629`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:11,229::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:11,229::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:11,229::task::990::TaskManager.Task::(_decref) Task=`2cef81da-19c9-44cf-be43-05f2c69c4629`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:57:11,419::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [539bab09]
Thread-35::DEBUG::2014-11-28 15:57:11,420::task::595::TaskManager.Task::(_updateState) Task=`2976c075-ebbe-4024-bce8-758e7666cc8a`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:57:11,420::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:57:11,426::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-35::ERROR::2014-11-28 15:57:11,444::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::ERROR::2014-11-28 15:57:11,445::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-35::DEBUG::2014-11-28 15:57:11,445::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-35::INFO::2014-11-28 15:57:11,445::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:57:11,445::task::1185::TaskManager.Task::(prepare) Task=`2976c075-ebbe-4024-bce8-758e7666cc8a`::finished: {'statuslist': [{'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:57:11,446::task::595::TaskManager.Task::(_updateState) Task=`2976c075-ebbe-4024-bce8-758e7666cc8a`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:11,446::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:11,446::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:11,446::task::990::TaskManager.Task::(_decref) Task=`2976c075-ebbe-4024-bce8-758e7666cc8a`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:57:12,982::task::595::TaskManager.Task::(_updateState) Task=`8d0ecfc8-a1dc-486e-aa6e-93d503df1e23`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:57:12,982::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:57:12,983::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:57:12,983::task::1185::TaskManager.Task::(prepare) Task=`8d0ecfc8-a1dc-486e-aa6e-93d503df1e23`::finished: {}
Thread-35::DEBUG::2014-11-28 15:57:12,983::task::595::TaskManager.Task::(_updateState) Task=`8d0ecfc8-a1dc-486e-aa6e-93d503df1e23`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:12,983::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:12,983::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:12,984::task::990::TaskManager.Task::(_decref) Task=`8d0ecfc8-a1dc-486e-aa6e-93d503df1e23`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:57:28,086::task::595::TaskManager.Task::(_updateState) Task=`b6bf6b61-a2e2-4eca-8beb-f0d0c1c2ae0c`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:57:28,086::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:57:28,086::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:57:28,086::task::1185::TaskManager.Task::(prepare) Task=`b6bf6b61-a2e2-4eca-8beb-f0d0c1c2ae0c`::finished: {}
Thread-35::DEBUG::2014-11-28 15:57:28,087::task::595::TaskManager.Task::(_updateState) Task=`b6bf6b61-a2e2-4eca-8beb-f0d0c1c2ae0c`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:28,087::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:28,087::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:28,087::task::990::TaskManager.Task::(_decref) Task=`b6bf6b61-a2e2-4eca-8beb-f0d0c1c2ae0c`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:57:43,172::task::595::TaskManager.Task::(_updateState) Task=`118779c0-1c44-4830-a0bc-8e169f0d3684`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:57:43,172::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:57:43,172::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:57:43,172::task::1185::TaskManager.Task::(prepare) Task=`118779c0-1c44-4830-a0bc-8e169f0d3684`::finished: {}
Thread-35::DEBUG::2014-11-28 15:57:43,173::task::595::TaskManager.Task::(_updateState) Task=`118779c0-1c44-4830-a0bc-8e169f0d3684`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:43,173::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:43,173::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:43,173::task::990::TaskManager.Task::(_decref) Task=`118779c0-1c44-4830-a0bc-8e169f0d3684`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:57:58,258::task::595::TaskManager.Task::(_updateState) Task=`b9bec747-10b9-4877-a9f5-56f70900e027`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:57:58,258::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:57:58,259::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:57:58,259::task::1185::TaskManager.Task::(prepare) Task=`b9bec747-10b9-4877-a9f5-56f70900e027`::finished: {}
Thread-35::DEBUG::2014-11-28 15:57:58,259::task::595::TaskManager.Task::(_updateState) Task=`b9bec747-10b9-4877-a9f5-56f70900e027`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:57:58,259::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:57:58,259::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:57:58,260::task::990::TaskManager.Task::(_decref) Task=`b9bec747-10b9-4877-a9f5-56f70900e027`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:58:10,989::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:58:10,990::task::595::TaskManager.Task::(_updateState) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:58:10,990::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-35::ERROR::2014-11-28 15:58:10,990::task::866::TaskManager.Task::(_setError) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-35::DEBUG::2014-11-28 15:58:10,991::task::885::TaskManager.Task::(_run) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::Task._run: 75b32def-be7a-49ec-9299-3e0b57e87c06 ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:58:10,991::task::1211::TaskManager.Task::(stop) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:58:10,991::task::990::TaskManager.Task::(_decref) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::ref 1 aborting True
Thread-35::INFO::2014-11-28 15:58:10,991::task::1168::TaskManager.Task::(prepare) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-35::DEBUG::2014-11-28 15:58:10,992::task::1173::TaskManager.Task::(prepare) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::Prepare: aborted: Unknown pool id, pool not connected
Thread-35::DEBUG::2014-11-28 15:58:10,992::task::990::TaskManager.Task::(_decref) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:58:10,992::task::925::TaskManager.Task::(_doAbort) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:58:10,992::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:58:10,992::task::595::TaskManager.Task::(_updateState) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:58:10,992::task::550::TaskManager.Task::(__state_aborting) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:58:10,993::task::595::TaskManager.Task::(_updateState) Task=`75b32def-be7a-49ec-9299-3e0b57e87c06`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:58:10,993::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:58:10,993::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:58:10,993::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-35::DEBUG::2014-11-28 15:58:11,006::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:58:11,006::task::595::TaskManager.Task::(_updateState) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:58:11,006::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='644b4e6d-aa9e-4628-b56c-7fb08ddc9584', masterVersion=24239, domainsMap=None, options=None)
Thread-35::DEBUG::2014-11-28 15:58:11,007::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a5072f1f-bbf3-474c-8696-06f63e312ff9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-35::DEBUG::2014-11-28 15:58:11,007::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-35::DEBUG::2014-11-28 15:58:11,007::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-35::DEBUG::2014-11-28 15:58:11,007::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`a5072f1f-bbf3-474c-8696-06f63e312ff9`::Granted request
Thread-35::DEBUG::2014-11-28 15:58:11,008::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`0fd3269a-be01-4f48-901d-b2823280be13`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-35::DEBUG::2014-11-28 15:58:11,008::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-35::DEBUG::2014-11-28 15:58:11,008::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-35::DEBUG::2014-11-28 15:58:11,008::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`0fd3269a-be01-4f48-901d-b2823280be13`::Granted request
Thread-35::INFO::2014-11-28 15:58:11,009::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f3cc80cbf00 to type StoragePoolDiskBackend instance 0x7f3c70245518
Thread-35::INFO::2014-11-28 15:58:11,009::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 (ver = 24239)
Thread-35::DEBUG::2014-11-28 15:58:11,009::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,010::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,010::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,010::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,010::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,011::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,011::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-35::DEBUG::2014-11-28 15:58:11,011::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:58:11,011::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-35::DEBUG::2014-11-28 15:58:11,011::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:58:11,011::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-35::DEBUG::2014-11-28 15:58:11,012::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-35::DEBUG::2014-11-28 15:58:11,023::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::DEBUG::2014-11-28 15:58:11,024::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-35::DEBUG::2014-11-28 15:58:11,075::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-35::DEBUG::2014-11-28 15:58:11,075::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,075::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,076::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,076::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,076::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,076::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:58:11,077::misc::748::SamplingMethod::(__call__) Returning last result
Thread-46::DEBUG::2014-11-28 15:58:11,565::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [63a38186]
Thread-46::DEBUG::2014-11-28 15:58:11,565::task::595::TaskManager.Task::(_updateState) Task=`c06bd34b-9f77-4ad1-84e9-e4aa8f8a1306`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:58:11,566::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-46::DEBUG::2014-11-28 15:58:11,572::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-46::DEBUG::2014-11-28 15:58:11,719::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-46::DEBUG::2014-11-28 15:58:11,719::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-46::INFO::2014-11-28 15:58:11,720::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-46::DEBUG::2014-11-28 15:58:11,720::task::1185::TaskManager.Task::(prepare) Task=`c06bd34b-9f77-4ad1-84e9-e4aa8f8a1306`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-46::DEBUG::2014-11-28 15:58:11,720::task::595::TaskManager.Task::(_updateState) Task=`c06bd34b-9f77-4ad1-84e9-e4aa8f8a1306`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:58:11,720::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:58:11,720::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:58:11,721::task::990::TaskManager.Task::(_decref) Task=`c06bd34b-9f77-4ad1-84e9-e4aa8f8a1306`::ref 0 aborting False
Thread-46::DEBUG::2014-11-28 15:58:11,794::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [c10cf1e]
Thread-46::DEBUG::2014-11-28 15:58:11,794::task::595::TaskManager.Task::(_updateState) Task=`c87dc791-ae66-4358-956d-39e30bee48f4`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:58:11,795::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-46::DEBUG::2014-11-28 15:58:11,801::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-46::DEBUG::2014-11-28 15:58:14,803::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-46::DEBUG::2014-11-28 15:58:14,804::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain, 644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain}
Thread-46::INFO::2014-11-28 15:58:14,804::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-46::DEBUG::2014-11-28 15:58:14,804::task::1185::TaskManager.Task::(prepare) Task=`c87dc791-ae66-4358-956d-39e30bee48f4`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-46::DEBUG::2014-11-28 15:58:14,805::task::595::TaskManager.Task::(_updateState) Task=`c87dc791-ae66-4358-956d-39e30bee48f4`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:58:14,806::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:58:14,807::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:58:14,808::task::990::TaskManager.Task::(_decref) Task=`c87dc791-ae66-4358-956d-39e30bee48f4`::ref 0 aborting False
Thread-46::DEBUG::2014-11-28 15:58:14,815::task::595::TaskManager.Task::(_updateState) Task=`04e8fe39-a1bc-4e3b-95a7-6042b047682b`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:58:14,816::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-46::INFO::2014-11-28 15:58:14,816::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-46::DEBUG::2014-11-28 15:58:14,817::task::1185::TaskManager.Task::(prepare) Task=`04e8fe39-a1bc-4e3b-95a7-6042b047682b`::finished: {}
Thread-46::DEBUG::2014-11-28 15:58:14,818::task::595::TaskManager.Task::(_updateState) Task=`04e8fe39-a1bc-4e3b-95a7-6042b047682b`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:58:14,819::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:58:14,820::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:58:14,821::task::990::TaskManager.Task::(_decref) Task=`04e8fe39-a1bc-4e3b-95a7-6042b047682b`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:58:14,821::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-35::DEBUG::2014-11-28 15:58:14,969::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-35::DEBUG::2014-11-28 15:58:14,972::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-46::DEBUG::2014-11-28 15:58:29,957::task::595::TaskManager.Task::(_updateState) Task=`bf4973ce-1f8e-438c-bf3e-704b9afc2f53`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:58:29,957::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-46::INFO::2014-11-28 15:58:29,957::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-46::DEBUG::2014-11-28 15:58:29,958::task::1185::TaskManager.Task::(prepare) Task=`bf4973ce-1f8e-438c-bf3e-704b9afc2f53`::finished: {}
Thread-46::DEBUG::2014-11-28 15:58:29,958::task::595::TaskManager.Task::(_updateState) Task=`bf4973ce-1f8e-438c-bf3e-704b9afc2f53`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:58:29,958::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:58:29,958::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:58:29,958::task::990::TaskManager.Task::(_decref) Task=`bf4973ce-1f8e-438c-bf3e-704b9afc2f53`::ref 0 aborting False
Thread-46::DEBUG::2014-11-28 15:58:45,040::task::595::TaskManager.Task::(_updateState) Task=`edbac68f-d456-4c25-b3de-e989a8c1d56b`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:58:45,040::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-46::INFO::2014-11-28 15:58:45,041::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-46::DEBUG::2014-11-28 15:58:45,041::task::1185::TaskManager.Task::(prepare) Task=`edbac68f-d456-4c25-b3de-e989a8c1d56b`::finished: {}
Thread-46::DEBUG::2014-11-28 15:58:45,041::task::595::TaskManager.Task::(_updateState) Task=`edbac68f-d456-4c25-b3de-e989a8c1d56b`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:58:45,041::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:58:45,041::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:58:45,041::task::990::TaskManager.Task::(_decref) Task=`edbac68f-d456-4c25-b3de-e989a8c1d56b`::ref 0 aborting False
Thread-46::DEBUG::2014-11-28 15:59:00,123::task::595::TaskManager.Task::(_updateState) Task=`6520e3a9-69d0-44e1-9fcf-9c16317bc0f3`::moving from state init -> state preparing
Thread-46::INFO::2014-11-28 15:59:00,123::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-46::INFO::2014-11-28 15:59:00,124::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-46::DEBUG::2014-11-28 15:59:00,124::task::1185::TaskManager.Task::(prepare) Task=`6520e3a9-69d0-44e1-9fcf-9c16317bc0f3`::finished: {}
Thread-46::DEBUG::2014-11-28 15:59:00,124::task::595::TaskManager.Task::(_updateState) Task=`6520e3a9-69d0-44e1-9fcf-9c16317bc0f3`::moving from state preparing -> state finished
Thread-46::DEBUG::2014-11-28 15:59:00,124::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-46::DEBUG::2014-11-28 15:59:00,124::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-46::DEBUG::2014-11-28 15:59:00,124::task::990::TaskManager.Task::(_decref) Task=`6520e3a9-69d0-44e1-9fcf-9c16317bc0f3`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:59:14,974::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-35::DEBUG::2014-11-28 15:59:14,974::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-35::DEBUG::2014-11-28 15:59:14,974::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-35::DEBUG::2014-11-28 15:59:14,974::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-35::DEBUG::2014-11-28 15:59:14,975::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-35::DEBUG::2014-11-28 15:59:14,975::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-35::DEBUG::2014-11-28 15:59:14,975::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-35::DEBUG::2014-11-28 15:59:14,975::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-35::ERROR::2014-11-28 15:59:14,976::task::866::TaskManager.Task::(_setError) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1357, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 165, in __init__
    self.imageGarbageCollector()
  File "/usr/share/vdsm/storage/fileSD.py", line 607, in imageGarbageCollector
    removedImages = self.oop.glob.glob(removedPattern)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-35::DEBUG::2014-11-28 15:59:14,976::task::885::TaskManager.Task::(_run) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::Task._run: e11805bd-93b4-49b5-8127-b3d42f69b8f3 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 24239, None) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:59:14,976::task::1211::TaskManager.Task::(stop) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:59:14,977::task::990::TaskManager.Task::(_decref) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::ref 1 aborting True
Thread-35::DEBUG::2014-11-28 15:59:14,977::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-35::INFO::2014-11-28 15:59:14,977::task::1168::TaskManager.Task::(prepare) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::aborting: Task is aborted: u'' - code 100
Thread-35::DEBUG::2014-11-28 15:59:14,977::task::1173::TaskManager.Task::(prepare) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::Prepare: aborted: 
Thread-35::DEBUG::2014-11-28 15:59:14,978::task::990::TaskManager.Task::(_decref) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:59:14,978::task::925::TaskManager.Task::(_doAbort) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:59:14,978::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:59:14,978::task::595::TaskManager.Task::(_updateState) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:59:14,978::task::550::TaskManager.Task::(__state_aborting) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:59:14,979::task::595::TaskManager.Task::(_updateState) Task=`e11805bd-93b4-49b5-8127-b3d42f69b8f3`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:59:14,979::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:59:14,979::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:59:14,979::dispatcher::68::Storage.Dispatcher.Protect::(run) 
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
Timeout
Thread-35::DEBUG::2014-11-28 15:59:15,000::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:59:15,000::task::595::TaskManager.Task::(_updateState) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:59:15,000::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', options=None)
Thread-35::ERROR::2014-11-28 15:59:15,001::task::866::TaskManager.Task::(_setError) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 629, in getSpmStatus
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-35::DEBUG::2014-11-28 15:59:15,001::task::885::TaskManager.Task::(_run) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::Task._run: e195b4ac-f942-4096-8eef-5459ac47af5e ('45995288-a7d6-4cbf-8b2f-63fa35642bad',) {} failed - stopping task
Thread-35::DEBUG::2014-11-28 15:59:15,001::task::1211::TaskManager.Task::(stop) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::stopping in state preparing (force False)
Thread-35::DEBUG::2014-11-28 15:59:15,001::task::990::TaskManager.Task::(_decref) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::ref 1 aborting True
Thread-35::INFO::2014-11-28 15:59:15,001::task::1168::TaskManager.Task::(prepare) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-35::DEBUG::2014-11-28 15:59:15,002::task::1173::TaskManager.Task::(prepare) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::Prepare: aborted: Unknown pool id, pool not connected
Thread-35::DEBUG::2014-11-28 15:59:15,002::task::990::TaskManager.Task::(_decref) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::ref 0 aborting True
Thread-35::DEBUG::2014-11-28 15:59:15,002::task::925::TaskManager.Task::(_doAbort) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::Task._doAbort: force False
Thread-35::DEBUG::2014-11-28 15:59:15,002::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:59:15,002::task::595::TaskManager.Task::(_updateState) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::moving from state preparing -> state aborting
Thread-35::DEBUG::2014-11-28 15:59:15,003::task::550::TaskManager.Task::(__state_aborting) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::_aborting: recover policy none
Thread-35::DEBUG::2014-11-28 15:59:15,003::task::595::TaskManager.Task::(_updateState) Task=`e195b4ac-f942-4096-8eef-5459ac47af5e`::moving from state aborting -> state failed
Thread-35::DEBUG::2014-11-28 15:59:15,003::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:59:15,003::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::ERROR::2014-11-28 15:59:15,003::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-35::DEBUG::2014-11-28 15:59:15,208::task::595::TaskManager.Task::(_updateState) Task=`037a0166-79bf-450f-9a0b-c6dd657f2126`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:59:15,208::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-35::INFO::2014-11-28 15:59:15,209::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-35::DEBUG::2014-11-28 15:59:15,209::task::1185::TaskManager.Task::(prepare) Task=`037a0166-79bf-450f-9a0b-c6dd657f2126`::finished: {}
Thread-35::DEBUG::2014-11-28 15:59:15,209::task::595::TaskManager.Task::(_updateState) Task=`037a0166-79bf-450f-9a0b-c6dd657f2126`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:59:15,209::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:59:15,209::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:59:15,210::task::990::TaskManager.Task::(_decref) Task=`037a0166-79bf-450f-9a0b-c6dd657f2126`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:59:15,240::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-35::DEBUG::2014-11-28 15:59:15,241::task::595::TaskManager.Task::(_updateState) Task=`f8eeaa3a-b6e1-42bc-bc49-5f42b27383d5`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:59:15,241::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, remove=False, options=None)
Thread-35::WARNING::2014-11-28 15:59:15,241::hsm::1126::Storage.HSM::(disconnectStoragePool) disconnect sp: 45995288-a7d6-4cbf-8b2f-63fa35642bad failed. Known pools {}
Thread-35::INFO::2014-11-28 15:59:15,241::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None
Thread-35::DEBUG::2014-11-28 15:59:15,241::task::1185::TaskManager.Task::(prepare) Task=`f8eeaa3a-b6e1-42bc-bc49-5f42b27383d5`::finished: None
Thread-35::DEBUG::2014-11-28 15:59:15,241::task::595::TaskManager.Task::(_updateState) Task=`f8eeaa3a-b6e1-42bc-bc49-5f42b27383d5`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:59:15,242::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:59:15,242::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:59:15,242::task::990::TaskManager.Task::(_decref) Task=`f8eeaa3a-b6e1-42bc-bc49-5f42b27383d5`::ref 0 aborting False
Thread-35::DEBUG::2014-11-28 15:59:15,330::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [12cd58e5]
Thread-35::DEBUG::2014-11-28 15:59:15,330::task::595::TaskManager.Task::(_updateState) Task=`b594d334-a4c1-4e8d-9951-12f865aa265e`::moving from state init -> state preparing
Thread-35::INFO::2014-11-28 15:59:15,330::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-35::DEBUG::2014-11-28 15:59:15,331::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-35::ERROR::2014-11-28 15:59:15,344::hsm::2487::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2483, in disconnectStorageServer
    conObj.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
    return self._mountCon.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
    self._mount.umount(True, True)
  File "/usr/share/vdsm/storage/mount.py", line 229, in umount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';umount: /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso: mountpoint not found\n')
Thread-35::DEBUG::2014-11-28 15:59:15,345::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-35::DEBUG::2014-11-28 15:59:15,372::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-35::DEBUG::2014-11-28 15:59:15,399::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-35::ERROR::2014-11-28 15:59:15,411::hsm::2487::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2483, in disconnectStorageServer
    conObj.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
    return self._mountCon.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
    self._mount.umount(True, True)
  File "/usr/share/vdsm/storage/mount.py", line 229, in umount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';umount: /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01: mountpoint not found\n')
Thread-35::DEBUG::2014-11-28 15:59:15,412::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-35::ERROR::2014-11-28 15:59:15,423::hsm::2487::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2483, in disconnectStorageServer
    conObj.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 326, in disconnect
    return self._mountCon.disconnect()
  File "/usr/share/vdsm/storage/storageServer.py", line 233, in disconnect
    self._mount.umount(True, True)
  File "/usr/share/vdsm/storage/mount.py", line 229, in umount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';umount: /rhev/data-center/mnt/172.16.241.98:_mnt_storage: mountpoint not found\n')
Thread-35::DEBUG::2014-11-28 15:59:15,423::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-35::DEBUG::2014-11-28 15:59:15,424::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:59:15,424::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-35::DEBUG::2014-11-28 15:59:15,424::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-35::DEBUG::2014-11-28 15:59:15,424::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-35::DEBUG::2014-11-28 15:59:15,424::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-35::DEBUG::2014-11-28 15:59:15,436::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::DEBUG::2014-11-28 15:59:15,437::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-35::DEBUG::2014-11-28 15:59:15,478::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-35::DEBUG::2014-11-28 15:59:15,478::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,479::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,479::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,479::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,479::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,480::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-35::DEBUG::2014-11-28 15:59:15,480::misc::748::SamplingMethod::(__call__) Returning last result
Thread-35::INFO::2014-11-28 15:59:15,480::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:59:15,480::task::1185::TaskManager.Task::(prepare) Task=`b594d334-a4c1-4e8d-9951-12f865aa265e`::finished: {'statuslist': [{'status': 477, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-35::DEBUG::2014-11-28 15:59:15,481::task::595::TaskManager.Task::(_updateState) Task=`b594d334-a4c1-4e8d-9951-12f865aa265e`::moving from state preparing -> state finished
Thread-35::DEBUG::2014-11-28 15:59:15,481::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-35::DEBUG::2014-11-28 15:59:15,481::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-35::DEBUG::2014-11-28 15:59:15,481::task::990::TaskManager.Task::(_decref) Task=`b594d334-a4c1-4e8d-9951-12f865aa265e`::ref 0 aborting False
MainThread::DEBUG::2014-11-28 16:06:32,612::vdsm::58::vds::(sigtermHandler) Received signal 15
MainThread::INFO::2014-11-28 16:06:33,163::vmChannels::187::vds::(stop) VM channels listener was stopped.
MainThread::INFO::2014-11-28 16:06:33,164::momIF::91::MOM::(stop) Shutting down MOM
MainThread::DEBUG::2014-11-28 16:06:33,164::task::595::TaskManager.Task::(_updateState) Task=`eeb0d9f0-5379-4b1e-8de1-d1c876ec1bad`::moving from state init -> state preparing
MainThread::INFO::2014-11-28 16:06:33,164::logUtils::44::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None)
Thread-11::DEBUG::2014-11-28 16:06:33,165::storageServer::698::ConnectionMonitor::(_monitorConnections) Monitoring stopped
VM Channels Listener::INFO::2014-11-28 16:06:33,168::vmChannels::182::vds::(run) VM channels listener thread has ended.
MainThread::INFO::2014-11-28 16:06:33,598::domainMonitor::125::Storage.DomainMonitor::(close) Stopping domain monitors
MainThread::DEBUG::2014-11-28 16:06:33,598::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks
MainThread::INFO::2014-11-28 16:06:33,598::logUtils::47::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None
MainThread::DEBUG::2014-11-28 16:06:33,598::task::1185::TaskManager.Task::(prepare) Task=`eeb0d9f0-5379-4b1e-8de1-d1c876ec1bad`::finished: None
MainThread::DEBUG::2014-11-28 16:06:33,598::task::595::TaskManager.Task::(_updateState) Task=`eeb0d9f0-5379-4b1e-8de1-d1c876ec1bad`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 16:06:33,598::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 16:06:33,599::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 16:06:33,599::task::990::TaskManager.Task::(_decref) Task=`eeb0d9f0-5379-4b1e-8de1-d1c876ec1bad`::ref 0 aborting False
MainThread::INFO::2014-11-28 16:06:33,599::vdsm::137::vds::(run) VDSM main thread ended. Waiting for 15 other threads...
MainThread::INFO::2014-11-28 16:06:33,599::vdsm::141::vds::(run) <Thread(libvirtEventLoop, started daemon 139898745853696)>
MainThread::INFO::2014-11-28 16:06:33,599::vdsm::141::vds::(run) <GuestManager(GuestManager, started daemon 139897352484608)>
MainThread::INFO::2014-11-28 16:06:33,599::vdsm::141::vds::(run) <PolicyEngine(PolicyEngine, started daemon 139898308761344)>
MainThread::INFO::2014-11-28 16:06:33,600::momIF::91::MOM::(stop) Shutting down MOM
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <MomThread(MOM, started 139897746745088)>
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <WorkerThread(Thread-6, started daemon 139898359117568)>
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <WorkerThread(Thread-10, started daemon 139898325546752)>
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <_MainThread(MainThread, started 139899033438016)>
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <WorkerThread(Thread-9, started daemon 139898333939456)>
MainThread::INFO::2014-11-28 16:06:33,600::vdsm::141::vds::(run) <WorkerThread(Thread-3, started daemon 139898720675584)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-8, started daemon 139898342332160)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-5, started daemon 139898703890176)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-4, started daemon 139898712282880)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-7, started daemon 139898350724864)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-2, started daemon 139898729068288)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <WorkerThread(Thread-1, started daemon 139898737460992)>
MainThread::INFO::2014-11-28 16:06:33,601::vdsm::141::vds::(run) <HostMonitor(HostMonitor, started daemon 139897698064128)>
MainThread::INFO::2014-11-28 16:08:27,963::vdsm::131::vds::(run) (PID: 4846) I am the actual vdsm 4.14.17-0.el7 cti12.ctisandbox.cfcc.edu (3.10.0-123.9.3.el7.x86_64)
MainThread::DEBUG::2014-11-28 16:08:27,995::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-11-28 16:08:27,995::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::WARNING::2014-11-28 16:08:27,998::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
MainThread::DEBUG::2014-11-28 16:08:28,505::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-11-28 16:08:28,547::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
MainThread::DEBUG::2014-11-28 16:08:28,547::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-11-28 16:08:28,566::hsm::434::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-11-28 16:08:28,567::hsm::458::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-11-28 16:08:28,572::hsm::490::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-11-28 16:08:28,572::hsm::491::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-11-28 16:08:28,572::hsm::493::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-11-28 16:08:28,574::hsm::536::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-11-28 16:08:28,574::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
MainThread::INFO::2014-11-28 16:08:28,575::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-11-28 16:08:29,217::misc::740::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-11-28 16:08:29,287::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
storageRefresh::DEBUG::2014-11-28 16:08:29,371::misc::740::SamplingMethod::(__call__) Got in to sampling method
MainThread::DEBUG::2014-11-28 16:08:29,372::task::595::TaskManager.Task::(_updateState) Task=`d92a1ce2-f8ad-4478-bd3d-e643a58832ff`::moving from state init -> state preparing
storageRefresh::DEBUG::2014-11-28 16:08:29,372::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
MainThread::INFO::2014-11-28 16:08:29,372::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x124c560>>)
storageRefresh::DEBUG::2014-11-28 16:08:29,372::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
MainThread::INFO::2014-11-28 16:08:29,373::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
MainThread::DEBUG::2014-11-28 16:08:29,376::task::1185::TaskManager.Task::(prepare) Task=`d92a1ce2-f8ad-4478-bd3d-e643a58832ff`::finished: None
MainThread::DEBUG::2014-11-28 16:08:29,377::task::595::TaskManager.Task::(_updateState) Task=`d92a1ce2-f8ad-4478-bd3d-e643a58832ff`::moving from state preparing -> state finished
MainThread::DEBUG::2014-11-28 16:08:29,377::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
MainThread::DEBUG::2014-11-28 16:08:29,377::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
MainThread::DEBUG::2014-11-28 16:08:29,378::task::990::TaskManager.Task::(_decref) Task=`d92a1ce2-f8ad-4478-bd3d-e643a58832ff`::ref 0 aborting False
MainThread::INFO::2014-11-28 16:08:29,379::momIF::49::MOM::(__init__) Starting up MOM
MainThread::INFO::2014-11-28 16:08:29,388::vmChannels::191::vds::(settimeout) Setting channels' timeout to 30 seconds.
VM Channels Listener::INFO::2014-11-28 16:08:29,390::vmChannels::174::vds::(run) Starting VM channels listener thread.
storageRefresh::DEBUG::2014-11-28 16:08:29,476::misc::748::SamplingMethod::(__call__) Returning last result
clientIFinit::DEBUG::2014-11-28 16:08:29,483::libvirtconnection::145::root::(get) trying to connect libvirt
MainThread::WARNING::2014-11-28 16:08:29,499::clientIF::180::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-11-28 16:08:29,501::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:29,577::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:29,579::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,580::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,581::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,581::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,582::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,583::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,584::misc::748::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-11-28 16:08:29,584::lvm::325::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,588::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:29,615::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:29,616::lvm::350::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,618::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,620::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:29,641::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:29,641::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,642::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:29,660::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:29,660::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-11-28 16:08:29,661::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-11-28 16:08:29,680::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-11-28 16:08:29,682::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::WARNING::2014-11-28 16:08:29,682::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
storageRefresh::DEBUG::2014-11-28 16:08:29,683::hsm::393::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-11-28 16:23:44,157::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getCapabilities with () {}
Thread-13::DEBUG::2014-11-28 16:23:44,195::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-11-28 16:23:44,201::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:23:44,532::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.9.3.el7.x86_64', 'buildtime': 1415304363.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1402359811L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el7', 'buildtime': 1411399056L, 'version': '4.14.17'}, 'qemu-kvm': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.3', 'buildtime': 1412359256L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.2', 'buildtime': 1411427266L, 'version': '1.5.3'}, 'mom': {'release': '1.el7', 'buildtime': 1410463505L, 'version': '0.4.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'gateway': '172.16.0.1', 'netmask': '255.255.0.0', 'stp': 'off', 'bridged': True, 'mtu': '9000', 'ipv6gateway': '::', 'ports': ['em2']}}, 'bridges': {'ovirtmgmt': {'addr': '172.16.0.12', 'cfg': {'DEFROUTE': 'yes', 'IPADDR': '172.16.0.12', 'HOTPLUG': 'no', 'GATEWAY': '172.16.0.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'MTU': '9000', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'mtu': '9000', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '172.16.0.1', 'ports': ['em2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '4C4C4544-0047-4710-8038-B1C04F325031', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:25', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em1', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '8f04642b-5fd9-4fdc-9884-9b4187d3e718', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': 'B8:AC:6F:91:13:25', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'em2': {'netmask': '', 'addr': '', 'hwaddr': 'b8:ac:6f:91:13:26', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '9000', 'HWADDR': 'b8:ac:6f:91:13:26', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::baac:6fff:fe91:1326/64'], 'speed': 1000, 'mtu': '9000'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:b676fcd2199', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '56:6c:8c:7c:39:e7', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15875', 'cpuSpeed': '2393.827', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '0.1406.el7.centos.2.5', 'version': '7', 'name': 'RHEL'}, 'lastClient': '172.16.241.100'}}
Thread-13::DEBUG::2014-11-28 16:23:44,548::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call getHardwareInfo with () {}
Thread-13::DEBUG::2014-11-28 16:23:44,660::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R210', 'systemUUID': '4C4C4544-0047-4710-8038-B1C04F325031', 'systemSerialNumber': '1GG82P1', 'systemManufacturer': 'Dell Inc.'}}
Thread-13::DEBUG::2014-11-28 16:23:44,813::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [489e92bf]
Thread-13::DEBUG::2014-11-28 16:23:44,814::task::595::TaskManager.Task::(_updateState) Task=`c28e5adb-211b-4b28-89b1-b099955607a9`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:23:44,814::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', conList=[{'connection': '172.16.241.100:/var/lib/exports/iso', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0', 'port': ''}, {'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}, {'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}, {'connection': '172.16.225.16:/var/lib/exports/studstore01', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6', 'port': ''}, {'connection': '172.16.241.98:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:23:44,820::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.100:/var/lib/exports/iso /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso' (cwd None)
Thread-13::DEBUG::2014-11-28 16:23:45,115::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso
Thread-13::DEBUG::2014-11-28 16:23:45,117::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('474f0809-26c6-4fca-ab8c-72a2e6dcd912',)
Thread-13::DEBUG::2014-11-28 16:23:45,118::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:23:45,124::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.2:/data/bs02store /rhev/data-center/mnt/172.16.225.2:_data_bs02store' (cwd None)
Thread-13::DEBUG::2014-11-28 16:23:45,154::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 16:23:45,155::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 16:23:45,156::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:23:45,162::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.99:/mnt/storage /rhev/data-center/mnt/172.16.241.99:_mnt_storage' (cwd None)
Thread-13::DEBUG::2014-11-28 16:23:45,193::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-13::DEBUG::2014-11-28 16:23:45,194::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 16:23:45,195::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain, 474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:23:45,201::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.225.16:/var/lib/exports/studstore01 /rhev/data-center/mnt/172.16.225.16:_var_lib_exports_studstore01' (cwd None)
Thread-13::ERROR::2014-11-28 16:25:56,313::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: Connection timed out\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::ERROR::2014-11-28 16:25:56,334::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: Connection timed out\n')
Thread-13::DEBUG::2014-11-28 16:25:56,344::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain, 474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::DEBUG::2014-11-28 16:25:56,351::mount::202::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.241.98:/mnt/storage /rhev/data-center/mnt/172.16.241.98:_mnt_storage' (cwd None)
Thread-13::ERROR::2014-11-28 16:25:56,370::storageServer::209::StorageServer.MountConnection::(connect) Mount failed: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storageServer.py", line 207, in connect
    self._mount.mount(self.options, self._vfsType)
  File "/usr/share/vdsm/storage/mount.py", line 198, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 214, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::ERROR::2014-11-28 16:25:56,371::hsm::2389::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2386, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 320, in connect
    return self._mountCon.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 215, in connect
    raise e
MountError: (32, ';mount.nfs: requested NFS version or transport protocol is not supported\n')
Thread-13::DEBUG::2014-11-28 16:25:56,371::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain, 474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:25:56,371::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:25:56,371::task::1185::TaskManager.Task::(prepare) Task=`c28e5adb-211b-4b28-89b1-b099955607a9`::finished: {'statuslist': [{'status': 0, 'id': '4dc8391a-5b31-4124-9d6c-942694cceae0'}, {'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}, {'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}, {'status': 477, 'id': '90917bf7-0502-4a0b-af39-c03dc91737e6'}, {'status': 477, 'id': 'd2221d67-2521-485b-bbab-f23bdcc95378'}]}
Thread-13::DEBUG::2014-11-28 16:25:56,372::task::595::TaskManager.Task::(_updateState) Task=`c28e5adb-211b-4b28-89b1-b099955607a9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:25:56,372::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:25:56,372::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:25:56,372::task::990::TaskManager.Task::(_decref) Task=`c28e5adb-211b-4b28-89b1-b099955607a9`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:25:58,645::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100]
Thread-13::DEBUG::2014-11-28 16:25:58,646::task::595::TaskManager.Task::(_updateState) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:25:58,646::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='e7eaae86-11ea-4c08-8c11-8b8c54373389', masterVersion=24242, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 16:25:58,646::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d05384f1-abeb-456a-a7d9-26329f6a5616`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:25:58,647::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:25:58,647::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:25:58,647::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`d05384f1-abeb-456a-a7d9-26329f6a5616`::Granted request
Thread-13::DEBUG::2014-11-28 16:25:58,648::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`126ff4fc-0bb3-4252-ba9b-24a759857c6c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:25:58,648::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:25:58,648::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:25:58,648::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`126ff4fc-0bb3-4252-ba9b-24a759857c6c`::Granted request
Thread-13::INFO::2014-11-28 16:25:58,649::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f4baec2bf00 to type StoragePoolDiskBackend instance 0x7f4b5c0c74d0
Thread-13::INFO::2014-11-28 16:25:58,649::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: e7eaae86-11ea-4c08-8c11-8b8c54373389 (ver = 24242)
Thread-13::DEBUG::2014-11-28 16:25:58,649::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,649::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,650::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,650::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,650::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,650::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,651::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 16:25:58,651::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:25:58,651::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 16:25:58,651::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:25:58,651::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 16:25:58,652::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 16:25:58,664::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:25:58,665::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 16:25:58,731::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:25:58,731::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,732::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,732::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,732::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,732::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,733::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:25:58,733::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:25:58,896::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:25:59,042::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 16:25:59,045::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24241', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f712eebad6dfcc88f8d75e665220ecff6e2c79e3']
Thread-13::DEBUG::2014-11-28 16:25:59,047::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::DEBUG::2014-11-28 16:25:59,047::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_imageNS'
Thread-13::DEBUG::2014-11-28 16:25:59,047::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e7eaae86-11ea-4c08-8c11-8b8c54373389_volumeNS'
Thread-13::ERROR::2014-11-28 16:25:59,048::spbackends::271::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain e7eaae86-11ea-4c08-8c11-8b8c54373389 does not have expected version 24242 it is version 24241
Thread-13::DEBUG::2014-11-28 16:25:59,048::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 16:25:59,048::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 16:25:59,049::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:25:59,049::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 16:25:59,049::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-11-28 16:25:59,049::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 16:25:59,049::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:25:59,050::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::ERROR::2014-11-28 16:25:59,050::task::866::TaskManager.Task::(_setError) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1020, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1091, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 630, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1153, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1373, in setMasterDomain
    self._backend.validateMasterDomainVersion(domain, masterVersion)
  File "/usr/share/vdsm/storage/spbackends.py", line 272, in validateMasterDomainVersion
    raise se.StoragePoolWrongMaster(self.spUUID, masterDomain.sdUUID)
StoragePoolWrongMaster: Wrong Master domain or its version: 'SD=e7eaae86-11ea-4c08-8c11-8b8c54373389, pool=45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 16:25:59,086::task::885::TaskManager.Task::(_run) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::Task._run: 83c26a6f-d33a-454c-b395-d66fb1143b03 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 1, 'e7eaae86-11ea-4c08-8c11-8b8c54373389', 24242, None) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 16:25:59,087::task::1211::TaskManager.Task::(stop) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 16:25:59,087::task::990::TaskManager.Task::(_decref) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::ref 1 aborting True
Thread-13::INFO::2014-11-28 16:25:59,087::task::1168::TaskManager.Task::(prepare) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324
Thread-13::DEBUG::2014-11-28 16:25:59,087::task::1173::TaskManager.Task::(prepare) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::Prepare: aborted: Wrong Master domain or its version
Thread-13::DEBUG::2014-11-28 16:25:59,088::task::990::TaskManager.Task::(_decref) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 16:25:59,088::task::925::TaskManager.Task::(_doAbort) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 16:25:59,088::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:25:59,088::task::595::TaskManager.Task::(_updateState) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 16:25:59,088::task::550::TaskManager.Task::(__state_aborting) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 16:25:59,088::task::595::TaskManager.Task::(_updateState) Task=`83c26a6f-d33a-454c-b395-d66fb1143b03`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 16:25:59,089::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:25:59,089::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-11-28 16:25:59,089::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Wrong Master domain or its version: 'SD=e7eaae86-11ea-4c08-8c11-8b8c54373389, pool=45995288-a7d6-4cbf-8b2f-63fa35642bad'", 'code': 324}}
Thread-13::DEBUG::2014-11-28 16:25:59,107::task::595::TaskManager.Task::(_updateState) Task=`6d31ec0e-65ce-48c7-9e31-767122589590`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:25:59,108::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:25:59,108::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:25:59,108::task::1185::TaskManager.Task::(prepare) Task=`6d31ec0e-65ce-48c7-9e31-767122589590`::finished: {}
Thread-13::DEBUG::2014-11-28 16:25:59,108::task::595::TaskManager.Task::(_updateState) Task=`6d31ec0e-65ce-48c7-9e31-767122589590`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:25:59,108::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:25:59,108::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:25:59,109::task::990::TaskManager.Task::(_decref) Task=`6d31ec0e-65ce-48c7-9e31-767122589590`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:25:59,123::BindingXMLRPC::1067::vds::(wrapper) client [172.16.241.100]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': False},) {} flowID [489e92bf]
Thread-13::DEBUG::2014-11-28 16:25:59,193::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-13::DEBUG::2014-11-28 16:26:02,258::task::595::TaskManager.Task::(_updateState) Task=`ad89c549-9032-4b99-a56d-ce98ac9b6ef5`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:26:02,258::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:26:02,258::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:26:02,258::task::1185::TaskManager.Task::(prepare) Task=`ad89c549-9032-4b99-a56d-ce98ac9b6ef5`::finished: {}
Thread-13::DEBUG::2014-11-28 16:26:02,259::task::595::TaskManager.Task::(_updateState) Task=`ad89c549-9032-4b99-a56d-ce98ac9b6ef5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:26:02,259::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:26:02,259::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:26:02,259::task::990::TaskManager.Task::(_decref) Task=`ad89c549-9032-4b99-a56d-ce98ac9b6ef5`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:26:17,351::task::595::TaskManager.Task::(_updateState) Task=`c9e4eb43-5287-4c37-a476-1bbaf82ce067`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:26:17,352::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:26:17,352::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:26:17,352::task::1185::TaskManager.Task::(prepare) Task=`c9e4eb43-5287-4c37-a476-1bbaf82ce067`::finished: {}
Thread-13::DEBUG::2014-11-28 16:26:17,352::task::595::TaskManager.Task::(_updateState) Task=`c9e4eb43-5287-4c37-a476-1bbaf82ce067`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:26:17,352::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:26:17,352::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:26:17,353::task::990::TaskManager.Task::(_decref) Task=`c9e4eb43-5287-4c37-a476-1bbaf82ce067`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:26:32,430::task::595::TaskManager.Task::(_updateState) Task=`2d6b7e44-a52e-449d-a09f-bb7ade629e15`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:26:32,431::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:26:32,431::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:26:32,431::task::1185::TaskManager.Task::(prepare) Task=`2d6b7e44-a52e-449d-a09f-bb7ade629e15`::finished: {}
Thread-13::DEBUG::2014-11-28 16:26:32,431::task::595::TaskManager.Task::(_updateState) Task=`2d6b7e44-a52e-449d-a09f-bb7ade629e15`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:26:32,431::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:26:32,432::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:26:32,432::task::990::TaskManager.Task::(_decref) Task=`2d6b7e44-a52e-449d-a09f-bb7ade629e15`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:26:47,509::task::595::TaskManager.Task::(_updateState) Task=`59c3d580-c4e5-436c-81b6-42f41fdab01d`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:26:47,509::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:26:47,509::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:26:47,510::task::1185::TaskManager.Task::(prepare) Task=`59c3d580-c4e5-436c-81b6-42f41fdab01d`::finished: {}
Thread-13::DEBUG::2014-11-28 16:26:47,510::task::595::TaskManager.Task::(_updateState) Task=`59c3d580-c4e5-436c-81b6-42f41fdab01d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:26:47,510::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:26:47,510::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:26:47,510::task::990::TaskManager.Task::(_decref) Task=`59c3d580-c4e5-436c-81b6-42f41fdab01d`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:27:02,587::task::595::TaskManager.Task::(_updateState) Task=`e4f4b638-9d31-4041-96d4-9e5e25608dab`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:02,588::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:27:02,588::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:27:02,588::task::1185::TaskManager.Task::(prepare) Task=`e4f4b638-9d31-4041-96d4-9e5e25608dab`::finished: {}
Thread-13::DEBUG::2014-11-28 16:27:02,588::task::595::TaskManager.Task::(_updateState) Task=`e4f4b638-9d31-4041-96d4-9e5e25608dab`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:02,588::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:02,588::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:02,589::task::990::TaskManager.Task::(_decref) Task=`e4f4b638-9d31-4041-96d4-9e5e25608dab`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:27:11,334::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [57baeb84]
Thread-13::DEBUG::2014-11-28 16:27:11,335::task::595::TaskManager.Task::(_updateState) Task=`08b9d892-21df-405e-9de5-63517cb7e8cf`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:11,335::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.241.99:/mnt/storage', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:27:11,343::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.241.99:_mnt_storage
Thread-13::DEBUG::2014-11-28 16:27:11,345::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('644b4e6d-aa9e-4628-b56c-7fb08ddc9584',)
Thread-13::DEBUG::2014-11-28 16:27:11,345::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain, 474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:27:11,345::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 16:27:11,345::task::1185::TaskManager.Task::(prepare) Task=`08b9d892-21df-405e-9de5-63517cb7e8cf`::finished: {'statuslist': [{'status': 0, 'id': '80a9704b-0ceb-4bd4-abd1-c9f45ab9770e'}]}
Thread-13::DEBUG::2014-11-28 16:27:11,345::task::595::TaskManager.Task::(_updateState) Task=`08b9d892-21df-405e-9de5-63517cb7e8cf`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:11,345::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:11,346::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:11,346::task::990::TaskManager.Task::(_decref) Task=`08b9d892-21df-405e-9de5-63517cb7e8cf`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:27:17,664::task::595::TaskManager.Task::(_updateState) Task=`d6aea4b9-8ae8-4f2a-84a8-810b603544f9`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:17,664::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:27:17,664::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-13::DEBUG::2014-11-28 16:27:17,664::task::1185::TaskManager.Task::(prepare) Task=`d6aea4b9-8ae8-4f2a-84a8-810b603544f9`::finished: {}
Thread-13::DEBUG::2014-11-28 16:27:17,664::task::595::TaskManager.Task::(_updateState) Task=`d6aea4b9-8ae8-4f2a-84a8-810b603544f9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:17,665::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:17,665::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:17,665::task::990::TaskManager.Task::(_decref) Task=`d6aea4b9-8ae8-4f2a-84a8-810b603544f9`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:27:20,925::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3114954]
Thread-13::DEBUG::2014-11-28 16:27:20,925::task::595::TaskManager.Task::(_updateState) Task=`40d7462d-ccec-4b32-9b8a-36f35675c56d`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:20,926::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.225.2:/data/bs02store', 'iqn': '', 'user': '', 'tpgt': '1', 'password': '******', 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430', 'port': ''}], options=None)
Thread-13::DEBUG::2014-11-28 16:27:20,932::hsm::2328::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/172.16.225.2:_data_bs02store
Thread-13::DEBUG::2014-11-28 16:27:20,933::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e7eaae86-11ea-4c08-8c11-8b8c54373389',)
Thread-13::DEBUG::2014-11-28 16:27:20,934::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {644b4e6d-aa9e-4628-b56c-7fb08ddc9584: storage.nfsSD.findDomain, 474f0809-26c6-4fca-ab8c-72a2e6dcd912: storage.nfsSD.findDomain, e7eaae86-11ea-4c08-8c11-8b8c54373389: storage.nfsSD.findDomain}
Thread-13::INFO::2014-11-28 16:27:20,934::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 16:27:20,934::task::1185::TaskManager.Task::(prepare) Task=`40d7462d-ccec-4b32-9b8a-36f35675c56d`::finished: {'statuslist': [{'status': 0, 'id': '6e5aca58-0ab2-45d8-bf02-cb1a3bf2b430'}]}
Thread-13::DEBUG::2014-11-28 16:27:20,934::task::595::TaskManager.Task::(_updateState) Task=`40d7462d-ccec-4b32-9b8a-36f35675c56d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:20,934::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:20,935::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:20,935::task::990::TaskManager.Task::(_decref) Task=`40d7462d-ccec-4b32-9b8a-36f35675c56d`::ref 0 aborting False
Thread-13::DEBUG::2014-11-28 16:27:20,946::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3114954]
Thread-13::DEBUG::2014-11-28 16:27:20,946::task::595::TaskManager.Task::(_updateState) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:20,946::logUtils::44::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', msdUUID='e7eaae86-11ea-4c08-8c11-8b8c54373389', masterVersion=24243, options=None)
Thread-13::DEBUG::2014-11-28 16:27:20,947::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`a3effc9e-b100-418a-b1e9-66a1375d7fa8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '870' at 'refreshStoragePool'
Thread-13::DEBUG::2014-11-28 16:27:20,947::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'shared'
Thread-13::DEBUG::2014-11-28 16:27:20,947::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'shared' (1 active user)
Thread-13::DEBUG::2014-11-28 16:27:20,947::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`a3effc9e-b100-418a-b1e9-66a1375d7fa8`::Granted request
Thread-13::DEBUG::2014-11-28 16:27:20,948::task::827::TaskManager.Task::(resourceAcquired) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::_resourcesAcquired: Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad (shared)
Thread-13::DEBUG::2014-11-28 16:27:20,948::task::990::TaskManager.Task::(_decref) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::ref 1 aborting False
Thread-13::ERROR::2014-11-28 16:27:20,948::task::866::TaskManager.Task::(_setError) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 877, in refreshStoragePool
    self.validateNotSPM(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 319, in validateNotSPM
    pool = self.getPool(spUUID)
  File "/usr/share/vdsm/storage/hsm.py", line 326, in getPool
    raise se.StoragePoolUnknown(spUUID)
StoragePoolUnknown: Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)
Thread-13::DEBUG::2014-11-28 16:27:20,949::task::885::TaskManager.Task::(_run) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::Task._run: 78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6 ('45995288-a7d6-4cbf-8b2f-63fa35642bad', 'e7eaae86-11ea-4c08-8c11-8b8c54373389', 24243) {} failed - stopping task
Thread-13::DEBUG::2014-11-28 16:27:20,949::task::1211::TaskManager.Task::(stop) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-11-28 16:27:20,949::task::990::TaskManager.Task::(_decref) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::ref 1 aborting True
Thread-13::INFO::2014-11-28 16:27:20,949::task::1168::TaskManager.Task::(prepare) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309
Thread-13::DEBUG::2014-11-28 16:27:20,949::task::1173::TaskManager.Task::(prepare) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::Prepare: aborted: Unknown pool id, pool not connected
Thread-13::DEBUG::2014-11-28 16:27:20,949::task::990::TaskManager.Task::(_decref) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::ref 0 aborting True
Thread-13::DEBUG::2014-11-28 16:27:20,950::task::925::TaskManager.Task::(_doAbort) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::Task._doAbort: force False
Thread-13::DEBUG::2014-11-28 16:27:20,950::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:20,950::task::595::TaskManager.Task::(_updateState) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-11-28 16:27:20,950::task::550::TaskManager.Task::(__state_aborting) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::_aborting: recover policy none
Thread-13::DEBUG::2014-11-28 16:27:20,950::task::595::TaskManager.Task::(_updateState) Task=`78081fd7-5c2e-4e83-b87d-f1a35ffa3cb6`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-11-28 16:27:20,951::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad': < ResourceRef 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', isValid: 'True' obj: 'None'>}
Thread-13::DEBUG::2014-11-28 16:27:20,951::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:20,951::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 16:27:20,951::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 16:27:20,951::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:27:20,952::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::ERROR::2014-11-28 16:27:20,952::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('45995288-a7d6-4cbf-8b2f-63fa35642bad',)", 'code': 309}}
Thread-13::DEBUG::2014-11-28 16:27:20,971::BindingXMLRPC::251::vds::(wrapper) client [172.16.241.100] flowID [3114954]
Thread-13::DEBUG::2014-11-28 16:27:20,971::task::595::TaskManager.Task::(_updateState) Task=`4bd2ad0e-bded-4efc-b28d-52775683b2e7`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:20,971::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='45995288-a7d6-4cbf-8b2f-63fa35642bad', hostID=1, msdUUID='e7eaae86-11ea-4c08-8c11-8b8c54373389', masterVersion=24243, domainsMap=None, options=None)
Thread-13::DEBUG::2014-11-28 16:27:20,972::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ee10f4f1-2d9b-4122-a5b3-594f86656d32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1018' at 'connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:27:20,972::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:27:20,972::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:27:20,972::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ee10f4f1-2d9b-4122-a5b3-594f86656d32`::Granted request
Thread-13::DEBUG::2014-11-28 16:27:20,973::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`038d96eb-3fcf-4d65-94a0-f6fd459b7300`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1066' at '_connectStoragePool'
Thread-13::DEBUG::2014-11-28 16:27:20,973::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' for lock type 'exclusive'
Thread-13::DEBUG::2014-11-28 16:27:20,973::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-11-28 16:27:20,974::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad`ReqID=`038d96eb-3fcf-4d65-94a0-f6fd459b7300`::Granted request
Thread-13::INFO::2014-11-28 16:27:20,974::sp::133::Storage.StoragePool::(setBackend) updating pool 45995288-a7d6-4cbf-8b2f-63fa35642bad backend from type NoneType instance 0x7f4baec2bf00 to type StoragePoolDiskBackend instance 0x7f4b5c194d40
Thread-13::INFO::2014-11-28 16:27:20,974::sp::623::Storage.StoragePool::(connect) Connect host #1 to the storage pool 45995288-a7d6-4cbf-8b2f-63fa35642bad with master domain: e7eaae86-11ea-4c08-8c11-8b8c54373389 (ver = 24243)
Thread-13::DEBUG::2014-11-28 16:27:20,975::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,975::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,975::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,975::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,975::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,976::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:20,976::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-11-28 16:27:20,976::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:27:20,976::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-11-28 16:27:20,976::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:27:20,977::iscsi::407::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-11-28 16:27:20,977::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:20,993::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:27:20,994::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:21,043::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:27:21,043::lvm::497::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,044::lvm::499::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,044::lvm::508::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,044::lvm::510::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,044::lvm::528::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,045::lvm::530::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-11-28 16:27:21,045::misc::748::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-11-28 16:27:21,055::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:27:21,057::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 16:27:21,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24243', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=515064a2c2060cc7d170e43f78b8516364d2923c']
Thread-13::DEBUG::2014-11-28 16:27:21,061::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::INFO::2014-11-28 16:27:21,061::sd::383::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace e7eaae86-11ea-4c08-8c11-8b8c54373389_imageNS already registered
Thread-13::INFO::2014-11-28 16:27:21,061::sd::391::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace e7eaae86-11ea-4c08-8c11-8b8c54373389_volumeNS already registered
Thread-13::DEBUG::2014-11-28 16:27:21,062::sp::1375::Storage.StoragePool::(setMasterDomain) Master domain e7eaae86-11ea-4c08-8c11-8b8c54373389 verified, version 24243
Thread-13::DEBUG::2014-11-28 16:27:21,062::misc::738::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
Thread-13::DEBUG::2014-11-28 16:27:21,062::misc::740::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-11-28 16:27:21,064::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24243', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=515064a2c2060cc7d170e43f78b8516364d2923c']
Thread-13::INFO::2014-11-28 16:27:21,064::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-20::DEBUG::2014-11-28 16:27:21,065::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-13::DEBUG::2014-11-28 16:27:21,065::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `1c4abda0-906b-4faf-a2e3-b6683729cc2f`
Thread-13::INFO::2014-11-28 16:27:21,066::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-20::ERROR::2014-11-28 16:27:21,066::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-13::DEBUG::2014-11-28 16:27:21,067::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `54b2d645-fc18-4784-99eb-0e17e296eee7`
Thread-20::ERROR::2014-11-28 16:27:21,067::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-21::DEBUG::2014-11-28 16:27:21,067::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::INFO::2014-11-28 16:27:21,067::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-20::DEBUG::2014-11-28 16:27:21,068::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-21::ERROR::2014-11-28 16:27:21,069::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::DEBUG::2014-11-28 16:27:21,069::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `644b4e6d-aa9e-4628-b56c-7fb08ddc9584`
Thread-22::DEBUG::2014-11-28 16:27:21,070::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-21::ERROR::2014-11-28 16:27:21,070::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-13::INFO::2014-11-28 16:27:21,070::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-21::DEBUG::2014-11-28 16:27:21,073::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-13::DEBUG::2014-11-28 16:27:21,074::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `e7eaae86-11ea-4c08-8c11-8b8c54373389`
Thread-23::DEBUG::2014-11-28 16:27:21,075::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::INFO::2014-11-28 16:27:21,076::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-13::DEBUG::2014-11-28 16:27:21,080::sp::1404::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `45995288-a7d6-4cbf-8b2f-63fa35642bad` started monitoring domain `474f0809-26c6-4fca-ab8c-72a2e6dcd912`
Thread-27::DEBUG::2014-11-28 16:27:21,081::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-13::DEBUG::2014-11-28 16:27:21,083::misc::748::SamplingMethod::(__call__) Returning last result
Thread-20::DEBUG::2014-11-28 16:27:21,087::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-21::DEBUG::2014-11-28 16:27:21,101::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:21,114::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=BS02Store', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=24243', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.225.2:/data/bs02store', 'ROLE=Master', 'SDUUID=e7eaae86-11ea-4c08-8c11-8b8c54373389', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=515064a2c2060cc7d170e43f78b8516364d2923c']
Thread-13::DEBUG::2014-11-28 16:27:21,121::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-20::DEBUG::2014-11-28 16:27:21,129::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-21::DEBUG::2014-11-28 16:27:21,132::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-20::WARNING::2014-11-28 16:27:21,137::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-21::WARNING::2014-11-28 16:27:21,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-13::DEBUG::2014-11-28 16:27:21,139::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f 54b2d645-fc18-4784-99eb-0e17e296eee7 644b4e6d-aa9e-4628-b56c-7fb08ddc9584 474f0809-26c6-4fca-ab8c-72a2e6dcd912' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:21,176::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n  Volume group "644b4e6d-aa9e-4628-b56c-7fb08ddc9584" not found\n  Skipping volume group 644b4e6d-aa9e-4628-b56c-7fb08ddc9584\n  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found\n  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912\n'; <rc> = 5
Thread-13::WARNING::2014-11-28 16:27:21,186::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7', '  Volume group "644b4e6d-aa9e-4628-b56c-7fb08ddc9584" not found', '  Skipping volume group 644b4e6d-aa9e-4628-b56c-7fb08ddc9584', '  Volume group "474f0809-26c6-4fca-ab8c-72a2e6dcd912" not found', '  Skipping volume group 474f0809-26c6-4fca-ab8c-72a2e6dcd912']
Thread-13::DEBUG::2014-11-28 16:27:21,190::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-23::DEBUG::2014-11-28 16:27:21,210::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-11-28 16:27:21,222::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-27::DEBUG::2014-11-28 16:27:21,246::fileSD::150::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-23::DEBUG::2014-11-28 16:27:21,277::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n641 bytes (641 B) copied, 0.0382107 s, 16.8 kB/s\n'; <rc> = 0
Thread-13::INFO::2014-11-28 16:27:21,281::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.241.99:_mnt_storage/644b4e6d-aa9e-4628-b56c-7fb08ddc9584 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/644b4e6d-aa9e-4628-b56c-7fb08ddc9584
Thread-13::INFO::2014-11-28 16:27:21,281::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/474f0809-26c6-4fca-ab8c-72a2e6dcd912
Thread-13::INFO::2014-11-28 16:27:21,325::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-23::DEBUG::2014-11-28 16:27:21,331::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e7eaae86-11ea-4c08-8c11-8b8c54373389 changed its status to Valid
Thread-23::INFO::2014-11-28 16:27:21,332::clusterlock::184::SANLock::(acquireHostId) Acquiring host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 (id: 1)
Thread-40::DEBUG::2014-11-28 16:27:21,332::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-40::DEBUG::2014-11-28 16:27:21,332::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-40::DEBUG::2014-11-28 16:27:21,333::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-40::DEBUG::2014-11-28 16:27:21,333::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-42::INFO::2014-11-28 16:27:21,337::clientIF::125::vds::(contEIOVms) vmContainerLock acquired
Thread-23::DEBUG::2014-11-28 16:27:21,354::clusterlock::202::SANLock::(acquireHostId) Host id for domain e7eaae86-11ea-4c08-8c11-8b8c54373389 successfully acquired (id: 1)
Thread-20::ERROR::2014-11-28 16:27:21,355::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-13::INFO::2014-11-28 16:27:21,359::sp::1113::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389 to /rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd
Thread-21::ERROR::2014-11-28 16:27:21,360::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 54b2d645-fc18-4784-99eb-0e17e296eee7 not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-20::ERROR::2014-11-28 16:27:21,369::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('1c4abda0-906b-4faf-a2e3-b6683729cc2f',)
Thread-21::ERROR::2014-11-28 16:27:21,369::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 54b2d645-fc18-4784-99eb-0e17e296eee7 monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('54b2d645-fc18-4784-99eb-0e17e296eee7',)
Thread-20::DEBUG::2014-11-28 16:27:21,377::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f changed its status to Invalid
Thread-21::DEBUG::2014-11-28 16:27:21,377::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 54b2d645-fc18-4784-99eb-0e17e296eee7 changed its status to Invalid
Thread-43::DEBUG::2014-11-28 16:27:21,378::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-44::DEBUG::2014-11-28 16:27:21,379::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-43::DEBUG::2014-11-28 16:27:21,379::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-43::DEBUG::2014-11-28 16:27:21,380::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-44::DEBUG::2014-11-28 16:27:21,380::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-43::DEBUG::2014-11-28 16:27:21,380::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-44::DEBUG::2014-11-28 16:27:21,381::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-44::DEBUG::2014-11-28 16:27:21,382::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-22::DEBUG::2014-11-28 16:27:21,391::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-11-28 16:27:21,393::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0
Thread-22::DEBUG::2014-11-28 16:27:21,396::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=vmdstore', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=699', 'POOL_DESCRIPTION=CTI', 'POOL_DOMAINS=1c4abda0-906b-4faf-a2e3-b6683729cc2f:Active,54b2d645-fc18-4784-99eb-0e17e296eee7:Active,644b4e6d-aa9e-4628-b56c-7fb08ddc9584:Active,e7eaae86-11ea-4c08-8c11-8b8c54373389:Active,474f0809-26c6-4fca-ab8c-72a2e6dcd912:Active', 'POOL_SPM_ID=8', 'POOL_SPM_LVER=2', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.99:/mnt/storage', 'ROLE=Master', 'SDUUID=644b4e6d-aa9e-4628-b56c-7fb08ddc9584', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8cac01169e729bfabfe170441c1ce2bb00a007f9']
Thread-13::DEBUG::2014-11-28 16:27:21,397::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd if=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/outbox iflag=direct,fullblock bs=512 count=8 skip=8' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:21,415::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.0117246 s, 349 kB/s\n'; <rc> = 0
Thread-13::INFO::2014-11-28 16:27:21,416::storage_mailbox::385::Storage.MailBox.HsmMailMonitor::(_sendMail) HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=512', 'seek=8']
Thread-13::DEBUG::2014-11-28 16:27:21,416::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) '/usr/bin/dd of=/rhev/data-center/45995288-a7d6-4cbf-8b2f-63fa35642bad/mastersd/dom_md/inbox iflag=fullblock oflag=direct conv=notrunc bs=512 seek=8' (cwd None)
Thread-27::DEBUG::2014-11-28 16:27:21,447::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-27::DEBUG::2014-11-28 16:27:21,450::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=45995288-a7d6-4cbf-8b2f-63fa35642bad', 'REMOTE_PATH=172.16.241.100:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=474f0809-26c6-4fca-ab8c-72a2e6dcd912', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=80de07a9b66165f2715b4747ae39d51945e0db7f']
Thread-27::DEBUG::2014-11-28 16:27:21,453::fileSD::609::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-27::DEBUG::2014-11-28 16:27:21,453::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '474f0809-26c6-4fca-ab8c-72a2e6dcd912_imageNS'
Thread-27::DEBUG::2014-11-28 16:27:21,453::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '474f0809-26c6-4fca-ab8c-72a2e6dcd912_volumeNS'
Thread-27::DEBUG::2014-11-28 16:27:21,461::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-11-28 16:27:21,467::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd) SUCCESS: <err> = '8+0 records in\n8+0 records out\n4096 bytes (4.1 kB) copied, 0.0424882 s, 96.4 kB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:27:21,469::storage_mailbox::207::Storage.Mailbox.HSM::(__init__) HSM_MailboxMonitor created for pool 45995288-a7d6-4cbf-8b2f-63fa35642bad
Thread-27::DEBUG::2014-11-28 16:27:21,469::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000311053 s, 1.2 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:27:21,469::sp::465::Storage.StoragePool::(__createMailboxMonitor) HSM mailbox ready for pool 45995288-a7d6-4cbf-8b2f-63fa35642bad on master domain e7eaae86-11ea-4c08-8c11-8b8c54373389
Thread-13::DEBUG::2014-11-28 16:27:21,470::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad'
Thread-13::DEBUG::2014-11-28 16:27:21,471::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' (0 active users)
Thread-13::DEBUG::2014-11-28 16:27:21,471::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-11-28 16:27:21,472::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45995288-a7d6-4cbf-8b2f-63fa35642bad', Clearing records.
Thread-13::DEBUG::2014-11-28 16:27:21,473::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-27::DEBUG::2014-11-28 16:27:21,473::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 474f0809-26c6-4fca-ab8c-72a2e6dcd912 changed its status to Valid
Thread-13::DEBUG::2014-11-28 16:27:21,474::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-11-28 16:27:21,475::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-55::DEBUG::2014-11-28 16:27:21,475::misc::860::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-13::DEBUG::2014-11-28 16:27:21,476::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-55::DEBUG::2014-11-28 16:27:21,476::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `_onDomainStateChange`
Thread-13::INFO::2014-11-28 16:27:21,477::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-13::DEBUG::2014-11-28 16:27:21,477::task::1185::TaskManager.Task::(prepare) Task=`4bd2ad0e-bded-4efc-b28d-52775683b2e7`::finished: True
Thread-55::DEBUG::2014-11-28 16:27:21,478::misc::870::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-13::DEBUG::2014-11-28 16:27:21,479::task::595::TaskManager.Task::(_updateState) Task=`4bd2ad0e-bded-4efc-b28d-52775683b2e7`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:21,480::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-55::DEBUG::2014-11-28 16:27:21,480::misc::880::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-57::INFO::2014-11-28 16:27:21,481::clientIF::125::vds::(contEIOVms) vmContainerLock acquired
Thread-13::DEBUG::2014-11-28 16:27:21,481::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:21,482::task::990::TaskManager.Task::(_decref) Task=`4bd2ad0e-bded-4efc-b28d-52775683b2e7`::ref 0 aborting False
Thread-23::DEBUG::2014-11-28 16:27:31,362::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-23::DEBUG::2014-11-28 16:27:31,368::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n641 bytes (641 B) copied, 0.000238041 s, 2.7 MB/s\n'; <rc> = 0
Thread-20::ERROR::2014-11-28 16:27:31,378::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-20::ERROR::2014-11-28 16:27:31,378::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 1c4abda0-906b-4faf-a2e3-b6683729cc2f
Thread-20::DEBUG::2014-11-28 16:27:31,378::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-21::ERROR::2014-11-28 16:27:31,379::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-21::ERROR::2014-11-28 16:27:31,379::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 54b2d645-fc18-4784-99eb-0e17e296eee7
Thread-21::DEBUG::2014-11-28 16:27:31,380::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-20::DEBUG::2014-11-28 16:27:31,382::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 1c4abda0-906b-4faf-a2e3-b6683729cc2f' (cwd None)
Thread-21::DEBUG::2014-11-28 16:27:31,387::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 54b2d645-fc18-4784-99eb-0e17e296eee7' (cwd None)
Thread-20::DEBUG::2014-11-28 16:27:31,405::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found\n  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f\n'; <rc> = 5
Thread-20::WARNING::2014-11-28 16:27:31,406::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "1c4abda0-906b-4faf-a2e3-b6683729cc2f" not found', '  Skipping volume group 1c4abda0-906b-4faf-a2e3-b6683729cc2f']
Thread-21::DEBUG::2014-11-28 16:27:31,407::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found\n  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7\n'; <rc> = 5
Thread-21::WARNING::2014-11-28 16:27:31,409::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "54b2d645-fc18-4784-99eb-0e17e296eee7" not found', '  Skipping volume group 54b2d645-fc18-4784-99eb-0e17e296eee7']
Thread-21::DEBUG::2014-11-28 16:27:31,411::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-27::DEBUG::2014-11-28 16:27:31,483::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-11-28 16:27:31,489::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000271464 s, 1.3 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:27:32,759::task::595::TaskManager.Task::(_updateState) Task=`c092d258-47de-4e35-a622-abe756ce2f31`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:32,760::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:27:32,760::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.7', 'valid': True}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000238041', 'lastCheck': '1.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000271464', 'lastCheck': '1.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:27:32,760::task::1185::TaskManager.Task::(prepare) Task=`c092d258-47de-4e35-a622-abe756ce2f31`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.4', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.4', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.7', 'valid': True}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000238041', 'lastCheck': '1.4', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000271464', 'lastCheck': '1.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:27:32,761::task::595::TaskManager.Task::(_updateState) Task=`c092d258-47de-4e35-a622-abe756ce2f31`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:32,761::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:32,761::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:32,761::task::990::TaskManager.Task::(_decref) Task=`c092d258-47de-4e35-a622-abe756ce2f31`::ref 0 aborting False
Thread-23::DEBUG::2014-11-28 16:27:41,379::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-23::DEBUG::2014-11-28 16:27:41,384::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n641 bytes (641 B) copied, 0.000174126 s, 3.7 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-11-28 16:27:41,498::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-11-28 16:27:41,504::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000338225 s, 1.1 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-11-28 16:27:47,845::task::595::TaskManager.Task::(_updateState) Task=`b1c05d13-b7f8-4b2b-a11e-b9bf497e3028`::moving from state init -> state preparing
Thread-13::INFO::2014-11-28 16:27:47,845::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-11-28 16:27:47,846::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.8', 'valid': True}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000174126', 'lastCheck': '6.5', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338225', 'lastCheck': '6.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:27:47,846::task::1185::TaskManager.Task::(prepare) Task=`b1c05d13-b7f8-4b2b-a11e-b9bf497e3028`::finished: {'1c4abda0-906b-4faf-a2e3-b6683729cc2f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.5', 'valid': False}, '54b2d645-fc18-4784-99eb-0e17e296eee7': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.5', 'valid': False}, '644b4e6d-aa9e-4628-b56c-7fb08ddc9584': {'code': 0, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.8', 'valid': True}, 'e7eaae86-11ea-4c08-8c11-8b8c54373389': {'code': 0, 'version': 3, 'acquired': False, 'delay': '0.000174126', 'lastCheck': '6.5', 'valid': True}, '474f0809-26c6-4fca-ab8c-72a2e6dcd912': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338225', 'lastCheck': '6.3', 'valid': True}}
Thread-13::DEBUG::2014-11-28 16:27:47,846::task::595::TaskManager.Task::(_updateState) Task=`b1c05d13-b7f8-4b2b-a11e-b9bf497e3028`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-11-28 16:27:47,846::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-11-28 16:27:47,846::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-11-28 16:27:47,847::task::990::TaskManager.Task::(_decref) Task=`b1c05d13-b7f8-4b2b-a11e-b9bf497e3028`::ref 0 aborting False
Thread-23::DEBUG::2014-11-28 16:27:51,395::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.225.2:_data_bs02store/e7eaae86-11ea-4c08-8c11-8b8c54373389/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-23::DEBUG::2014-11-28 16:27:51,402::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n641 bytes (641 B) copied, 0.000171831 s, 3.7 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-11-28 16:27:51,513::fileSD::259::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd if=/rhev/data-center/mnt/172.16.241.100:_var_lib_exports_iso/474f0809-26c6-4fca-ab8c-72a2e6dcd912/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-11-28 16:27:51,519::fileSD::259::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n359 bytes (359 B) copied, 0.000538593 s, 667 kB/s\n'; <rc> = 0