MainThread::INFO::2014-04-27 12:41:39,581::vdsm::119::vds::(run) (PID: 3754) I am the actual vdsm 4.14.6-0.el6 ovirt-node.example.local (2.6.32-431.11.2.el6.x86_64)
MainThread::DEBUG::2014-04-27 12:41:39,597::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-04-27 12:41:39,597::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::WARNING::2014-04-27 12:41:39,609::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
MainThread::DEBUG::2014-04-27 12:41:40,149::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-04-27 12:41:40,160::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
MainThread::DEBUG::2014-04-27 12:41:40,160::hsm::423::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-04-27 12:41:40,170::hsm::423::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-04-27 12:41:40,171::hsm::447::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-04-27 12:41:40,176::hsm::479::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-04-27 12:41:40,176::hsm::480::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-04-27 12:41:40,177::hsm::482::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-04-27 12:41:40,390::hsm::525::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-04-27 12:41:40,391::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
MainThread::INFO::2014-04-27 12:41:40,391::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-04-27 12:41:40,392::misc::764::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-04-27 12:41:40,585::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
storageRefresh::DEBUG::2014-04-27 12:41:40,585::misc::764::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-04-27 12:41:40,585::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
storageRefresh::DEBUG::2014-04-27 12:41:40,585::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,666::misc::772::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-04-27 12:41:40,666::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
MainThread::DEBUG::2014-04-27 12:41:40,737::task::595::TaskManager.Task::(_updateState) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::moving from state init -> state preparing
MainThread::INFO::2014-04-27 12:41:40,737::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x1d00b90>>)
MainThread::INFO::2014-04-27 12:41:40,737::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
storageRefresh::DEBUG::2014-04-27 12:41:40,737::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-04-27 12:41:40,738::task::1185::TaskManager.Task::(prepare) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::finished: None
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::task::595::TaskManager.Task::(_updateState) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::moving from state preparing -> state finished
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,739::task::990::TaskManager.Task::(_decref) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::ref 0 aborting False
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,740::misc::772::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-04-27 12:41:40,740::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,741::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
MainThread::INFO::2014-04-27 12:41:40,747::momIF::47::MOM::(__init__) Starting up MOM
storageRefresh::DEBUG::2014-04-27 12:41:40,752::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
MainThread::INFO::2014-04-27 12:41:40,758::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds.
VM Channels Listener::INFO::2014-04-27 12:41:40,758::vmChannels::170::vds::(run) Starting VM channels listener thread.
clientIFinit::DEBUG::2014-04-27 12:41:40,838::libvirtconnection::145::root::(get) trying to connect libvirt
MainThread::WARNING::2014-04-27 12:41:40,871::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-04-27 12:41:40,872::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,873::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,876::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,925::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,926::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,927::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,940::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,941::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::WARNING::2014-04-27 12:41:40,942::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
storageRefresh::DEBUG::2014-04-27 12:41:40,958::hsm::391::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-04-27 12:42:02,760::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getCapabilities with () {}
Thread-13::DEBUG::2014-04-27 12:42:02,800::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:02,806::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:03,101::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395777595.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el6', 'buildtime': 1395806448L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'bkupnet': {'iface': 'bkupnet', 'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'gateway': '15.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth3']}, 'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'gateway': '10.50.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}, 'vmnet': {'iface': 'vmnet', 'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'gateway': '12.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth2']}}, 'bridges': {'ovirtmgmt': {'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '10.50.0.1', 'ports': ['eth0']}, 'bkupnet': {'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '15.0.0.12', 'ports': ['eth3']}, 'vmnet': {'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '12.0.0.12', 'ports': ['eth2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth3': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c8', 'cfg': {'BRIDGE': 'bkupnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c8', 'STP': 'no', 'DEVICE': 'eth3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'speed': 1000, 'mtu': '1500'}, 'eth2': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c6', 'cfg': {'BRIDGE': 'vmnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c6', 'STP': 'no', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'speed': 1000, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:19:99:ff:d5:85', 'cfg': {'UUID': '469d4529-920d-46ae-9441-786d12b85eb7', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:19:99:FF:D5:85', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '90:1b:0e:06:f9:39', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'HWADDR': '90:1b:0e:06:f9:39', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'speed': 100, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15917', 'cpuSpeed': '3092.855', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '10.50.0.18'}}
Thread-13::DEBUG::2014-04-27 12:42:03,246::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getHardwareInfo with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,324::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PRIMERGY TX100 S3P', 'systemSerialNumber': 'YLLD033823', 'systemFamily': '', 'systemVersion': '', 'systemUUID': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'systemManufacturer': 'FUJITSU'}}
Thread-13::DEBUG::2014-04-27 12:42:03,437::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getCapabilities with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,437::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:03,441::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:03,463::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395777595.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el6', 'buildtime': 1395806448L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'bkupnet': {'iface': 'bkupnet', 'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'gateway': '15.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth3']}, 'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'gateway': '10.50.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}, 'vmnet': {'iface': 'vmnet', 'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'gateway': '12.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth2']}}, 'bridges': {'ovirtmgmt': {'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '10.50.0.1', 'ports': ['eth0']}, 'bkupnet': {'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '15.0.0.12', 'ports': ['eth3']}, 'vmnet': {'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '12.0.0.12', 'ports': ['eth2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth3': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c8', 'cfg': {'BRIDGE': 'bkupnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c8', 'STP': 'no', 'DEVICE': 'eth3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'speed': 1000, 'mtu': '1500'}, 'eth2': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c6', 'cfg': {'BRIDGE': 'vmnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c6', 'STP': 'no', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'speed': 1000, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:19:99:ff:d5:85', 'cfg': {'UUID': '469d4529-920d-46ae-9441-786d12b85eb7', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:19:99:FF:D5:85', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '90:1b:0e:06:f9:39', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'HWADDR': '90:1b:0e:06:f9:39', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'speed': 100, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15917', 'cpuSpeed': '3092.855', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '10.50.0.18'}}
Thread-13::DEBUG::2014-04-27 12:42:03,487::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getHardwareInfo with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,488::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PRIMERGY TX100 S3P', 'systemSerialNumber': 'YLLD033823', 'systemFamily': '', 'systemVersion': '', 'systemUUID': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'systemManufacturer': 'FUJITSU'}}
Thread-13::DEBUG::2014-04-27 12:42:04,684::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18] flowID [16b415b2]
Thread-13::DEBUG::2014-04-27 12:42:04,684::task::595::TaskManager.Task::(_updateState) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:04,684::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', conList=[{'port': '', 'connection': 'ovirt-noded:/iso', 'iqn': '', 'portal': '', 'user': '', 'protocol_version': '4', 'password': '******', 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}], options=None)
Thread-13::WARNING::2014-04-27 12:42:04,691::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ovirt-noded:_iso already exists
Thread-13::DEBUG::2014-04-27 12:42:04,691::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=4 ovirt-noded:/iso /rhev/data-center/mnt/ovirt-noded:_iso' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,051::hsm::2317::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ovirt-noded:_iso
Thread-13::DEBUG::2014-04-27 12:42:05,054::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('7b083758-45f9-4896-913d-11fe02043e6e',)
Thread-13::DEBUG::2014-04-27 12:42:05,054::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain}
Thread-13::INFO::2014-04-27 12:42:05,054::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,054::task::1185::TaskManager.Task::(prepare) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::finished: {'statuslist': [{'status': 0, 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,054::task::595::TaskManager.Task::(_updateState) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:42:05,054::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:42:05,055::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:42:05,055::task::990::TaskManager.Task::(_decref) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:42:05,064::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18] flowID [16b415b2]
Thread-13::DEBUG::2014-04-27 12:42:05,064::task::595::TaskManager.Task::(_updateState) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:05,065::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=4, spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', conList=[{'connection': '/vdsm_store/s2data1_s2mgt_boot1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb', 'port': ''}, {'connection': '/vdsm_store/s2data1_s2mgt_app1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '779cabae-1b55-4186-a275-7e07878579df', 'port': ''}, {'connection': '/vdsm_store/s2data1_s2usr_boot1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe', 'port': ''}], options=None)
Thread-13::DEBUG::2014-04-27 12:42:05,067::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2mgt_boot1
Thread-13::DEBUG::2014-04-27 12:42:05,103::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('0a897f2e-1b01-4577-9f91-cd136ef4a978',)
Thread-13::DEBUG::2014-04-27 12:42:05,103::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain}
Thread-13::DEBUG::2014-04-27 12:42:05,106::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2mgt_app1
Thread-13::DEBUG::2014-04-27 12:42:05,116::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('c2c4ade6-049e-4159-a294-a0c151f4983d',)
Thread-13::DEBUG::2014-04-27 12:42:05,117::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain, c2c4ade6-049e-4159-a294-a0c151f4983d: storage.localFsSD.findDomain}
Thread-13::DEBUG::2014-04-27 12:42:05,119::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2usr_boot1
Thread-13::DEBUG::2014-04-27 12:42:05,133::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e637bb04-a8b7-4c77-809c-d58051494c52',)
Thread-13::DEBUG::2014-04-27 12:42:05,134::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, e637bb04-a8b7-4c77-809c-d58051494c52: storage.localFsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain, c2c4ade6-049e-4159-a294-a0c151f4983d: storage.localFsSD.findDomain}
Thread-13::INFO::2014-04-27 12:42:05,134::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb'}, {'status': 0, 'id': '779cabae-1b55-4186-a275-7e07878579df'}, {'status': 0, 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::1185::TaskManager.Task::(prepare) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::finished: {'statuslist': [{'status': 0, 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb'}, {'status': 0, 'id': '779cabae-1b55-4186-a275-7e07878579df'}, {'status': 0, 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::595::TaskManager.Task::(_updateState) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:42:05,134::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:42:05,134::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::990::TaskManager.Task::(_decref) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:42:05,193::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:42:05,193::task::595::TaskManager.Task::(_updateState) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:05,193::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', hostID=1, msdUUID='0a897f2e-1b01-4577-9f91-cd136ef4a978', masterVersion=1, domainsMap=None, options=None)
Thread-13::DEBUG::2014-04-27 12:42:05,193::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ecee1a91-309a-46e9-9ead-9d0ad4261f55`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1007' at 'connectStoragePool'
Thread-13::DEBUG::2014-04-27 12:42:05,193::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ecee1a91-309a-46e9-9ead-9d0ad4261f55`::Granted request
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`196c5690-1da2-4687-9d13-18eee96e83e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1055' at '_connectStoragePool'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:42:05,195::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`196c5690-1da2-4687-9d13-18eee96e83e1`::Granted request
Thread-13::INFO::2014-04-27 12:42:05,195::sp::133::Storage.StoragePool::(setBackend) updating pool c713062f-300f-4256-9ac8-2d3fcfcdb002 backend from type NoneType instance 0x7fcd5b7fc210 to type StoragePoolDiskBackend instance 0x7fcd000eb200
Thread-13::INFO::2014-04-27 12:42:05,195::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool c713062f-300f-4256-9ac8-2d3fcfcdb002 with master domain: 0a897f2e-1b01-4577-9f91-cd136ef4a978 (ver = 1)
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,196::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-04-27 12:42:05,197::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,204::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-04-27 12:42:05,204::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,291::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:05,291::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-04-27 12:42:05,468::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::DEBUG::2014-04-27 12:42:05,562::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-04-27 12:42:05,604::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::DEBUG::2014-04-27 12:42:05,613::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::DEBUG::2014-04-27 12:42:05,614::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS'
Thread-13::DEBUG::2014-04-27 12:42:05,614::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS'
Thread-13::DEBUG::2014-04-27 12:42:05,614::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 verified, version 1
Thread-13::DEBUG::2014-04-27 12:42:05,614::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
Thread-13::DEBUG::2014-04-27 12:42:05,614::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,624::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::INFO::2014-04-27 12:42:05,624::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 7b083758-45f9-4896-913d-11fe02043e6e
Thread-13::DEBUG::2014-04-27 12:42:05,625::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `7b083758-45f9-4896-913d-11fe02043e6e`
Thread-17::DEBUG::2014-04-27 12:42:05,625::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 7b083758-45f9-4896-913d-11fe02043e6e
Thread-13::INFO::2014-04-27 12:42:05,625::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e637bb04-a8b7-4c77-809c-d58051494c52
Thread-13::DEBUG::2014-04-27 12:42:05,626::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `e637bb04-a8b7-4c77-809c-d58051494c52`
Thread-18::DEBUG::2014-04-27 12:42:05,626::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e637bb04-a8b7-4c77-809c-d58051494c52
Thread-13::INFO::2014-04-27 12:42:05,628::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::DEBUG::2014-04-27 12:42:05,635::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
Thread-22::DEBUG::2014-04-27 12:42:05,635::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::INFO::2014-04-27 12:42:05,636::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-13::DEBUG::2014-04-27 12:42:05,643::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `c2c4ade6-049e-4159-a294-a0c151f4983d`
Thread-13::INFO::2014-04-27 12:42:05,644::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-27::DEBUG::2014-04-27 12:42:05,645::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-13::DEBUG::2014-04-27 12:42:05,646::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `9a4a80a1-5377-4a94-ade3-e58183e916ae`
Thread-13::DEBUG::2014-04-27 12:42:05,647::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::WARNING::2014-04-27 12:42:05,648::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002 already exists
Thread-29::DEBUG::2014-04-27 12:42:05,648::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:42:05,654::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:42:05,655::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:42:05,655::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:42:05,656::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::DEBUG::2014-04-27 12:42:05,666::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-13::DEBUG::2014-04-27 12:42:05,675::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 7b083758-45f9-4896-913d-11fe02043e6e e637bb04-a8b7-4c77-809c-d58051494c52 c2c4ade6-049e-4159-a294-a0c151f4983d 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:05,698::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52
Thread-27::DEBUG::2014-04-27 12:42:05,727::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-29::DEBUG::2014-04-27 12:42:05,736::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-13::DEBUG::2014-04-27 12:42:05,739::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "7b083758-45f9-4896-913d-11fe02043e6e" not found\n  Skipping volume group 7b083758-45f9-4896-913d-11fe02043e6e\n  Volume group "e637bb04-a8b7-4c77-809c-d58051494c52" not found\n  Skipping volume group e637bb04-a8b7-4c77-809c-d58051494c52\n  Volume group "c2c4ade6-049e-4159-a294-a0c151f4983d" not found\n  Skipping volume group c2c4ade6-049e-4159-a294-a0c151f4983d\n  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-22::DEBUG::2014-04-27 12:42:05,763::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:42:05,774::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e
Thread-22::DEBUG::2014-04-27 12:42:05,789::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.0139692 s, 47.6 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:05,802::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 changed its status to Valid
Thread-22::DEBUG::2014-04-27 12:42:05,802::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-34::INFO::2014-04-27 12:42:05,804::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-27::DEBUG::2014-04-27 12:42:05,830::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-18::DEBUG::2014-04-27 12:42:05,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::WARNING::2014-04-27 12:42:05,855::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "7b083758-45f9-4896-913d-11fe02043e6e" not found', '  Skipping volume group 7b083758-45f9-4896-913d-11fe02043e6e', '  Volume group "e637bb04-a8b7-4c77-809c-d58051494c52" not found', '  Skipping volume group e637bb04-a8b7-4c77-809c-d58051494c52', '  Volume group "c2c4ade6-049e-4159-a294-a0c151f4983d" not found', '  Skipping volume group c2c4ade6-049e-4159-a294-a0c151f4983d', '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::WARNING::2014-04-27 12:42:05,855::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:42:05,858::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-18::DEBUG::2014-04-27 12:42:05,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-18::DEBUG::2014-04-27 12:42:05,967::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-18::DEBUG::2014-04-27 12:42:05,967::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e637bb04-a8b7-4c77-809c-d58051494c52_imageNS'
Thread-18::DEBUG::2014-04-27 12:42:05,967::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e637bb04-a8b7-4c77-809c-d58051494c52_volumeNS'
Thread-18::DEBUG::2014-04-27 12:42:05,978::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:05,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-18::DEBUG::2014-04-27 12:42:05,985::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000404616 s, 912 kB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:05,986::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e637bb04-a8b7-4c77-809c-d58051494c52 changed its status to Valid
Thread-48::DEBUG::2014-04-27 12:42:05,986::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-18::DEBUG::2014-04-27 12:42:05,986::clusterlock::337::LocalLock::(acquireHostId) Host id for domain e637bb04-a8b7-4c77-809c-d58051494c52 successfully acquired (id: 1)
Thread-48::DEBUG::2014-04-27 12:42:05,987::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-48::DEBUG::2014-04-27 12:42:05,987::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-27::DEBUG::2014-04-27 12:42:05,988::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-49::INFO::2014-04-27 12:42:05,988::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-27::DEBUG::2014-04-27 12:42:05,989::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'c2c4ade6-049e-4159-a294-a0c151f4983d_imageNS'
Thread-27::DEBUG::2014-04-27 12:42:05,989::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'c2c4ade6-049e-4159-a294-a0c151f4983d_volumeNS'
Thread-27::DEBUG::2014-04-27 12:42:06,002::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:06,006::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 8.4325e-05 s, 4.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:06,007::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain c2c4ade6-049e-4159-a294-a0c151f4983d changed its status to Valid
Thread-50::DEBUG::2014-04-27 12:42:06,007::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-27::DEBUG::2014-04-27 12:42:06,007::clusterlock::337::LocalLock::(acquireHostId) Host id for domain c2c4ade6-049e-4159-a294-a0c151f4983d successfully acquired (id: 1)
Thread-50::DEBUG::2014-04-27 12:42:06,008::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-50::DEBUG::2014-04-27 12:42:06,009::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-51::INFO::2014-04-27 12:42:06,009::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-22::DEBUG::2014-04-27 12:42:15,812::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:15,816::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000128207 s, 5.2 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:16,001::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:16,005::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000115492 s, 3.2 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:16,021::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:16,025::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 8.102e-05 s, 4.5 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:25,828::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:25,832::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000124018 s, 5.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:26,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:26,023::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 8.4621e-05 s, 4.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:26,039::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:26,043::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.2273e-05 s, 5.9 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:35,843::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:35,847::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000112682 s, 5.9 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:36,035::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:36,039::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000100668 s, 3.7 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:36,057::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:36,061::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.1211e-05 s, 6.0 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:45,859::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:45,862::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000105352 s, 6.3 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:46,054::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:46,058::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000130353 s, 2.8 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:46,075::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:46,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 9.7641e-05 s, 3.8 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:55,874::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:55,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000101233 s, 6.6 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:56,073::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:56,077::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000103071 s, 3.6 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:56,093::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:56,096::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.2305e-05 s, 5.9 MB/s\n'; <rc> = 0
Thread-17::ERROR::2014-04-27 12:43:05,778::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 7b083758-45f9-4896-913d-11fe02043e6e monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 147, in __init__
    validateFileSystemFeatures(sdUUID, self.mountpoint)
  File "/usr/share/vdsm/storage/fileSD.py", line 76, in validateFileSystemFeatures
    oop.getProcessPool(sdUUID).directTouch(testFilePath)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
Thread-13::INFO::2014-04-27 12:43:05,781::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/7b083758-45f9-4896-913d-11fe02043e6e
Thread-29::ERROR::2014-04-27 12:43:05,781::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:05,824::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 7b083758-45f9-4896-913d-11fe02043e6e changed its status to Invalid
Thread-13::INFO::2014-04-27 12:43:05,825::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/e637bb04-a8b7-4c77-809c-d58051494c52
Thread-29::ERROR::2014-04-27 12:43:05,825::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-52::DEBUG::2014-04-27 12:43:05,826::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-13::INFO::2014-04-27 12:43:05,826::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-17::DEBUG::2014-04-27 12:43:05,827::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-29::DEBUG::2014-04-27 12:43:05,827::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 9a4a80a1-5377-4a94-ade3-e58183e916ae changed its status to Invalid
Thread-52::DEBUG::2014-04-27 12:43:05,828::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-13::INFO::2014-04-27 12:43:05,829::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-53::DEBUG::2014-04-27 12:43:05,829::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-52::DEBUG::2014-04-27 12:43:05,830::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-13::INFO::2014-04-27 12:43:05,831::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/mastersd
Thread-53::DEBUG::2014-04-27 12:43:05,831::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-53::DEBUG::2014-04-27 12:43:05,832::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::INFO::2014-04-27 12:43:05,834::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-13::DEBUG::2014-04-27 12:43:05,834::task::1185::TaskManager.Task::(prepare) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::finished: True
Thread-13::DEBUG::2014-04-27 12:43:05,834::task::595::TaskManager.Task::(_updateState) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:05,835::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:05,835::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:05,835::task::990::TaskManager.Task::(_decref) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:05,881::task::595::TaskManager.Task::(_updateState) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:05,882::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:05,884::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000103071', 'lastCheck': '9.8', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000101233', 'lastCheck': '10.0', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '6.2305e-05', 'lastCheck': '9.8', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:05,885::task::1185::TaskManager.Task::(prepare) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000103071', 'lastCheck': '9.8', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000101233', 'lastCheck': '10.0', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '6.2305e-05', 'lastCheck': '9.8', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:05,886::task::595::TaskManager.Task::(_updateState) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:05,886::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:05,887::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:05,888::task::990::TaskManager.Task::(_decref) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::ref 0 aborting False
Thread-22::DEBUG::2014-04-27 12:43:05,915::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-04-27 12:43:05,920::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': True},) {} flowID [16b415b2]
Thread-22::DEBUG::2014-04-27 12:43:05,926::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000113663 s, 5.9 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:05,974::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-18::DEBUG::2014-04-27 12:43:06,088::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:06,092::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000119876 s, 3.1 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:06,110::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:06,114::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000118739 s, 3.1 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:09,620::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:09,620::task::595::TaskManager.Task::(_updateState) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:09,620::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:09,620::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::1185::TaskManager.Task::(prepare) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::595::TaskManager.Task::(_updateState) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:09,621::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:09,621::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::990::TaskManager.Task::(_decref) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:09,654::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:09,654::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:09,654::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:09,654::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`8d3b4f80-acf6-4f5a-950e-d2ba4e192644`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`8d3b4f80-acf6-4f5a-950e-d2ba4e192644`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::827::TaskManager.Task::(resourceAcquired) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:09,655::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::1180::TaskManager.Task::(prepare) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:09,656::taskManager::50::TaskManager::(_queueTask) queuing task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:09,656::taskManager::56::TaskManager::(_queueTask) task queued: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::1182::TaskManager.Task::(prepare) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::returning
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 0 aborting False
Thread-6::DEBUG::2014-04-27 12:43:09,656::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,656::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4 running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00189d88>> with: None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::1194::TaskManager.Task::(commit) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::committing task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state queued -> state running
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::905::TaskManager.Task::(_runJobs) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::331::TaskManager.Task::(run) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::misc::764::SamplingMethod::(__call__) Got in to sampling method
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::misc::772::SamplingMethod::(__call__) Returning last result
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,667::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,699::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1fde14b7-9079-4f20-aa21-442823d1f323`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1fde14b7-9079-4f20-aa21-442823d1f323`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,700::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`96ca8e8b-6df3-42ff-8e8c-bcf6d7758878`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,701::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,701::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`96ca8e8b-6df3-42ff-8e8c-bcf6d7758878`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,702::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,722::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,722::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,723::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-60::DEBUG::2014-04-27 12:43:09,732::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-61::DEBUG::2014-04-27 12:43:09,733::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-62::ERROR::2014-04-27 12:43:09,733::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,733::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-60::DEBUG::2014-04-27 12:43:09,734::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`3ab79dd8-4826-4364-9f0e-8c6ad5de90c1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-62::ERROR::2014-04-27 12:43:09,735::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-61::DEBUG::2014-04-27 12:43:09,735::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5263a1de-de00-433a-ac28-b871630d4576`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,735::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,735::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-62::DEBUG::2014-04-27 12:43:09,736::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-61::DEBUG::2014-04-27 12:43:09,736::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,736::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-62::DEBUG::2014-04-27 12:43:09,737::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,738::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-61::DEBUG::2014-04-27 12:43:09,742::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-61::DEBUG::2014-04-27 12:43:09,744::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5263a1de-de00-433a-ac28-b871630d4576`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,744::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-60::DEBUG::2014-04-27 12:43:09,745::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,747::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`3ab79dd8-4826-4364-9f0e-8c6ad5de90c1`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,747::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-61::DEBUG::2014-04-27 12:43:09,747::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`8a50d7b4-ad5a-4a85-967e-3f95dfcb379b`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-59::DEBUG::2014-04-27 12:43:09,749::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e
Thread-61::DEBUG::2014-04-27 12:43:09,749::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-60::DEBUG::2014-04-27 12:43:09,750::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ddcb64fe-fe94-4c2a-abb4-c708ec085d4d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-61::DEBUG::2014-04-27 12:43:09,750::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,754::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-61::DEBUG::2014-04-27 12:43:09,755::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`8a50d7b4-ad5a-4a85-967e-3f95dfcb379b`::Granted request
Thread-60::DEBUG::2014-04-27 12:43:09,757::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,758::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ddcb64fe-fe94-4c2a-abb4-c708ec085d4d`::Granted request
Thread-62::DEBUG::2014-04-27 12:43:09,758::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,762::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-62::WARNING::2014-04-27 12:43:09,762::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-62::DEBUG::2014-04-27 12:43:09,764::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-61::DEBUG::2014-04-27 12:43:09,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-61::DEBUG::2014-04-27 12:43:09,773::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-61::DEBUG::2014-04-27 12:43:09,773::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-61::DEBUG::2014-04-27 12:43:09,775::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-61::DEBUG::2014-04-27 12:43:09,776::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-62::ERROR::2014-04-27 12:43:09,782::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-60::DEBUG::2014-04-27 12:43:09,798::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-60::DEBUG::2014-04-27 12:43:09,798::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-60::DEBUG::2014-04-27 12:43:09,798::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-59::DEBUG::2014-04-27 12:43:09,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-59::DEBUG::2014-04-27 12:43:09,862::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO1_ZFS', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=f027ec99-913f-4f00-ac95-ad484c9c6a4b,c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=ovirt-noded:/iso', 'ROLE=Regular', 'SDUUID=7b083758-45f9-4896-913d-11fe02043e6e', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=25bcbb3617da81289b741097860834da2ed0040e']
Thread-59::DEBUG::2014-04-27 12:43:09,864::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-59::DEBUG::2014-04-27 12:43:09,865::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '7b083758-45f9-4896-913d-11fe02043e6e_imageNS'
Thread-59::DEBUG::2014-04-27 12:43:09,865::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '7b083758-45f9-4896-913d-11fe02043e6e_volumeNS'
Thread-59::DEBUG::2014-04-27 12:43:09,865::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,866::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-59::DEBUG::2014-04-27 12:43:09,867::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`5ba44554-9c56-4fc8-a462-76d23f3c72bb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,867::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-59::DEBUG::2014-04-27 12:43:09,867::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,868::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-59::DEBUG::2014-04-27 12:43:09,869::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-59::DEBUG::2014-04-27 12:43:09,869::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`5ba44554-9c56-4fc8-a462-76d23f3c72bb`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,871::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-59::DEBUG::2014-04-27 12:43:09,871::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`416fc1f9-3f25-4d6a-9d51-2f72f48f86e0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-59::DEBUG::2014-04-27 12:43:09,875::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-59::DEBUG::2014-04-27 12:43:09,876::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-59::DEBUG::2014-04-27 12:43:09,876::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`416fc1f9-3f25-4d6a-9d51-2f72f48f86e0`::Granted request
Thread-59::DEBUG::2014-04-27 12:43:09,876::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,885::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::WARNING::2014-04-27 12:43:09,886::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,886::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,895::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,895::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
    self.checkBackupDomain(__securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,896::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,896::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,896::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,896::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,909::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::WARNING::2014-04-27 12:43:09,910::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,910::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,920::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,921::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,921::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f38ae238-755c-4e45-8cfc-906f765396a0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f38ae238-755c-4e45-8cfc-906f765396a0`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`5e8feec4-83e0-4266-b943-c260f00f1136`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`5e8feec4-83e0-4266-b943-c260f00f1136`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,925::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,925::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,925::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,926::task::866::TaskManager.Task::(_setError) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::885::TaskManager.Task::(_run) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task._run: feedf3e9-23d6-40b8-9545-7c3a4dba8be4 () {} failed - stopping task
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::1211::TaskManager.Task::(stop) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::stopping in state running (force False)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 1 aborting True
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::916::TaskManager.Task::(_runJobs) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 0 aborting True
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::925::TaskManager.Task::(_doAbort) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task._doAbort: force False
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state running -> state aborting
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::550::TaskManager.Task::(__state_aborting) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_aborting: recover policy auto
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state aborting -> state racquiring
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state racquiring -> state recovering
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::798::TaskManager.Task::(_recover) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_recover
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::805::TaskManager.Task::(_recover) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::running recovery None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::786::TaskManager.Task::(_recoverDone) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Recover Done: state recovering
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state recovering -> state recovered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:10,701::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,701::task::595::TaskManager.Task::(_updateState) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,701::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='feedf3e9-23d6-40b8-9545-7c3a4dba8be4', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:10,701::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:10,701::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}
Thread-13::INFO::2014-04-27 12:43:10,701::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::1185::TaskManager.Task::(prepare) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::595::TaskManager.Task::(_updateState) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,702::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,702::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::990::TaskManager.Task::(_decref) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:10,708::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,708::task::595::TaskManager.Task::(_updateState) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,708::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:10,709::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::1185::TaskManager.Task::(prepare) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::595::TaskManager.Task::(_updateState) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,709::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::990::TaskManager.Task::(_decref) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:10,755::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,755::task::595::TaskManager.Task::(_updateState) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,755::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='feedf3e9-23d6-40b8-9545-7c3a4dba8be4', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:10,755::taskManager::161::TaskManager::(clearTask) Entry. taskID: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:10,755::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:10,756::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::1185::TaskManager.Task::(prepare) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::595::TaskManager.Task::(_updateState) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,756::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::990::TaskManager.Task::(_decref) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:15,830::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:15,831::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:15,832::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:15,833::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:15,849::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:15,851::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:15,852::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:15,869::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:15,870::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:15,877::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:15,882::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000689305 s, 541 kB/s\n'; <rc> = 0
Thread-17::DEBUG::2014-04-27 12:43:15,884::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 7b083758-45f9-4896-913d-11fe02043e6e changed its status to Valid
Thread-100::DEBUG::2014-04-27 12:43:15,884::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-100::DEBUG::2014-04-27 12:43:15,884::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-100::DEBUG::2014-04-27 12:43:15,885::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-101::INFO::2014-04-27 12:43:15,885::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-22::DEBUG::2014-04-27 12:43:15,944::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:15,948::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.00013004 s, 5.1 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:16,107::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:16,111::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000113178 s, 3.3 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:16,128::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:16,131::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000142305 s, 2.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:20,849::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,850::task::595::TaskManager.Task::(_updateState) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,851::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:20,851::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::1185::TaskManager.Task::(prepare) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::595::TaskManager.Task::(_updateState) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:20,851::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,851::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::990::TaskManager.Task::(_decref) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:20,856::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,856::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,856::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:20,856::task::866::TaskManager.Task::(_setError) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
    allTasksStatus = sp.getAllTasksStatuses()
  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
    raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::885::TaskManager.Task::(_run) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Task._run: 47a9679a-408e-489d-83fa-2387e4c1d64d () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::1211::TaskManager.Task::(stop) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::990::TaskManager.Task::(_decref) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:20,922::task::1168::TaskManager.Task::(prepare) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::1173::TaskManager.Task::(prepare) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::990::TaskManager.Task::(_decref) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::925::TaskManager.Task::(_doAbort) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::550::TaskManager.Task::(__state_aborting) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:20,923::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,972::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,972::task::595::TaskManager.Task::(_updateState) Task=`a7250895-430c-4166-9062-e4ee751bc621`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,972::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:20,973::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::1185::TaskManager.Task::(prepare) Task=`a7250895-430c-4166-9062-e4ee751bc621`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::595::TaskManager.Task::(_updateState) Task=`a7250895-430c-4166-9062-e4ee751bc621`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:20,973::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,973::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::990::TaskManager.Task::(_decref) Task=`a7250895-430c-4166-9062-e4ee751bc621`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:21,046::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:21,046::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:21,046::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`574424cb-3cfa-45c3-8286-065fcb6b22e2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`574424cb-3cfa-45c3-8286-065fcb6b22e2`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::827::TaskManager.Task::(resourceAcquired) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:21,047::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::1180::TaskManager.Task::(prepare) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:21,048::taskManager::50::TaskManager::(_queueTask) queuing task: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:21,048::taskManager::56::TaskManager::(_queueTask) task queued: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::1182::TaskManager.Task::(prepare) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::returning
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 0 aborting False
Thread-1::DEBUG::2014-04-27 12:43:21,048::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 6c81094c-b1cf-4565-a4b3-c11042363b34 running: <bound method Task.commit of <storage.task.Task instance at 0x1c29a70>> with: None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::1194::TaskManager.Task::(commit) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::committing task: 6c81094c-b1cf-4565-a4b3-c11042363b34
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state queued -> state running
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::905::TaskManager.Task::(_runJobs) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::331::TaskManager.Task::(run) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::misc::764::SamplingMethod::(__call__) Got in to sampling method
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::misc::772::SamplingMethod::(__call__) Returning last result
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,059::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,060::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,070::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`198baa3b-e695-4027-ad2e-2922433555f1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`198baa3b-e695-4027-ad2e-2922433555f1`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,071::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`16f8dba2-c2e9-47b8-a592-46eab358c97d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`16f8dba2-c2e9-47b8-a592-46eab358c97d`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,074::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,084::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,085::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,085::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,094::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,095::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,095::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-105::DEBUG::2014-04-27 12:43:21,095::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-105::DEBUG::2014-04-27 12:43:21,095::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a5ca1a94-a356-455e-b82c-808408e53ab8`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,096::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-107::DEBUG::2014-04-27 12:43:21,096::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a5ca1a94-a356-455e-b82c-808408e53ab8`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,097::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-108::ERROR::2014-04-27 12:43:21,097::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-107::DEBUG::2014-04-27 12:43:21,097::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`2260c82e-6d6e-4b4b-8648-9e6a29b1a9d0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-106::DEBUG::2014-04-27 12:43:21,097::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ad4dd919-e918-4226-9e10-8bcdbaadbafc`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,098::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`ddd5e1a7-4937-458f-b7cb-68c7ddeab8f4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,098::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-108::ERROR::2014-04-27 12:43:21,098::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-107::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-106::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,098::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-108::DEBUG::2014-04-27 12:43:21,098::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,099::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-108::DEBUG::2014-04-27 12:43:21,099::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-107::DEBUG::2014-04-27 12:43:21,099::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,103::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-107::DEBUG::2014-04-27 12:43:21,103::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`2260c82e-6d6e-4b4b-8648-9e6a29b1a9d0`::Granted request
Thread-106::DEBUG::2014-04-27 12:43:21,103::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,104::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ad4dd919-e918-4226-9e10-8bcdbaadbafc`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,104::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-105::DEBUG::2014-04-27 12:43:21,104::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-107::DEBUG::2014-04-27 12:43:21,104::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`68c7edb8-45f3-4423-ba0b-61b654e480b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-106::DEBUG::2014-04-27 12:43:21,105::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ebf341b1-9f45-494e-9067-5ef60b7df58d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,105::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`ddd5e1a7-4937-458f-b7cb-68c7ddeab8f4`::Granted request
Thread-107::DEBUG::2014-04-27 12:43:21,105::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,106::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-106::DEBUG::2014-04-27 12:43:21,106::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-107::DEBUG::2014-04-27 12:43:21,106::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-105::DEBUG::2014-04-27 12:43:21,106::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-107::DEBUG::2014-04-27 12:43:21,107::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`68c7edb8-45f3-4423-ba0b-61b654e480b4`::Granted request
Thread-106::DEBUG::2014-04-27 12:43:21,107::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,107::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ebf341b1-9f45-494e-9067-5ef60b7df58d`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-107::DEBUG::2014-04-27 12:43:21,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-105::DEBUG::2014-04-27 12:43:21,118::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-107::DEBUG::2014-04-27 12:43:21,119::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-105::DEBUG::2014-04-27 12:43:21,120::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-105::DEBUG::2014-04-27 12:43:21,122::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-106::DEBUG::2014-04-27 12:43:21,123::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-108::DEBUG::2014-04-27 12:43:21,123::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-107::DEBUG::2014-04-27 12:43:21,124::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-107::DEBUG::2014-04-27 12:43:21,127::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-105::DEBUG::2014-04-27 12:43:21,129::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-106::DEBUG::2014-04-27 12:43:21,130::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-107::DEBUG::2014-04-27 12:43:21,131::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-108::WARNING::2014-04-27 12:43:21,134::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-105::DEBUG::2014-04-27 12:43:21,134::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,137::task::595::TaskManager.Task::(_updateState) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::moving from state init -> state preparing
Thread-105::DEBUG::2014-04-27 12:43:21,137::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-107::DEBUG::2014-04-27 12:43:21,138::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-108::DEBUG::2014-04-27 12:43:21,138::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-13::INFO::2014-04-27 12:43:21,139::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-105::DEBUG::2014-04-27 12:43:21,139::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-106::DEBUG::2014-04-27 12:43:21,141::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-106::DEBUG::2014-04-27 12:43:21,144::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-13::INFO::2014-04-27 12:43:21,144::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000689305', 'lastCheck': '5.3', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000113178', 'lastCheck': '5.0', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00013004', 'lastCheck': '5.2', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000142305', 'lastCheck': '5.0', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.3', 'valid': False}}
Thread-106::DEBUG::2014-04-27 12:43:21,146::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:21,147::task::1185::TaskManager.Task::(prepare) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000689305', 'lastCheck': '5.3', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000113178', 'lastCheck': '5.0', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00013004', 'lastCheck': '5.2', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000142305', 'lastCheck': '5.0', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.3', 'valid': False}}
Thread-107::DEBUG::2014-04-27 12:43:21,150::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,151::task::595::TaskManager.Task::(_updateState) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::moving from state preparing -> state finished
Thread-107::DEBUG::2014-04-27 12:43:21,154::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:21,154::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-106::DEBUG::2014-04-27 12:43:21,154::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-107::DEBUG::2014-04-27 12:43:21,156::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:21,157::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-106::DEBUG::2014-04-27 12:43:21,159::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,160::task::990::TaskManager.Task::(_decref) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::ref 0 aborting False
Thread-106::DEBUG::2014-04-27 12:43:21,161::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-106::DEBUG::2014-04-27 12:43:21,164::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-108::ERROR::2014-04-27 12:43:21,170::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,171::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,172::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,173::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,177::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,190::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
6c81094c-b1cf-4565-a4b3-c11042363b34::WARNING::2014-04-27 12:43:21,191::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,191::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,204::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,204::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
    self.checkBackupDomain(__securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,205::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,205::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,205::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,206::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,219::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
6c81094c-b1cf-4565-a4b3-c11042363b34::WARNING::2014-04-27 12:43:21,219::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,220::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,230::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`224f5b16-8485-4f1d-bcd3-0fc6ae0b618f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`224f5b16-8485-4f1d-bcd3-0fc6ae0b618f`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`c6289201-7b36-4cc7-a69d-5fc62a0240f3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`c6289201-7b36-4cc7-a69d-5fc62a0240f3`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,234::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,234::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,235::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,235::task::866::TaskManager.Task::(_setError) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::885::TaskManager.Task::(_run) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task._run: 6c81094c-b1cf-4565-a4b3-c11042363b34 () {} failed - stopping task
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::1211::TaskManager.Task::(stop) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::stopping in state running (force False)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 1 aborting True
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::916::TaskManager.Task::(_runJobs) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 0 aborting True
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::925::TaskManager.Task::(_doAbort) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task._doAbort: force False
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state running -> state aborting
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::550::TaskManager.Task::(__state_aborting) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_aborting: recover policy auto
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state aborting -> state racquiring
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state racquiring -> state recovering
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::798::TaskManager.Task::(_recover) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_recover
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::805::TaskManager.Task::(_recover) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::running recovery None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::786::TaskManager.Task::(_recoverDone) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Recover Done: state recovering
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state recovering -> state recovered
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,238::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,238::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:22,094::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,094::task::595::TaskManager.Task::(_updateState) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,094::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='6c81094c-b1cf-4565-a4b3-c11042363b34', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:22,094::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:22,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}
Thread-13::INFO::2014-04-27 12:43:22,095::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::1185::TaskManager.Task::(prepare) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::595::TaskManager.Task::(_updateState) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,095::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::990::TaskManager.Task::(_decref) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:22,101::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::595::TaskManager.Task::(_updateState) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,101::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:22,101::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::1185::TaskManager.Task::(prepare) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::595::TaskManager.Task::(_updateState) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,101::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,102::task::990::TaskManager.Task::(_decref) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:22,146::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::595::TaskManager.Task::(_updateState) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,147::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='6c81094c-b1cf-4565-a4b3-c11042363b34', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:22,147::taskManager::161::TaskManager::(clearTask) Entry. taskID: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:22,147::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:22,147::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::1185::TaskManager.Task::(prepare) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::595::TaskManager.Task::(_updateState) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,147::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,147::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,148::task::990::TaskManager.Task::(_decref) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:25,872::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:25,872::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:25,872::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:25,872::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:25,888::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:25,890::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:25,892::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:25,909::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:25,910::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:25,931::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:25,935::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000781446 s, 477 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:25,962::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:25,965::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000128537 s, 5.2 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:26,126::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:26,130::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000108243 s, 3.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:26,145::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:26,149::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000100579 s, 3.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:32,220::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,220::task::595::TaskManager.Task::(_updateState) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,221::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:32,221::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::1185::TaskManager.Task::(prepare) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::595::TaskManager.Task::(_updateState) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:32,221::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,221::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::990::TaskManager.Task::(_decref) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:32,225::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,226::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:32,226::task::866::TaskManager.Task::(_setError) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
    allTasksStatus = sp.getAllTasksStatuses()
  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
    raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::885::TaskManager.Task::(_run) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Task._run: 96ad43f1-38cb-4249-a2f9-8c02a5abe445 () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::1211::TaskManager.Task::(stop) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::990::TaskManager.Task::(_decref) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:32,226::task::1168::TaskManager.Task::(prepare) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::1173::TaskManager.Task::(prepare) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::990::TaskManager.Task::(_decref) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::925::TaskManager.Task::(_doAbort) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::550::TaskManager.Task::(__state_aborting) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:32,227::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,276::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::595::TaskManager.Task::(_updateState) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,277::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:32,277::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::1185::TaskManager.Task::(prepare) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::595::TaskManager.Task::(_updateState) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:32,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,278::task::990::TaskManager.Task::(_decref) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:32,349::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,349::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,350::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`fe669ea3-2285-48e5-9a1f-7f40c1ad7abf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`fe669ea3-2285-48e5-9a1f-7f40c1ad7abf`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::827::TaskManager.Task::(resourceAcquired) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:32,351::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::1180::TaskManager.Task::(prepare) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:32,351::taskManager::50::TaskManager::(_queueTask) queuing task: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:32,351::taskManager::56::TaskManager::(_queueTask) task queued: 5a22ef66-e6a9-4d46-88be-70a384dc5442
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:32,352::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:32,352::task::1182::TaskManager.Task::(prepare) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::returning
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,352::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 5a22ef66-e6a9-4d46-88be-70a384dc5442 running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00188ef0>> with: None
Thread-13::DEBUG::2014-04-27 12:43:32,353::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 0 aborting False
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,353::task::1194::TaskManager.Task::(commit) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::committing task: 5a22ef66-e6a9-4d46-88be-70a384dc5442
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,354::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state queued -> state running
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::task::905::TaskManager.Task::(_runJobs) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::task::331::TaskManager.Task::(run) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::misc::764::SamplingMethod::(__call__) Got in to sampling method
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::misc::772::SamplingMethod::(__call__) Returning last result
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,365::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,366::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,376::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`6ad8f73a-54d4-4840-9e50-f3a5b0892163`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`6ad8f73a-54d4-4840-9e50-f3a5b0892163`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,377::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`743af4fb-133f-4c84-9102-d44bc6260e8f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`743af4fb-133f-4c84-9102-d44bc6260e8f`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,382::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,383::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,392::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,393::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,393::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-144::DEBUG::2014-04-27 12:43:32,404::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-145::DEBUG::2014-04-27 12:43:32,404::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-146::DEBUG::2014-04-27 12:43:32,405::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-147::ERROR::2014-04-27 12:43:32,407::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,407::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-147::ERROR::2014-04-27 12:43:32,408::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-144::DEBUG::2014-04-27 12:43:32,409::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`7ab40a6b-b38b-4487-b919-d09f70b94309`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-145::DEBUG::2014-04-27 12:43:32,410::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`0c85280a-218d-4933-8ffc-b6a91684614a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,411::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,411::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`58397b55-c600-4331-a969-c0fd4a439554`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-144::DEBUG::2014-04-27 12:43:32,411::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-147::DEBUG::2014-04-27 12:43:32,412::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-145::DEBUG::2014-04-27 12:43:32,412::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,412::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-146::DEBUG::2014-04-27 12:43:32,413::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,414::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-144::DEBUG::2014-04-27 12:43:32,415::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,416::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`7ab40a6b-b38b-4487-b919-d09f70b94309`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,418::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-145::DEBUG::2014-04-27 12:43:32,418::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-147::DEBUG::2014-04-27 12:43:32,419::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-145::DEBUG::2014-04-27 12:43:32,420::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`0c85280a-218d-4933-8ffc-b6a91684614a`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,425::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-144::DEBUG::2014-04-27 12:43:32,425::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`67ac383c-c84b-4309-bc44-f341f8c490e3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-146::DEBUG::2014-04-27 12:43:32,427::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-146::DEBUG::2014-04-27 12:43:32,428::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`58397b55-c600-4331-a969-c0fd4a439554`::Granted request
Thread-144::DEBUG::2014-04-27 12:43:32,428::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-145::DEBUG::2014-04-27 12:43:32,431::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ef073285-3ea7-4f61-81c3-5593454f1797`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-145::DEBUG::2014-04-27 12:43:32,432::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-146::DEBUG::2014-04-27 12:43:32,434::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5ca6f208-4b81-4db4-b3f3-66ed05bdc66d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-146::DEBUG::2014-04-27 12:43:32,434::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-147::DEBUG::2014-04-27 12:43:32,435::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-147::WARNING::2014-04-27 12:43:32,436::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-147::DEBUG::2014-04-27 12:43:32,436::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-144::DEBUG::2014-04-27 12:43:32,431::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,443::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`67ac383c-c84b-4309-bc44-f341f8c490e3`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-145::DEBUG::2014-04-27 12:43:32,448::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-145::DEBUG::2014-04-27 12:43:32,450::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ef073285-3ea7-4f61-81c3-5593454f1797`::Granted request
Thread-144::DEBUG::2014-04-27 12:43:32,451::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-147::ERROR::2014-04-27 12:43:32,451::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-146::DEBUG::2014-04-27 12:43:32,453::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,454::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,455::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-146::DEBUG::2014-04-27 12:43:32,456::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5ca6f208-4b81-4db4-b3f3-66ed05bdc66d`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,456::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-144::DEBUG::2014-04-27 12:43:32,458::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,458::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-144::DEBUG::2014-04-27 12:43:32,459::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-144::DEBUG::2014-04-27 12:43:32,459::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-144::DEBUG::2014-04-27 12:43:32,461::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-144::DEBUG::2014-04-27 12:43:32,461::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,462::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-144::DEBUG::2014-04-27 12:43:32,462::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-144::DEBUG::2014-04-27 12:43:32,466::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-145::DEBUG::2014-04-27 12:43:32,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-145::DEBUG::2014-04-27 12:43:32,476::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-145::DEBUG::2014-04-27 12:43:32,477::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,477::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-145::DEBUG::2014-04-27 12:43:32,478::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-145::DEBUG::2014-04-27 12:43:32,479::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::WARNING::2014-04-27 12:43:32,480::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-146::DEBUG::2014-04-27 12:43:32,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-145::DEBUG::2014-04-27 12:43:32,481::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,482::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-145::DEBUG::2014-04-27 12:43:32,484::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,484::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-145::DEBUG::2014-04-27 12:43:32,485::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-145::DEBUG::2014-04-27 12:43:32,486::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-146::DEBUG::2014-04-27 12:43:32,488::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,490::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-146::DEBUG::2014-04-27 12:43:32,491::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-146::DEBUG::2014-04-27 12:43:32,494::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-146::DEBUG::2014-04-27 12:43:32,497::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,499::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,500::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-146::DEBUG::2014-04-27 12:43:32,500::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,501::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
    self.checkBackupDomain(__securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,502::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,502::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,502::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,503::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,516::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
5a22ef66-e6a9-4d46-88be-70a384dc5442::WARNING::2014-04-27 12:43:32,517::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,517::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`467354ae-fac1-4cb3-8462-107de32251d3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`467354ae-fac1-4cb3-8462-107de32251d3`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`0dd6852a-2ca2-4348-8d47-9166ee323b5d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`0dd6852a-2ca2-4348-8d47-9166ee323b5d`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,531::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,531::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,532::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,532::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,532::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,532::task::866::TaskManager.Task::(_setError) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::885::TaskManager.Task::(_run) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task._run: 5a22ef66-e6a9-4d46-88be-70a384dc5442 () {} failed - stopping task
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::1211::TaskManager.Task::(stop) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::stopping in state running (force False)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 1 aborting True
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::916::TaskManager.Task::(_runJobs) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 0 aborting True
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::925::TaskManager.Task::(_doAbort) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task._doAbort: force False
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state running -> state aborting
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::550::TaskManager.Task::(__state_aborting) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_aborting: recover policy auto
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state aborting -> state racquiring
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state racquiring -> state recovering
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::798::TaskManager.Task::(_recover) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_recover
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::805::TaskManager.Task::(_recover) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::running recovery None
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::786::TaskManager.Task::(_recoverDone) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Recover Done: state recovering
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state recovering -> state recovered
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:33,397::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::595::TaskManager.Task::(_updateState) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,398::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='5a22ef66-e6a9-4d46-88be-70a384dc5442', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:33,398::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:33,398::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}
Thread-13::INFO::2014-04-27 12:43:33,398::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::1185::TaskManager.Task::(prepare) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::595::TaskManager.Task::(_updateState) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,398::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,398::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::990::TaskManager.Task::(_decref) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:33,404::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::595::TaskManager.Task::(_updateState) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,404::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:33,404::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::1185::TaskManager.Task::(prepare) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::595::TaskManager.Task::(_updateState) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,405::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,405::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,405::task::990::TaskManager.Task::(_decref) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:33,449::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,449::task::595::TaskManager.Task::(_updateState) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,449::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='5a22ef66-e6a9-4d46-88be-70a384dc5442', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:33,450::taskManager::161::TaskManager::(clearTask) Entry. taskID: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:33,450::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:33,450::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::1185::TaskManager.Task::(prepare) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::595::TaskManager.Task::(_updateState) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,450::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,450::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::990::TaskManager.Task::(_decref) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:35,911::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:35,912::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:35,912::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:35,912::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:35,926::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:35,926::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:35,926::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:35,937::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:35,937::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:35,960::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:35,965::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000713226 s, 523 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:35,979::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:35,983::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 7.9599e-05 s, 8.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:36,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:36,145::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000109191 s, 3.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:36,163::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:36,167::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000137089 s, 2.7 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:36,329::task::595::TaskManager.Task::(_updateState) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:36,330::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:36,330::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000713226', 'lastCheck': '0.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000109191', 'lastCheck': '0.2', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '7.9599e-05', 'lastCheck': '0.3', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000137089', 'lastCheck': '0.2', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:36,330::task::1185::TaskManager.Task::(prepare) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000713226', 'lastCheck': '0.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000109191', 'lastCheck': '0.2', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '7.9599e-05', 'lastCheck': '0.3', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000137089', 'lastCheck': '0.2', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:36,330::task::595::TaskManager.Task::(_updateState) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:36,330::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:36,330::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:36,331::task::990::TaskManager.Task::(_decref) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,550::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::595::TaskManager.Task::(_updateState) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,551::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:43,551::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::1185::TaskManager.Task::(prepare) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::595::TaskManager.Task::(_updateState) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:43,551::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,551::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,552::task::990::TaskManager.Task::(_decref) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,556::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,556::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,556::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:43,556::task::866::TaskManager.Task::(_setError) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
    allTasksStatus = sp.getAllTasksStatuses()
  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
    raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::885::TaskManager.Task::(_run) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Task._run: 2e2e74ad-be80-486e-adac-011cadf3a713 () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::1211::TaskManager.Task::(stop) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::990::TaskManager.Task::(_decref) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:43,557::task::1168::TaskManager.Task::(prepare) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::1173::TaskManager.Task::(prepare) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::990::TaskManager.Task::(_decref) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::925::TaskManager.Task::(_doAbort) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:43,557::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:43,558::task::550::TaskManager.Task::(__state_aborting) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:43,558::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:43,558::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,558::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:43,558::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,607::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::595::TaskManager.Task::(_updateState) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,607::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:43,607::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::1185::TaskManager.Task::(prepare) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::595::TaskManager.Task::(_updateState) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:43,607::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,608::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,608::task::990::TaskManager.Task::(_decref) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,677::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,677::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,677::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`68802a3e-fc25-4bae-a693-81462470ecdf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`68802a3e-fc25-4bae-a693-81462470ecdf`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:43,678::task::827::TaskManager.Task::(resourceAcquired) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:43,679::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::1180::TaskManager.Task::(prepare) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:43,679::taskManager::50::TaskManager::(_queueTask) queuing task: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:43,679::taskManager::56::TaskManager::(_queueTask) task queued: 65d28cfc-d761-4f97-94f1-48886843966f
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:43,680::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:43,680::task::1182::TaskManager.Task::(prepare) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::returning
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,680::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 65d28cfc-d761-4f97-94f1-48886843966f running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00169638>> with: None
Thread-13::DEBUG::2014-04-27 12:43:43,681::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 0 aborting False
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,681::task::1194::TaskManager.Task::(commit) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::committing task: 65d28cfc-d761-4f97-94f1-48886843966f
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,682::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state queued -> state running
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::task::905::TaskManager.Task::(_runJobs) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::task::331::TaskManager.Task::(run) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::misc::764::SamplingMethod::(__call__) Got in to sampling method
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,692::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::misc::772::SamplingMethod::(__call__) Returning last result
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,693::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,704::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`5df371d5-0cfa-467c-855e-66c9b8db67db`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`5df371d5-0cfa-467c-855e-66c9b8db67db`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,705::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`34b26d27-705f-4f25-80e8-b2bc2f3b0d66`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,706::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,706::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`34b26d27-705f-4f25-80e8-b2bc2f3b0d66`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,710::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,720::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,720::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,721::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-183::DEBUG::2014-04-27 12:43:43,731::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-184::DEBUG::2014-04-27 12:43:43,732::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-185::DEBUG::2014-04-27 12:43:43,733::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-186::ERROR::2014-04-27 12:43:43,734::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,735::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-186::ERROR::2014-04-27 12:43:43,736::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,736::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`fe18f291-25c0-405b-8f1d-6bde9dc0d3c0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-184::DEBUG::2014-04-27 12:43:43,737::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`6fd2488a-0158-4ade-9833-bece0980ac9d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,738::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,738::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`6b0db895-87c9-48dd-b121-5cd800612ccb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-183::DEBUG::2014-04-27 12:43:43,738::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-186::DEBUG::2014-04-27 12:43:43,739::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-184::DEBUG::2014-04-27 12:43:43,739::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,740::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,740::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,742::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-183::DEBUG::2014-04-27 12:43:43,742::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,743::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`fe18f291-25c0-405b-8f1d-6bde9dc0d3c0`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,745::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-184::DEBUG::2014-04-27 12:43:43,745::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-184::DEBUG::2014-04-27 12:43:43,747::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`6fd2488a-0158-4ade-9833-bece0980ac9d`::Granted request
Thread-186::DEBUG::2014-04-27 12:43:43,747::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,752::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-183::DEBUG::2014-04-27 12:43:43,753::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`3dbc4063-c6d1-4115-a6dc-3d4d08a97da8`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-185::DEBUG::2014-04-27 12:43:43,753::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,755::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-185::DEBUG::2014-04-27 12:43:43,756::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`6b0db895-87c9-48dd-b121-5cd800612ccb`::Granted request
Thread-184::DEBUG::2014-04-27 12:43:43,758::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`83868e7a-a661-4661-ac4e-92b09e4bb711`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-183::DEBUG::2014-04-27 12:43:43,759::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-185::DEBUG::2014-04-27 12:43:43,760::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`3051cfcb-c306-4e5d-a9ad-7e836f551a94`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-186::DEBUG::2014-04-27 12:43:43,761::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-186::WARNING::2014-04-27 12:43:43,763::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-186::DEBUG::2014-04-27 12:43:43,763::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-184::DEBUG::2014-04-27 12:43:43,768::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-183::DEBUG::2014-04-27 12:43:43,768::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`3dbc4063-c6d1-4115-a6dc-3d4d08a97da8`::Granted request
Thread-185::DEBUG::2014-04-27 12:43:43,769::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-184::DEBUG::2014-04-27 12:43:43,771::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-184::DEBUG::2014-04-27 12:43:43,774::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`83868e7a-a661-4661-ac4e-92b09e4bb711`::Granted request
Thread-183::DEBUG::2014-04-27 12:43:43,774::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-185::DEBUG::2014-04-27 12:43:43,778::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,778::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-185::DEBUG::2014-04-27 12:43:43,781::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`3051cfcb-c306-4e5d-a9ad-7e836f551a94`::Granted request
Thread-183::DEBUG::2014-04-27 12:43:43,783::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-183::DEBUG::2014-04-27 12:43:43,785::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-183::DEBUG::2014-04-27 12:43:43,787::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-185::DEBUG::2014-04-27 12:43:43,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-183::DEBUG::2014-04-27 12:43:43,803::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-184::DEBUG::2014-04-27 12:43:43,804::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-186::ERROR::2014-04-27 12:43:43,807::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-183::DEBUG::2014-04-27 12:43:43,807::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,807::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-184::DEBUG::2014-04-27 12:43:43,808::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,809::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,809::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,810::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,811::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,812::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-185::DEBUG::2014-04-27 12:43:43,812::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,813::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,813::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-184::DEBUG::2014-04-27 12:43:43,814::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-184::DEBUG::2014-04-27 12:43:43,815::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,815::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-184::DEBUG::2014-04-27 12:43:43,816::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,817::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-185::DEBUG::2014-04-27 12:43:43,817::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,821::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-184::DEBUG::2014-04-27 12:43:43,822::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-185::DEBUG::2014-04-27 12:43:43,822::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-184::DEBUG::2014-04-27 12:43:43,823::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-184::DEBUG::2014-04-27 12:43:43,823::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-184::DEBUG::2014-04-27 12:43:43,824::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,831::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
65d28cfc-d761-4f97-94f1-48886843966f::WARNING::2014-04-27 12:43:43,831::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,831::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,841::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,841::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
    self.checkBackupDomain(__securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,842::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,842::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,842::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,843::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,856::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
65d28cfc-d761-4f97-94f1-48886843966f::WARNING::2014-04-27 12:43:43,856::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,857::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,867::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,867::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,868::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`e0ccd375-ac40-42cb-bd7f-a00809cf4eb0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`e0ccd375-ac40-42cb-bd7f-a00809cf4eb0`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`f265244e-fe3e-4433-86a4-15a2873ab7b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`f265244e-fe3e-4433-86a4-15a2873ab7b4`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,872::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,872::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,872::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,872::task::866::TaskManager.Task::(_setError) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::885::TaskManager.Task::(_run) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task._run: 65d28cfc-d761-4f97-94f1-48886843966f () {} failed - stopping task
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::1211::TaskManager.Task::(stop) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::stopping in state running (force False)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 1 aborting True
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::916::TaskManager.Task::(_runJobs) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 0 aborting True
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::925::TaskManager.Task::(_doAbort) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task._doAbort: force False
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state running -> state aborting
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::550::TaskManager.Task::(__state_aborting) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_aborting: recover policy auto
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state aborting -> state racquiring
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state racquiring -> state recovering
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::798::TaskManager.Task::(_recover) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_recover
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::805::TaskManager.Task::(_recover) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::running recovery None
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::786::TaskManager.Task::(_recoverDone) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Recover Done: state recovering
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state recovering -> state recovered
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:44,725::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::595::TaskManager.Task::(_updateState) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,726::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='65d28cfc-d761-4f97-94f1-48886843966f', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:44,726::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:44,726::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}
Thread-13::INFO::2014-04-27 12:43:44,726::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}}
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::1185::TaskManager.Task::(prepare) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}}
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::595::TaskManager.Task::(_updateState) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,727::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,727::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,727::task::990::TaskManager.Task::(_decref) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:44,733::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::595::TaskManager.Task::(_updateState) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,733::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:44,733::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::1185::TaskManager.Task::(prepare) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::595::TaskManager.Task::(_updateState) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,734::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,734::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,734::task::990::TaskManager.Task::(_decref) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:44,779::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::595::TaskManager.Task::(_updateState) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,780::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='65d28cfc-d761-4f97-94f1-48886843966f', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:44,780::taskManager::161::TaskManager::(clearTask) Entry. taskID: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:44,780::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:44,780::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::1185::TaskManager.Task::(prepare) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::595::TaskManager.Task::(_updateState) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,780::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,781::task::990::TaskManager.Task::(_decref) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:45,938::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:45,938::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:45,938::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:45,938::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:45,952::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:45,952::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:45,952::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:45,963::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:45,963::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:46,019::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:46,029::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000653768 s, 571 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:46,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:46,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000108903 s, 6.1 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:46,160::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:46,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000115133 s, 3.2 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:46,181::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:46,184::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.00010829 s, 3.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:51,471::task::595::TaskManager.Task::(_updateState) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:51,472::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:51,472::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653768', 'lastCheck': '5.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000115133', 'lastCheck': '5.3', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000108903', 'lastCheck': '5.4', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00010829', 'lastCheck': '5.3', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::1185::TaskManager.Task::(prepare) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653768', 'lastCheck': '5.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000115133', 'lastCheck': '5.3', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000108903', 'lastCheck': '5.4', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00010829', 'lastCheck': '5.3', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::595::TaskManager.Task::(_updateState) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:51,472::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:51,472::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::990::TaskManager.Task::(_decref) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:54,869::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::595::TaskManager.Task::(_updateState) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,869::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:54,869::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::1185::TaskManager.Task::(prepare) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::595::TaskManager.Task::(_updateState) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:54,870::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,870::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,870::task::990::TaskManager.Task::(_decref) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:54,874::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,874::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,874::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:54,874::task::866::TaskManager.Task::(_setError) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
    allTasksStatus = sp.getAllTasksStatuses()
  File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
    raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::885::TaskManager.Task::(_run) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Task._run: b40fc62c-69e4-4198-8688-fe0a29553a3d () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::1211::TaskManager.Task::(stop) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::990::TaskManager.Task::(_decref) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:54,875::task::1168::TaskManager.Task::(prepare) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::1173::TaskManager.Task::(prepare) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::990::TaskManager.Task::(_decref) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::925::TaskManager.Task::(_doAbort) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:54,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::550::TaskManager.Task::(__state_aborting) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:54,876::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:54,876::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,876::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:54,876::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
    raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,925::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::595::TaskManager.Task::(_updateState) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,925::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:54,925::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::1185::TaskManager.Task::(prepare) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::595::TaskManager.Task::(_updateState) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:54,926::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,926::task::990::TaskManager.Task::(_decref) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:55,006::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:55,006::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:55,007::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`41387658-59c1-4051-89e0-99b6b07ecde9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`41387658-59c1-4051-89e0-99b6b07ecde9`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:55,007::task::827::TaskManager.Task::(resourceAcquired) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:55,008::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::1180::TaskManager.Task::(prepare) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:55,008::taskManager::50::TaskManager::(_queueTask) queuing task: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:55,008::taskManager::56::TaskManager::(_queueTask) task queued: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::1182::TaskManager.Task::(prepare) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::returning
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:55,008::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:55,009::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 0 aborting False
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 007f8290-5a3f-44ed-8802-c9a26c3718ae running: <bound method Task.commit of <storage.task.Task instance at 0x1c29a70>> with: None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::1194::TaskManager.Task::(commit) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::committing task: 007f8290-5a3f-44ed-8802-c9a26c3718ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state queued -> state running
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::905::TaskManager.Task::(_runJobs) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::331::TaskManager.Task::(run) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,010::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,010::misc::764::SamplingMethod::(__call__) Got in to sampling method
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,019::misc::772::SamplingMethod::(__call__) Returning last result
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,020::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,029::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,030::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,030::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`bc5cd286-6e68-45d3-9220-a09aa59a3c27`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`bc5cd286-6e68-45d3-9220-a09aa59a3c27`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,031::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`ea94ba76-06d3-49b9-9a52-a6f4ac6f5f53`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`ea94ba76-06d3-49b9-9a52-a6f4ac6f5f53`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,034::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,043::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,044::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,044::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,045::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,045::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-222::DEBUG::2014-04-27 12:43:55,054::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-223::DEBUG::2014-04-27 12:43:55,055::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-224::DEBUG::2014-04-27 12:43:55,055::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-222::DEBUG::2014-04-27 12:43:55,055::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`1b4af6e7-372b-459e-bc09-9e1743d8340f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-225::ERROR::2014-04-27 12:43:55,056::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,056::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-223::DEBUG::2014-04-27 12:43:55,056::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`43f758a8-c60c-4991-8849-f1f5281c034e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-224::DEBUG::2014-04-27 12:43:55,056::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`7a4c6bbc-a127-4b64-8ffd-8cde731c5746`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-222::DEBUG::2014-04-27 12:43:55,056::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-225::ERROR::2014-04-27 12:43:55,057::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,057::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,057::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-224::DEBUG::2014-04-27 12:43:55,057::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-225::DEBUG::2014-04-27 12:43:55,057::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,057::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-225::DEBUG::2014-04-27 12:43:55,058::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,058::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-222::DEBUG::2014-04-27 12:43:55,062::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,062::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`1b4af6e7-372b-459e-bc09-9e1743d8340f`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,062::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-223::DEBUG::2014-04-27 12:43:55,062::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-223::DEBUG::2014-04-27 12:43:55,063::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`43f758a8-c60c-4991-8849-f1f5281c034e`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,063::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a87d0814-57f4-4980-8f30-ad2c26dd120c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,063::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-223::DEBUG::2014-04-27 12:43:55,064::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ae9bcb49-96e6-4285-8254-21172289e844`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-224::DEBUG::2014-04-27 12:43:55,064::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,064::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-223::DEBUG::2014-04-27 12:43:55,064::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-224::DEBUG::2014-04-27 12:43:55,064::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`7a4c6bbc-a127-4b64-8ffd-8cde731c5746`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,065::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-224::DEBUG::2014-04-27 12:43:55,065::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`f620074f-c736-415b-acbd-35639f6451b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-222::DEBUG::2014-04-27 12:43:55,065::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a87d0814-57f4-4980-8f30-ad2c26dd120c`::Granted request
Thread-224::DEBUG::2014-04-27 12:43:55,066::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-223::DEBUG::2014-04-27 12:43:55,066::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-223::DEBUG::2014-04-27 12:43:55,066::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ae9bcb49-96e6-4285-8254-21172289e844`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,066::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-224::DEBUG::2014-04-27 12:43:55,066::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,067::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-224::DEBUG::2014-04-27 12:43:55,067::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`f620074f-c736-415b-acbd-35639f6451b4`::Granted request
Thread-225::DEBUG::2014-04-27 12:43:55,072::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-225::WARNING::2014-04-27 12:43:55,073::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-225::DEBUG::2014-04-27 12:43:55,073::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-222::DEBUG::2014-04-27 12:43:55,078::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-222::DEBUG::2014-04-27 12:43:55,082::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-224::DEBUG::2014-04-27 12:43:55,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-223::DEBUG::2014-04-27 12:43:55,086::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-222::DEBUG::2014-04-27 12:43:55,085::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-224::DEBUG::2014-04-27 12:43:55,090::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-225::ERROR::2014-04-27 12:43:55,091::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-223::DEBUG::2014-04-27 12:43:55,092::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-222::DEBUG::2014-04-27 12:43:55,092::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,093::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-224::DEBUG::2014-04-27 12:43:55,093::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,095::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-224::DEBUG::2014-04-27 12:43:55,096::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,096::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-224::DEBUG::2014-04-27 12:43:55,096::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-223::DEBUG::2014-04-27 12:43:55,097::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,098::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-224::DEBUG::2014-04-27 12:43:55,098::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-223::DEBUG::2014-04-27 12:43:55,099::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-222::DEBUG::2014-04-27 12:43:55,100::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-222::DEBUG::2014-04-27 12:43:55,101::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-223::DEBUG::2014-04-27 12:43:55,101::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,102::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-222::DEBUG::2014-04-27 12:43:55,102::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-224::DEBUG::2014-04-27 12:43:55,106::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-224::DEBUG::2014-04-27 12:43:55,107::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-224::DEBUG::2014-04-27 12:43:55,107::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-223::DEBUG::2014-04-27 12:43:55,107::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,108::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-223::DEBUG::2014-04-27 12:43:55,108::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,116::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
007f8290-5a3f-44ed-8802-c9a26c3718ae::WARNING::2014-04-27 12:43:55,116::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,117::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
    self.checkBackupDomain(__securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
    dom = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,128::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,128::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,142::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
007f8290-5a3f-44ed-8802-c9a26c3718ae::WARNING::2014-04-27 12:43:55,142::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,142::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,153::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f2cbe0e5-6641-4b51-a705-d9e9935275b6`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f2cbe0e5-6641-4b51-a705-d9e9935275b6`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`46da9ab1-a47d-4017-91d9-223eb4614818`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`46da9ab1-a47d-4017-91d9-223eb4614818`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,157::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,158::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,158::task::866::TaskManager.Task::(_setError) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
    self._updateDomainsRole()
  File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
    domain = sdCache.produce(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::885::TaskManager.Task::(_run) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task._run: 007f8290-5a3f-44ed-8802-c9a26c3718ae () {} failed - stopping task
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::1211::TaskManager.Task::(stop) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::stopping in state running (force False)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 1 aborting True
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::916::TaskManager.Task::(_runJobs) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 0 aborting True
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::925::TaskManager.Task::(_doAbort) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task._doAbort: force False
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state running -> state aborting
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::550::TaskManager.Task::(__state_aborting) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_aborting: recover policy auto
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state aborting -> state racquiring
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state racquiring -> state recovering
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::798::TaskManager.Task::(_recover) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_recover
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::805::TaskManager.Task::(_recover) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::running recovery None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::786::TaskManager.Task::(_recoverDone) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Recover Done: state recovering
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state recovering -> state recovered
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,161::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,161::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-29::ERROR::2014-04-27 12:43:55,963::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:55,964::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:55,964::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:55,964::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:55,977::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:55,978::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '  Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:55,978::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:55,988::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:55,989::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
    self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-13::DEBUG::2014-04-27 12:43:56,055::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,057::task::595::TaskManager.Task::(_updateState) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,059::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='007f8290-5a3f-44ed-8802-c9a26c3718ae', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:56,060::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:56,062::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}
Thread-13::INFO::2014-04-27 12:43:56,063::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}}
Thread-13::DEBUG::2014-04-27 12:43:56,065::task::1185::TaskManager.Task::(prepare) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}}
Thread-13::DEBUG::2014-04-27 12:43:56,066::task::595::TaskManager.Task::(_updateState) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,068::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,069::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,071::task::990::TaskManager.Task::(_decref) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:56,083::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,085::task::595::TaskManager.Task::(_updateState) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,087::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:56,089::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:56,091::task::1185::TaskManager.Task::(prepare) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:56,092::task::595::TaskManager.Task::(_updateState) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,094::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,097::task::990::TaskManager.Task::(_decref) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:56,146::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,149::task::595::TaskManager.Task::(_updateState) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,151::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='007f8290-5a3f-44ed-8802-c9a26c3718ae', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:56,152::taskManager::161::TaskManager::(clearTask) Entry. taskID: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:56,153::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:56,154::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-22::DEBUG::2014-04-27 12:43:56,155::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-04-27 12:43:56,159::task::1185::TaskManager.Task::(prepare) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:56,161::task::595::TaskManager.Task::(_updateState) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,163::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,166::task::990::TaskManager.Task::(_decref) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::ref 0 aborting False
Thread-22::DEBUG::2014-04-27 12:43:56,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 8.9833e-05 s, 7.4 MB/s\n'; <rc> = 0
Thread-17::DEBUG::2014-04-27 12:43:56,179::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:56,198::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.00062052 s, 601 kB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:56,290::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:56,295::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:56,302::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.00015041 s, 2.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:56,304::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 7.0876e-05 s, 5.2 MB/s\n'; <rc> = 0