[ovirt-users] Storage Domain Not Found After upgrade from 3.3.2 to 3.4
regpm at mccleary.me.uk
regpm at mccleary.me.uk
Sun Apr 27 08:10:27 EDT 2014
Hi,
Yes, you're correct, Gadi. I have a single Ovirt engine, which has two
Datacenters; one is local storage on the Engine server and the other is
local storage on the Ovirt Node. I've renamed the servers in the
attached vdsm log output (from the ovirt-node) to ovirt-engine
(10.50.0.18) and ovirt-node (10.50.0.19).
BDS_DataCentre1 is on the ovirt-engine server and this works fine after
the upgrade.
BDS_DataCentre2 is on the ovirt-node and this is the one that fails to
activate due to the storage domains not being accessible.
The Master storage domain on the ovirt-node is s2data1_s2mgt_boot1.
There are two other storage domains as well: s2data1_s2mgt_app1 and
s2data1_s2usr_boot1. The underlying filesystems are mounted fine, and
as I said, if I restore the server (root vg partitions; the storage
domain filesystems are not touched) then it works fine again. So the
upgrade is not playing nicely for some reason, but it's not clear to me
from the log what the issue is.
Thanks,
Paul
On 27/04/2014 07:56, Gadi Ickowicz wrote:
> Hi,
>
> Could you please attach the vdsm log as a file (it is easier to read) for the failing node?
>
> Also - I am a bit confused regarding what exactly your setup is - Do you have only a single engine (the all-in-one) and 2 dcs, one for the all in one and one for the oVirt node, which is the one that is failing?
>
> Thanks,
> Gadi Ickowicz
>
> ----- Original Message -----
> From: regpm at mccleary.me.uk
> To: users at ovirt.org
> Sent: Friday, April 25, 2014 10:46:35 PM
> Subject: [ovirt-users] Storage Domain Not Found After upgrade from 3.3.2 to 3.4
>
> Hi,
>
> I have an All-in-one installation engine node and an Ovirt node. They
> each use local storage and are thus configured in separate Data
> Centres. I upgraded both from 3.3.2 to 3.4 and this completed without
> error. I completed the engine-setup upgrade and this complete ok. I
> rebooted both the servers and the Ovirt engine node worked fine and I
> could start it's VMs. The Ovirt Node's datacenter is not activating,
> which seems to be due to none of the storage domains coming online.
> I've checked and the storage is mounted and available fine on the Ovirt
> node.
>
> Looking at the vdsm.log I can see errors stating that it can't find the
> storage domain. I have restored the entire node back to the pre-upgrade
> state and it works fine. It breaks again when I upgrade it. The
> approach I used was to put the Ovirt node in maintenance mode and run
> yum update. Anybody have similar issues or understand the log errors below?
>
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
> result = ctask.prepare(self.func, *args, **kwargs)
> File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
> return m(self, *a, **kw)
> File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
> raise self.error
> SecureError: Secured object is not in safe state
> Thread-180::DEBUG::2014-04-25
> 19:58:17,581::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
> flowID [21f7a1a7]
> Thread-180::DEBUG::2014-04-25
> 19:58:17,581::task::595::TaskManager.Task::(_updateState)
> Task=`545a6b49-bcd5-4e42-b64c-ee99329d3b7a`::moving from state init ->
> state preparing
> Thread-180::INFO::2014-04-25
> 19:58:17,582::logUtils::44::dispatcher::(wrapper) Run and protect:
> getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
> Thread-180::INFO::2014-04-25
> 19:58:17,582::logUtils::47::dispatcher::(wrapper) Run and protect:
> getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus':
> 'Free', 'spmLver': -1}}
> Thread-180::DEBUG::2014-04-25
> 19:58:17,582::task::1185::TaskManager.Task::(prepare)
> Task=`545a6b49-bcd5-4e42-b64c-ee99329d3b7a`::finished: {'spm_st':
> {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
> Thread-180::DEBUG::2014-04-25
> 19:58:17,582::task::595::TaskManager.Task::(_updateState)
> Task=`545a6b49-bcd5-4e42-b64c-ee99329d3b7a`::moving from state preparing
> -> state finished
> Thread-180::DEBUG::2014-04-25
> 19:58:17,582::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-180::DEBUG::2014-04-25
> 19:58:17,582::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-180::DEBUG::2014-04-25
> 19:58:17,582::task::990::TaskManager.Task::(_decref)
> Task=`545a6b49-bcd5-4e42-b64c-ee99329d3b7a`::ref 0 aborting False
> Thread-180::DEBUG::2014-04-25
> 19:58:17,732::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
> flowID [21f7a1a7]
> Thread-180::DEBUG::2014-04-25
> 19:58:17,733::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state init ->
> state preparing
> Thread-180::INFO::2014-04-25
> 19:58:17,733::logUtils::44::dispatcher::(wrapper) Run and protect:
> spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1,
> prevLVER='-1', maxHostID=250, domVersion='3', options=None)
> Thread-180::DEBUG::2014-04-25
> 19:58:17,733::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`ce97fd39-373c-4464-9921-2180f0f2e2c5`::Request
> was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
> Thread-180::DEBUG::2014-04-25
> 19:58:17,733::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
> Thread-180::DEBUG::2014-04-25
> 19:58:17,733::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now
> locking as 'exclusive' (1 active user)
> Thread-180::DEBUG::2014-04-25
> 19:58:17,733::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`ce97fd39-373c-4464-9921-2180f0f2e2c5`::Granted
> request
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::827::TaskManager.Task::(resourceAcquired)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::_resourcesAcquired:
> Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::990::TaskManager.Task::(_decref)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::ref 1 aborting False
> Thread-180::INFO::2014-04-25
> 19:58:17,734::logUtils::47::dispatcher::(wrapper) Run and protect:
> spmStart, Return response: None
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::1180::TaskManager.Task::(prepare)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Prepare: 1 jobs exist, move
> to acquiring
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state preparing
> -> state acquiring
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state acquiring
> -> state queued
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::taskManager::50::TaskManager::(_queueTask) queuing task:
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::taskManager::56::TaskManager::(_queueTask) task queued:
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97
> Thread-180::DEBUG::2014-04-25
> 19:58:17,734::task::1182::TaskManager.Task::(prepare)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::returning
> Thread-180::DEBUG::2014-04-25
> 19:58:17,735::task::990::TaskManager.Task::(_decref)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::ref 0 aborting False
> Thread-2::DEBUG::2014-04-25
> 19:58:17,735::threadPool::57::Misc.ThreadPool::(setRunningTask) Number
> of running tasks: 1
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,735::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask)
> Task: b9623891-cda5-4ac3-b4ab-efde3cc5ac97 running: <bound method
> Task.commit of <storage.task.Task instance at 0x7f95a41007e8>> with: None
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,735::task::1194::TaskManager.Task::(commit)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::committing task:
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,735::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state queued ->
> state running
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,736::task::905::TaskManager.Task::(_runJobs)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Task.run: running job 0:
> spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool
> object at 0x7f95a4195cd0>> (args: (-1, '-1', 250, 3) kwargs: {})
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,736::task::331::TaskManager.Task::(run)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Job.run: running spmStart:
> <bound method StoragePool.startSpm of <storage.sp.StoragePool object at
> 0x7f95a4195cd0>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,736::misc::762::SamplingMethod::(__call__) Trying to enter
> sampling method (storage.sp.updateMonitoringThreads)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,736::misc::764::SamplingMethod::(__call__) Got in to sampling
> method
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,746::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,746::misc::772::SamplingMethod::(__call__) Returning last result
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,746::clusterlock::337::LocalLock::(acquireHostId) Host id for
> domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::INFO::2014-04-25
> 19:58:17,746::clusterlock::365::LocalLock::(acquire) Acquiring local
> lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,746::clusterlock::400::LocalLock::(acquire) Local lock for
> domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,746::sp::266::Storage.StoragePool::(startSpm) spm lock acquired
> successfully
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,756::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,756::persistentDict::167::Storage.PersistentDict::(transaction)
> Starting transaction
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,756::persistentDict::173::Storage.PersistentDict::(transaction)
> Flushing changes
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,756::persistentDict::299::Storage.PersistentDict::(flush) about
> to write lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,757::persistentDict::175::Storage.PersistentDict::(transaction)
> Finished transaction
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,757::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`479828cf-c60b-465a-96f8-edc76af93804`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,757::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type
> 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,757::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free.
> Now locking as 'exclusive' (1 active user)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,758::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`479828cf-c60b-465a-96f8-edc76af93804`::Granted
> request
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::INFO::2014-04-25
> 19:58:17,758::sp::431::Storage.StoragePool::(_upgradePool) Trying to
> upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,759::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`cb5a1648-5020-4509-9d9e-60db98cf7c75`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,759::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,759::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now
> locking as 'exclusive' (1 active user)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,759::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`cb5a1648-5020-4509-9d9e-60db98cf7c75`::Granted
> request
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,761::fileSD::140::Storage.StorageDomain::(__init__) Reading
> domain in path
> /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,761::persistentDict::192::Storage.PersistentDict::(__init__)
> Created a persistent dict with FileMetadataRW backend
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,770::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,771::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing
> remnants of deleted images []
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::INFO::2014-04-25
> 19:58:17,771::sd::374::Storage.StorageDomain::(_registerResourceNamespaces)
> Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already
> registered
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::INFO::2014-04-25
> 19:58:17,771::sd::382::Storage.StorageDomain::(_registerResourceNamespaces)
> Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already
> registered
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,772::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,772::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0
> active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,772::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding
> out if anyone is waiting for it.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,772::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,772::sp::436::Storage.StoragePool::(_upgradePool) Marking all
> domains for upgrade
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,781::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,782::sp::443::Storage.StoragePool::(_upgradePool) Registering
> with state change event
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,782::sp::446::Storage.StoragePool::(_upgradePool) Running
> initial domain upgrade threads
> Thread-857::DEBUG::2014-04-25
> 19:58:17,782::sp::152::Storage.StoragePool::(_upgradePoolDomain)
> Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
> Thread-858::DEBUG::2014-04-25
> 19:58:17,782::sp::152::Storage.StoragePool::(_upgradePoolDomain)
> Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
> Thread-857::DEBUG::2014-04-25
> 19:58:17,782::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`c50b5f1d-d9fd-4317-bc98-4d40fad3e553`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '163' at
> '_upgradePoolDomain'
> Thread-858::DEBUG::2014-04-25
> 19:58:17,783::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`89314c30-0d9e-4b1d-b5f9-1ab61127274d`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '163' at
> '_upgradePoolDomain'
> Thread-859::DEBUG::2014-04-25
> 19:58:17,783::sp::152::Storage.StoragePool::(_upgradePoolDomain)
> Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
> Thread-857::DEBUG::2014-04-25
> 19:58:17,783::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type
> 'exclusive'
> Thread-860::ERROR::2014-04-25
> 19:58:17,783::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-859::DEBUG::2014-04-25
> 19:58:17,784::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`40a09482-c4be-472b-ae79-0b62c4640b5b`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '163' at
> '_upgradePoolDomain'
> Thread-858::DEBUG::2014-04-25
> 19:58:17,784::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type
> 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,784::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
> Thread-857::DEBUG::2014-04-25
> 19:58:17,784::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free.
> Now locking as 'exclusive' (1 active user)
> Thread-860::ERROR::2014-04-25
> 19:58:17,784::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-859::DEBUG::2014-04-25
> 19:58:17,784::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type
> 'exclusive'
> Thread-857::DEBUG::2014-04-25
> 19:58:17,785::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`c50b5f1d-d9fd-4317-bc98-4d40fad3e553`::Granted
> request
> Thread-860::DEBUG::2014-04-25
> 19:58:17,785::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' got the operation mutex
> Thread-858::DEBUG::2014-04-25
> 19:58:17,785::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free.
> Now locking as 'exclusive' (1 active user)
> Thread-857::DEBUG::2014-04-25
> 19:58:17,786::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a81f0cf3-f66a-4620-a380-9776968f5686`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '165' at
> '_upgradePoolDomain'
> Thread-860::DEBUG::2014-04-25
> 19:58:17,786::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
> /sbin/lvm vgs --config " devices { preferred_names =
> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
> \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1
> wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } "
> --noheadings --units b --nosuffix --separator | -o
> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
> 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
> Thread-858::DEBUG::2014-04-25
> 19:58:17,786::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`89314c30-0d9e-4b1d-b5f9-1ab61127274d`::Granted
> request
> Thread-857::DEBUG::2014-04-25
> 19:58:17,786::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,791::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
> (0 active users)
> Thread-858::DEBUG::2014-04-25
> 19:58:17,791::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`c5d0a1e7-9ebd-4175-86dd-30ef0e8f0a5c`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '165' at
> '_upgradePoolDomain'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,791::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free,
> finding out if anyone is waiting for it.
> Thread-858::DEBUG::2014-04-25
> 19:58:17,791::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,792::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
> Thread-859::DEBUG::2014-04-25
> 19:58:17,792::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free.
> Now locking as 'exclusive' (1 active user)
> Thread-859::DEBUG::2014-04-25
> 19:58:17,792::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`40a09482-c4be-472b-ae79-0b62c4640b5b`::Granted
> request
> Thread-857::DEBUG::2014-04-25
> 19:58:17,793::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now
> locking as 'exclusive' (1 active user)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,793::sd::623::Storage.StorageDomain::(createMasterTree)
> creating vms dir:
> /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
> Thread-859::DEBUG::2014-04-25
> 19:58:17,793::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`47c69548-2b50-4d71-98f1-4bd3a70edaff`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '165' at
> '_upgradePoolDomain'
> Thread-859::DEBUG::2014-04-25
> 19:58:17,793::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
> Thread-857::DEBUG::2014-04-25
> 19:58:17,793::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a81f0cf3-f66a-4620-a380-9776968f5686`::Granted
> request
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,794::sd::626::Storage.StorageDomain::(createMasterTree)
> creating task dir:
> /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
> Thread-858::DEBUG::2014-04-25
> 19:58:17,794::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now
> locking as 'exclusive' (1 active user)
> Thread-857::DEBUG::2014-04-25
> 19:58:17,794::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain
> `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2
> domain, not upgrading
> Thread-858::DEBUG::2014-04-25
> 19:58:17,795::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`c5d0a1e7-9ebd-4175-86dd-30ef0e8f0a5c`::Granted
> request
> Thread-860::DEBUG::2014-04-25
> 19:58:17,800::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
> Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
> Thread-857::DEBUG::2014-04-25
> 19:58:17,804::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
> Thread-859::DEBUG::2014-04-25
> 19:58:17,804::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now
> locking as 'exclusive' (1 active user)
> Thread-859::DEBUG::2014-04-25
> 19:58:17,807::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`47c69548-2b50-4d71-98f1-4bd3a70edaff`::Granted
> request
> Thread-860::WARNING::2014-04-25
> 19:58:17,808::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
> [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,809::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
> Thread-857::DEBUG::2014-04-25
> 19:58:17,810::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0
> active users)
> Thread-857::DEBUG::2014-04-25
> 19:58:17,811::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding
> out if anyone is waiting for it.
> Thread-857::DEBUG::2014-04-25
> 19:58:17,813::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
> Thread-857::DEBUG::2014-04-25
> 19:58:17,813::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
> Thread-857::DEBUG::2014-04-25
> 19:58:17,814::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
> (0 active users)
> Thread-857::DEBUG::2014-04-25
> 19:58:17,814::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free,
> finding out if anyone is waiting for it.
> Thread-857::DEBUG::2014-04-25
> 19:58:17,814::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
> Thread-860::DEBUG::2014-04-25
> 19:58:17,820::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' released the operation mutex
> Thread-858::DEBUG::2014-04-25
> 19:58:17,826::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular',
> 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
> Thread-859::DEBUG::2014-04-25
> 19:58:17,827::persistentDict::234::Storage.PersistentDict::(refresh)
> read lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3',
> 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5',
> 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular',
> 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
> Thread-858::DEBUG::2014-04-25
> 19:58:17,831::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
> Thread-859::DEBUG::2014-04-25
> 19:58:17,832::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
> Thread-858::DEBUG::2014-04-25
> 19:58:17,834::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0
> active users)
> Thread-858::DEBUG::2014-04-25
> 19:58:17,836::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding
> out if anyone is waiting for it.
> Thread-858::DEBUG::2014-04-25
> 19:58:17,838::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
> Thread-860::ERROR::2014-04-25
> 19:58:17,838::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> Thread-859::DEBUG::2014-04-25
> 19:58:17,839::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0
> active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,840::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-859::DEBUG::2014-04-25
> 19:58:17,840::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding
> out if anyone is waiting for it.
> Thread-858::DEBUG::2014-04-25
> 19:58:17,841::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,841::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-859::DEBUG::2014-04-25
> 19:58:17,842::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
> Thread-858::DEBUG::2014-04-25
> 19:58:17,843::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
> (0 active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,843::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' got the operation mutex
> Thread-858::DEBUG::2014-04-25
> 19:58:17,844::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free,
> finding out if anyone is waiting for it.
> Thread-859::DEBUG::2014-04-25
> 19:58:17,844::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
> Thread-858::DEBUG::2014-04-25
> 19:58:17,845::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
> Thread-859::DEBUG::2014-04-25
> 19:58:17,846::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
> (0 active users)
> Thread-859::DEBUG::2014-04-25
> 19:58:17,847::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free,
> finding out if anyone is waiting for it.
> Thread-859::DEBUG::2014-04-25
> 19:58:17,848::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,848::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
> /sbin/lvm vgs --config " devices { preferred_names =
> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
> \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1
> wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } "
> --noheadings --units b --nosuffix --separator | -o
> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
> 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,862::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
> Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::WARNING::2014-04-25
> 19:58:17,863::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
> [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,863::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' released the operation mutex
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,873::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,874::sp::288::Storage.StoragePool::(startSpm) Backup domain
> validation failed
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
> self.checkBackupDomain(__securityOverride=True)
> File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
> return method(self, *args, **kwargs)
> File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
> dom = sdCache.produce(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,874::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,874::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,874::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' got the operation mutex
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,875::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
> /sbin/lvm vgs --config " devices { preferred_names =
> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
> \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1
> wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } "
> --noheadings --units b --nosuffix --separator | -o
> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
> 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,888::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
> Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::WARNING::2014-04-25
> 19:58:17,889::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
> [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,889::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' released the operation mutex
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,899::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,899::sp::329::Storage.StoragePool::(startSpm) Unexpected error
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
> self._updateDomainsRole()
> File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
> return method(self, *args, **kwargs)
> File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
> domain = sdCache.produce(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,900::sp::330::Storage.StoragePool::(startSpm) failed: Storage
> domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,900::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting
> down upgrade process
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,900::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1d51aee8-4f09-4c7d-9fad-8b13801dce28`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,900::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type
> 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,900::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free.
> Now locking as 'exclusive' (1 active user)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,901::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1d51aee8-4f09-4c7d-9fad-8b13801dce28`::Granted
> request
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,901::resourceManager::198::ResourceManager.Request::(__init__)
> ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`7b50d1d9-ac64-4a8a-93df-3faee4744b2b`::Request
> was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,901::resourceManager::542::ResourceManager::(registerResource)
> Trying to register resource
> 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type
> 'exclusive'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,901::resourceManager::601::ResourceManager::(registerResource)
> Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free.
> Now locking as 'exclusive' (1 active user)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,901::resourceManager::238::ResourceManager.Request::(grant)
> ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`7b50d1d9-ac64-4a8a-93df-3faee4744b2b`::Granted
> request
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
> (0 active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free,
> finding out if anyone is waiting for it.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
> (0 active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free,
> finding out if anyone is waiting for it.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,902::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,903::persistentDict::167::Storage.PersistentDict::(transaction)
> Starting transaction
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,903::persistentDict::173::Storage.PersistentDict::(transaction)
> Flushing changes
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,903::persistentDict::299::Storage.PersistentDict::(flush) about
> to write lines (FileMetadataRW)=['CLASS=Data',
> 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10',
> 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=',
> 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1',
> 'POOL_DESCRIPTION=BDS_DataCentre2',
> 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active',
> 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0',
> 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002',
> 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master',
> 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS',
> 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,904::persistentDict::175::Storage.PersistentDict::(transaction)
> Finished transaction
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::INFO::2014-04-25
> 19:58:17,904::clusterlock::409::LocalLock::(release) Releasing local
> lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,904::clusterlock::422::LocalLock::(release) Local lock for
> domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::ERROR::2014-04-25
> 19:58:17,905::task::866::TaskManager.Task::(_setError)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Unexpected error
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/task.py", line 873, in _run
> return fn(*args, **kargs)
> File "/usr/share/vdsm/storage/task.py", line 334, in run
> return self.cmd(*self.argslist, **self.argsdict)
> File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
> self._updateDomainsRole()
> File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
> return method(self, *args, **kwargs)
> File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
> domain = sdCache.produce(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,905::task::885::TaskManager.Task::(_run)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Task._run:
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97 () {} failed - stopping task
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,905::task::1211::TaskManager.Task::(stop)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::stopping in state running
> (force False)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,905::task::990::TaskManager.Task::(_decref)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::ref 1 aborting True
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,905::task::916::TaskManager.Task::(_runJobs)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::aborting: Task is aborted:
> 'Storage domain does not exist' - code 358
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,905::task::990::TaskManager.Task::(_decref)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::ref 0 aborting True
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::925::TaskManager.Task::(_doAbort)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Task._doAbort: force False
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state running
> -> state aborting
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::550::TaskManager.Task::(__state_aborting)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::_aborting: recover policy auto
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state aborting
> -> state racquiring
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state
> racquiring -> state recovering
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::798::TaskManager.Task::(_recover)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::_recover
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::805::TaskManager.Task::(_recover)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::running recovery None
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,906::task::786::TaskManager.Task::(_recoverDone)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::Recover Done: state recovering
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::task::595::TaskManager.Task::(_updateState)
> Task=`b9623891-cda5-4ac3-b4ab-efde3cc5ac97`::moving from state
> recovering -> state recovered
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources
> {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef
> 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj:
> 'None'>}
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::616::ResourceManager::(releaseResource)
> Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::635::ResourceManager::(releaseResource)
> Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0
> active users)
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::641::ResourceManager::(releaseResource)
> Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding
> out if anyone is waiting for it.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::resourceManager::649::ResourceManager::(releaseResource)
> No one is waiting for resource
> 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97::DEBUG::2014-04-25
> 19:58:17,907::threadPool::57::Misc.ThreadPool::(setRunningTask) Number
> of running tasks: 0
> Thread-180::DEBUG::2014-04-25
> 19:58:18,739::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
> flowID [21f7a1a7]
> Thread-180::DEBUG::2014-04-25
> 19:58:18,739::task::595::TaskManager.Task::(_updateState)
> Task=`c57b61b2-fa6d-437d-b77f-694dda11efee`::moving from state init ->
> state preparing
> Thread-180::INFO::2014-04-25
> 19:58:18,739::logUtils::44::dispatcher::(wrapper) Run and protect:
> getTaskStatus(taskID='b9623891-cda5-4ac3-b4ab-efde3cc5ac97',
> spUUID=None, options=None)
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::taskManager::93::TaskManager::(getTaskStatus) Entry.
> taskID: b9623891-cda5-4ac3-b4ab-efde3cc5ac97
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::taskManager::96::TaskManager::(getTaskStatus) Return.
> Response: {'code': 358, 'message': 'Storage domain does not exist',
> 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID':
> 'b9623891-cda5-4ac3-b4ab-efde3cc5ac97'}
> Thread-180::INFO::2014-04-25
> 19:58:18,740::logUtils::47::dispatcher::(wrapper) Run and protect:
> getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message':
> 'Storage domain does not exist', 'taskState': 'finished', 'taskResult':
> 'cleanSuccess', 'taskID': 'b9623891-cda5-4ac3-b4ab-efde3cc5ac97'}}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::task::1185::TaskManager.Task::(prepare)
> Task=`c57b61b2-fa6d-437d-b77f-694dda11efee`::finished: {'taskStatus':
> {'code': 358, 'message': 'Storage domain does not exist', 'taskState':
> 'finished', 'taskResult': 'cleanSuccess', 'taskID':
> 'b9623891-cda5-4ac3-b4ab-efde3cc5ac97'}}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::task::595::TaskManager.Task::(_updateState)
> Task=`c57b61b2-fa6d-437d-b77f-694dda11efee`::moving from state preparing
> -> state finished
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,740::task::990::TaskManager.Task::(_decref)
> Task=`c57b61b2-fa6d-437d-b77f-694dda11efee`::ref 0 aborting False
> Thread-180::DEBUG::2014-04-25
> 19:58:18,745::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
> flowID [21f7a1a7]
> Thread-180::DEBUG::2014-04-25
> 19:58:18,745::task::595::TaskManager.Task::(_updateState)
> Task=`e3e1bdc1-e68d-40b1-ada1-229a29416c27`::moving from state init ->
> state preparing
> Thread-180::INFO::2014-04-25
> 19:58:18,746::logUtils::44::dispatcher::(wrapper) Run and protect:
> getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
> Thread-180::INFO::2014-04-25
> 19:58:18,746::logUtils::47::dispatcher::(wrapper) Run and protect:
> getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus':
> 'Free', 'spmLver': -1}}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,746::task::1185::TaskManager.Task::(prepare)
> Task=`e3e1bdc1-e68d-40b1-ada1-229a29416c27`::finished: {'spm_st':
> {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,746::task::595::TaskManager.Task::(_updateState)
> Task=`e3e1bdc1-e68d-40b1-ada1-229a29416c27`::moving from state preparing
> -> state finished
> Thread-180::DEBUG::2014-04-25
> 19:58:18,746::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,746::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,746::task::990::TaskManager.Task::(_decref)
> Task=`e3e1bdc1-e68d-40b1-ada1-229a29416c27`::ref 0 aborting False
> Thread-180::DEBUG::2014-04-25
> 19:58:18,790::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
> flowID [21f7a1a7]
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::task::595::TaskManager.Task::(_updateState)
> Task=`bc8109f7-7ca9-4900-afc2-653120a0f0b7`::moving from state init ->
> state preparing
> Thread-180::INFO::2014-04-25
> 19:58:18,791::logUtils::44::dispatcher::(wrapper) Run and protect:
> clearTask(taskID='b9623891-cda5-4ac3-b4ab-efde3cc5ac97', spUUID=None,
> options=None)
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::taskManager::161::TaskManager::(clearTask) Entry. taskID:
> b9623891-cda5-4ac3-b4ab-efde3cc5ac97
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::taskManager::166::TaskManager::(clearTask) Return.
> Thread-180::INFO::2014-04-25
> 19:58:18,791::logUtils::47::dispatcher::(wrapper) Run and protect:
> clearTask, Return response: None
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::task::1185::TaskManager.Task::(prepare)
> Task=`bc8109f7-7ca9-4900-afc2-653120a0f0b7`::finished: None
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::task::595::TaskManager.Task::(_updateState)
> Task=`bc8109f7-7ca9-4900-afc2-653120a0f0b7`::moving from state preparing
> -> state finished
> Thread-180::DEBUG::2014-04-25
> 19:58:18,791::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,792::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-180::DEBUG::2014-04-25
> 19:58:18,792::task::990::TaskManager.Task::(_decref)
> Task=`bc8109f7-7ca9-4900-afc2-653120a0f0b7`::ref 0 aborting False
> Thread-22::DEBUG::2014-04-25
> 19:58:24,268::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd
> iflag=direct
> if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata
> bs=4096 count=1' (cwd None)
> Thread-22::DEBUG::2014-04-25
> 19:58:24,272::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
> <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied,
> 0.000127032 s, 5.2 MB/s\n'; <rc> = 0
> Thread-29::ERROR::2014-04-25
> 19:58:24,475::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-29::ERROR::2014-04-25
> 19:58:24,475::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> Thread-29::DEBUG::2014-04-25
> 19:58:24,475::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' got the operation mutex
> Thread-29::DEBUG::2014-04-25
> 19:58:24,476::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
> /sbin/lvm vgs --config " devices { preferred_names =
> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
> \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1
> wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } "
> --noheadings --units b --nosuffix --separator | -o
> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
> 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
> Thread-29::DEBUG::2014-04-25
> 19:58:24,489::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = '
> Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
> Thread-29::WARNING::2014-04-25
> 19:58:24,490::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
> [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', '
> Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
> Thread-29::DEBUG::2014-04-25
> 19:58:24,490::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' released the operation mutex
> Thread-29::ERROR::2014-04-25
> 19:58:24,501::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> Thread-29::ERROR::2014-04-25
> 19:58:24,501::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
> Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
> monitoring information
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in
> _monitorDomain
> self.domain = sdCache.produce(self.sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
> Thread-27::DEBUG::2014-04-25
> 19:58:24,628::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd
> iflag=direct
> if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata
> bs=4096 count=1' (cwd None)
> Thread-27::DEBUG::2014-04-25
> 19:58:24,633::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
> <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied,
> 0.000137914 s, 2.7 MB/s\n'; <rc> = 0
> Thread-18::DEBUG::2014-04-25
> 19:58:24,689::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd
> iflag=direct
> if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata
> bs=4096 count=1' (cwd None)
> Thread-18::DEBUG::2014-04-25
> 19:58:24,694::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
> <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied,
> 0.000117437 s, 3.1 MB/s\n'; <rc> = 0
> Thread-17::DEBUG::2014-04-25
> 19:58:25,365::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd
> iflag=direct
> if=/rhev/data-center/mnt/bdsvn001d:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata
> bs=4096 count=1' (cwd None)
> Thread-17::DEBUG::2014-04-25
> 19:58:25,370::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
> <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied,
> 0.000802138 s, 465 kB/s\n'; <rc> = 0
>
>
> Thanks, Paul
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
-------------- next part --------------
MainThread::INFO::2014-04-27 12:41:39,581::vdsm::119::vds::(run) (PID: 3754) I am the actual vdsm 4.14.6-0.el6 ovirt-node.example.local (2.6.32-431.11.2.el6.x86_64)
MainThread::DEBUG::2014-04-27 12:41:39,597::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'Storage'
MainThread::DEBUG::2014-04-27 12:41:39,597::threadPool::35::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0
MainThread::WARNING::2014-04-27 12:41:39,609::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt already exists
MainThread::DEBUG::2014-04-27 12:41:40,149::supervdsm::77::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
MainThread::DEBUG::2014-04-27 12:41:40,160::multipath::142::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving
MainThread::DEBUG::2014-04-27 12:41:40,160::hsm::423::Storage.Misc.excCmd::(__validateLvmLockingType) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None)
MainThread::DEBUG::2014-04-27 12:41:40,170::hsm::423::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-04-27 12:41:40,171::hsm::447::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center'
MainThread::DEBUG::2014-04-27 12:41:40,176::hsm::479::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt']
MainThread::DEBUG::2014-04-27 12:41:40,176::hsm::480::Storage.HSM::(__cleanStorageRepository) Mount list: []
MainThread::DEBUG::2014-04-27 12:41:40,177::hsm::482::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers
MainThread::DEBUG::2014-04-27 12:41:40,390::hsm::525::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center'
storageRefresh::DEBUG::2014-04-27 12:41:40,391::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
MainThread::INFO::2014-04-27 12:41:40,391::dispatcher::97::Storage.Dispatcher::(__init__) Starting StorageDispatcher...
storageRefresh::DEBUG::2014-04-27 12:41:40,392::misc::764::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-04-27 12:41:40,585::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
storageRefresh::DEBUG::2014-04-27 12:41:40,585::misc::764::SamplingMethod::(__call__) Got in to sampling method
storageRefresh::DEBUG::2014-04-27 12:41:40,585::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
storageRefresh::DEBUG::2014-04-27 12:41:40,585::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,666::misc::772::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-04-27 12:41:40,666::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
MainThread::DEBUG::2014-04-27 12:41:40,737::task::595::TaskManager.Task::(_updateState) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::moving from state init -> state preparing
MainThread::INFO::2014-04-27 12:41:40,737::logUtils::44::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=<bound method clientIF.contEIOVms of <clientIF.clientIF instance at 0x1d00b90>>)
MainThread::INFO::2014-04-27 12:41:40,737::logUtils::47::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None
storageRefresh::DEBUG::2014-04-27 12:41:40,737::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2014-04-27 12:41:40,738::task::1185::TaskManager.Task::(prepare) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::finished: None
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::task::595::TaskManager.Task::(_updateState) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::moving from state preparing -> state finished
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
storageRefresh::DEBUG::2014-04-27 12:41:40,738::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,738::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
MainThread::DEBUG::2014-04-27 12:41:40,739::task::990::TaskManager.Task::(_decref) Task=`192e75e7-7bc4-4f39-8f15-e13687a61d5c`::ref 0 aborting False
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,739::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,740::misc::772::SamplingMethod::(__call__) Returning last result
storageRefresh::DEBUG::2014-04-27 12:41:40,740::lvm::324::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,741::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
MainThread::INFO::2014-04-27 12:41:40,747::momIF::47::MOM::(__init__) Starting up MOM
storageRefresh::DEBUG::2014-04-27 12:41:40,752::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::349::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,753::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
MainThread::INFO::2014-04-27 12:41:40,758::vmChannels::187::vds::(settimeout) Setting channels' timeout to 30 seconds.
VM Channels Listener::INFO::2014-04-27 12:41:40,758::vmChannels::170::vds::(run) Starting VM channels listener thread.
clientIFinit::DEBUG::2014-04-27 12:41:40,838::libvirtconnection::145::root::(get) trying to connect libvirt
MainThread::WARNING::2014-04-27 12:41:40,871::clientIF::181::vds::(_prepareBindings) Unable to load the json rpc server module. Please make sure it is installed.
storageRefresh::DEBUG::2014-04-27 12:41:40,872::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,873::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,876::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,925::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,926::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
storageRefresh::DEBUG::2014-04-27 12:41:40,927::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name' (cwd None)
storageRefresh::DEBUG::2014-04-27 12:41:40,940::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No volume groups found\n'; <rc> = 0
storageRefresh::DEBUG::2014-04-27 12:41:40,941::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
storageRefresh::WARNING::2014-04-27 12:41:40,942::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/hsm-tasks already exists
storageRefresh::DEBUG::2014-04-27 12:41:40,958::hsm::391::Storage.HSM::(storageRefresh) HSM is ready
Thread-13::DEBUG::2014-04-27 12:42:02,760::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getCapabilities with () {}
Thread-13::DEBUG::2014-04-27 12:42:02,800::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:02,806::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:03,101::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395777595.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el6', 'buildtime': 1395806448L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'bkupnet': {'iface': 'bkupnet', 'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'gateway': '15.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth3']}, 'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'gateway': '10.50.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}, 'vmnet': {'iface': 'vmnet', 'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'gateway': '12.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth2']}}, 'bridges': {'ovirtmgmt': {'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '10.50.0.1', 'ports': ['eth0']}, 'bkupnet': {'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '15.0.0.12', 'ports': ['eth3']}, 'vmnet': {'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '12.0.0.12', 'ports': ['eth2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth3': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c8', 'cfg': {'BRIDGE': 'bkupnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c8', 'STP': 'no', 'DEVICE': 'eth3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'speed': 1000, 'mtu': '1500'}, 'eth2': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c6', 'cfg': {'BRIDGE': 'vmnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c6', 'STP': 'no', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'speed': 1000, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:19:99:ff:d5:85', 'cfg': {'UUID': '469d4529-920d-46ae-9441-786d12b85eb7', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:19:99:FF:D5:85', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '90:1b:0e:06:f9:39', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'HWADDR': '90:1b:0e:06:f9:39', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'speed': 100, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15917', 'cpuSpeed': '3092.855', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '10.50.0.18'}}
Thread-13::DEBUG::2014-04-27 12:42:03,246::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getHardwareInfo with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,324::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PRIMERGY TX100 S3P', 'systemSerialNumber': 'YLLD033823', 'systemFamily': '', 'systemVersion': '', 'systemUUID': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'systemManufacturer': 'FUJITSU'}}
Thread-13::DEBUG::2014-04-27 12:42:03,437::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getCapabilities with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,437::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:03,441::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:03,463::BindingXMLRPC::1074::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580'}], 'FC': []}, 'packages2': {'kernel': {'release': '431.11.2.el6.x86_64', 'buildtime': 1395777595.0, 'version': '2.6.32'}, 'spice-server': {'release': '6.el6_5.1', 'buildtime': 1386756528L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el6', 'buildtime': 1395806448L, 'version': '4.14.6'}, 'qemu-kvm': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'libvirt': {'release': '29.el6_5.7', 'buildtime': 1396856799L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.415.el6_5.8', 'buildtime': 1398190363L, 'version': '0.12.1.2'}, 'mom': {'release': '1.el6', 'buildtime': 1391183641L, 'version': '0.4.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'bkupnet': {'iface': 'bkupnet', 'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'gateway': '15.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth3']}, 'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'gateway': '10.50.0.1', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}, 'vmnet': {'iface': 'vmnet', 'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'gateway': '12.0.0.12', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth2']}}, 'bridges': {'ovirtmgmt': {'addr': '10.50.0.19', 'cfg': {'DEFROUTE': 'yes', 'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '10.50.0.1', 'ports': ['eth0']}, 'bkupnet': {'addr': '15.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '15.0.0.12', 'GATEWAY': '15.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'bkupnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '15.0.0.12', 'ports': ['eth3']}, 'vmnet': {'addr': '12.0.0.12', 'cfg': {'DEFROUTE': 'no', 'IPADDR': '12.0.0.12', 'GATEWAY': '12.0.0.12', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'vmnet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '12.0.0.12', 'ports': ['eth2']}, ';vdsmdummy;': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ipv6gateway': '::', 'gateway': '', 'ports': []}}, 'uuid': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth3': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c8', 'cfg': {'BRIDGE': 'bkupnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c8', 'STP': 'no', 'DEVICE': 'eth3', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c8/64'], 'speed': 1000, 'mtu': '1500'}, 'eth2': {'netmask': '', 'addr': '', 'hwaddr': '80:1f:02:00:26:c6', 'cfg': {'BRIDGE': 'vmnet', 'NM_CONTROLLED': 'no', 'HWADDR': '80:1f:02:00:26:c6', 'STP': 'no', 'DEVICE': 'eth2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::821f:2ff:fe00:26c6/64'], 'speed': 1000, 'mtu': '1500'}, 'eth1': {'netmask': '', 'addr': '', 'hwaddr': '00:19:99:ff:d5:85', 'cfg': {'UUID': '469d4529-920d-46ae-9441-786d12b85eb7', 'NM_CONTROLLED': 'yes', 'HWADDR': '00:19:99:FF:D5:85', 'BOOTPROTO': 'dhcp', 'DEVICE': 'eth1', 'TYPE': 'Ethernet', 'ONBOOT': 'no'}, 'ipv6addrs': [], 'speed': 0, 'mtu': '1500'}, 'eth0': {'netmask': '', 'addr': '', 'hwaddr': '90:1b:0e:06:f9:39', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'HWADDR': '90:1b:0e:06:f9:39', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::921b:eff:fe06:f939/64'], 'speed': 100, 'mtu': '1500'}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:99b375f5d580', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4'], 'reservedMem': '321', 'bondings': {'bond4': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond0': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond1': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond2': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}, 'bond3': {'netmask': '', 'addr': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500'}}, 'software_version': '4.14', 'memSize': '15917', 'cpuSpeed': '3092.855', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '0.0.0.0', 'cpuThreads': '4', 'emulatedMachines': [u'rhel6.5.0', u'pc', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '5.el6.centos.11.2', 'version': '6', 'name': 'RHEL'}, 'lastClient': '10.50.0.18'}}
Thread-13::DEBUG::2014-04-27 12:42:03,487::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call getHardwareInfo with () {}
Thread-13::DEBUG::2014-04-27 12:42:03,488::BindingXMLRPC::1074::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PRIMERGY TX100 S3P', 'systemSerialNumber': 'YLLD033823', 'systemFamily': '', 'systemVersion': '', 'systemUUID': '52D5CD73-1D0C-E311-A8C2-0B9FEBFD5C92', 'systemManufacturer': 'FUJITSU'}}
Thread-13::DEBUG::2014-04-27 12:42:04,684::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18] flowID [16b415b2]
Thread-13::DEBUG::2014-04-27 12:42:04,684::task::595::TaskManager.Task::(_updateState) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:04,684::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', conList=[{'port': '', 'connection': 'ovirt-noded:/iso', 'iqn': '', 'portal': '', 'user': '', 'protocol_version': '4', 'password': '******', 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}], options=None)
Thread-13::WARNING::2014-04-27 12:42:04,691::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/mnt/ovirt-noded:_iso already exists
Thread-13::DEBUG::2014-04-27 12:42:04,691::mount::226::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=4 ovirt-noded:/iso /rhev/data-center/mnt/ovirt-noded:_iso' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,051::hsm::2317::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/ovirt-noded:_iso
Thread-13::DEBUG::2014-04-27 12:42:05,054::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('7b083758-45f9-4896-913d-11fe02043e6e',)
Thread-13::DEBUG::2014-04-27 12:42:05,054::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain}
Thread-13::INFO::2014-04-27 12:42:05,054::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,054::task::1185::TaskManager.Task::(prepare) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::finished: {'statuslist': [{'status': 0, 'id': '2198bb7f-8044-4abd-823e-34cc5752e20b'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,054::task::595::TaskManager.Task::(_updateState) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:42:05,054::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:42:05,055::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:42:05,055::task::990::TaskManager.Task::(_decref) Task=`5c274b95-182c-466b-92da-a53d50dab79c`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:42:05,064::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18] flowID [16b415b2]
Thread-13::DEBUG::2014-04-27 12:42:05,064::task::595::TaskManager.Task::(_updateState) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:05,065::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=4, spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', conList=[{'connection': '/vdsm_store/s2data1_s2mgt_boot1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb', 'port': ''}, {'connection': '/vdsm_store/s2data1_s2mgt_app1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '779cabae-1b55-4186-a275-7e07878579df', 'port': ''}, {'connection': '/vdsm_store/s2data1_s2usr_boot1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe', 'port': ''}], options=None)
Thread-13::DEBUG::2014-04-27 12:42:05,067::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2mgt_boot1
Thread-13::DEBUG::2014-04-27 12:42:05,103::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('0a897f2e-1b01-4577-9f91-cd136ef4a978',)
Thread-13::DEBUG::2014-04-27 12:42:05,103::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain}
Thread-13::DEBUG::2014-04-27 12:42:05,106::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2mgt_app1
Thread-13::DEBUG::2014-04-27 12:42:05,116::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('c2c4ade6-049e-4159-a294-a0c151f4983d',)
Thread-13::DEBUG::2014-04-27 12:42:05,117::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain, c2c4ade6-049e-4159-a294-a0c151f4983d: storage.localFsSD.findDomain}
Thread-13::DEBUG::2014-04-27 12:42:05,119::hsm::2334::Storage.HSM::(__prefetchDomains) local _path: /vdsm_store/s2data1_s2usr_boot1
Thread-13::DEBUG::2014-04-27 12:42:05,133::hsm::2341::Storage.HSM::(__prefetchDomains) Found SD uuids: ('e637bb04-a8b7-4c77-809c-d58051494c52',)
Thread-13::DEBUG::2014-04-27 12:42:05,134::hsm::2397::Storage.HSM::(connectStorageServer) knownSDs: {7b083758-45f9-4896-913d-11fe02043e6e: storage.nfsSD.findDomain, e637bb04-a8b7-4c77-809c-d58051494c52: storage.localFsSD.findDomain, 0a897f2e-1b01-4577-9f91-cd136ef4a978: storage.localFsSD.findDomain, c2c4ade6-049e-4159-a294-a0c151f4983d: storage.localFsSD.findDomain}
Thread-13::INFO::2014-04-27 12:42:05,134::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb'}, {'status': 0, 'id': '779cabae-1b55-4186-a275-7e07878579df'}, {'status': 0, 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::1185::TaskManager.Task::(prepare) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::finished: {'statuslist': [{'status': 0, 'id': '0c37031e-43cd-4f90-8856-0cce1b5d75bb'}, {'status': 0, 'id': '779cabae-1b55-4186-a275-7e07878579df'}, {'status': 0, 'id': 'e86d99a1-dcba-4869-ac38-5d87052d8dbe'}]}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::595::TaskManager.Task::(_updateState) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:42:05,134::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:42:05,134::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:42:05,134::task::990::TaskManager.Task::(_decref) Task=`1784e391-2568-4e67-ae4b-6970c9eb39e5`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:42:05,193::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:42:05,193::task::595::TaskManager.Task::(_updateState) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:42:05,193::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', hostID=1, msdUUID='0a897f2e-1b01-4577-9f91-cd136ef4a978', masterVersion=1, domainsMap=None, options=None)
Thread-13::DEBUG::2014-04-27 12:42:05,193::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ecee1a91-309a-46e9-9ead-9d0ad4261f55`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1007' at 'connectStoragePool'
Thread-13::DEBUG::2014-04-27 12:42:05,193::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`ecee1a91-309a-46e9-9ead-9d0ad4261f55`::Granted request
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`196c5690-1da2-4687-9d13-18eee96e83e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1055' at '_connectStoragePool'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:42:05,194::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:42:05,195::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`196c5690-1da2-4687-9d13-18eee96e83e1`::Granted request
Thread-13::INFO::2014-04-27 12:42:05,195::sp::133::Storage.StoragePool::(setBackend) updating pool c713062f-300f-4256-9ac8-2d3fcfcdb002 backend from type NoneType instance 0x7fcd5b7fc210 to type StoragePoolDiskBackend instance 0x7fcd000eb200
Thread-13::INFO::2014-04-27 12:42:05,195::sp::618::Storage.StoragePool::(connect) Connect host #1 to the storage pool c713062f-300f-4256-9ac8-2d3fcfcdb002 with master domain: 0a897f2e-1b01-4577-9f91-cd136ef4a978 (ver = 1)
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,195::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-13::DEBUG::2014-04-27 12:42:05,196::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,196::iscsi::393::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-13::DEBUG::2014-04-27 12:42:05,197::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,204::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-04-27 12:42:05,204::multipath::110::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,291::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:42:05,291::lvm::496::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::498::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::507::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::509::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::527::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::lvm::529::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-13::DEBUG::2014-04-27 12:42:05,292::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::DEBUG::2014-04-27 12:42:05,468::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::DEBUG::2014-04-27 12:42:05,562::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::DEBUG::2014-04-27 12:42:05,604::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::DEBUG::2014-04-27 12:42:05,613::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-13::DEBUG::2014-04-27 12:42:05,614::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS'
Thread-13::DEBUG::2014-04-27 12:42:05,614::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS'
Thread-13::DEBUG::2014-04-27 12:42:05,614::sp::1366::Storage.StoragePool::(setMasterDomain) Master domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 verified, version 1
Thread-13::DEBUG::2014-04-27 12:42:05,614::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
Thread-13::DEBUG::2014-04-27 12:42:05,614::misc::764::SamplingMethod::(__call__) Got in to sampling method
Thread-13::DEBUG::2014-04-27 12:42:05,624::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::INFO::2014-04-27 12:42:05,624::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 7b083758-45f9-4896-913d-11fe02043e6e
Thread-13::DEBUG::2014-04-27 12:42:05,625::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `7b083758-45f9-4896-913d-11fe02043e6e`
Thread-17::DEBUG::2014-04-27 12:42:05,625::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 7b083758-45f9-4896-913d-11fe02043e6e
Thread-13::INFO::2014-04-27 12:42:05,625::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring e637bb04-a8b7-4c77-809c-d58051494c52
Thread-13::DEBUG::2014-04-27 12:42:05,626::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `e637bb04-a8b7-4c77-809c-d58051494c52`
Thread-18::DEBUG::2014-04-27 12:42:05,626::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for e637bb04-a8b7-4c77-809c-d58051494c52
Thread-13::INFO::2014-04-27 12:42:05,628::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::DEBUG::2014-04-27 12:42:05,635::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
Thread-22::DEBUG::2014-04-27 12:42:05,635::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-13::INFO::2014-04-27 12:42:05,636::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-13::DEBUG::2014-04-27 12:42:05,643::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `c2c4ade6-049e-4159-a294-a0c151f4983d`
Thread-13::INFO::2014-04-27 12:42:05,644::domainMonitor::99::Storage.DomainMonitor::(startMonitoring) Start monitoring 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-27::DEBUG::2014-04-27 12:42:05,645::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-13::DEBUG::2014-04-27 12:42:05,646::sp::1395::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `c713062f-300f-4256-9ac8-2d3fcfcdb002` started monitoring domain `9a4a80a1-5377-4a94-ade3-e58183e916ae`
Thread-13::DEBUG::2014-04-27 12:42:05,647::misc::772::SamplingMethod::(__call__) Returning last result
Thread-13::WARNING::2014-04-27 12:42:05,648::fileUtils::167::Storage.fileUtils::(createdir) Dir /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002 already exists
Thread-29::DEBUG::2014-04-27 12:42:05,648::domainMonitor::165::Storage.DomainMonitorThread::(_monitorLoop) Starting domain monitor for 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:42:05,654::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:42:05,655::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:42:05,655::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:42:05,656::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-13::DEBUG::2014-04-27 12:42:05,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
Thread-13::DEBUG::2014-04-27 12:42:05,666::lvm::372::OperationMutex::(_reloadvgs) Got the operational mutex
Thread-13::DEBUG::2014-04-27 12:42:05,675::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 7b083758-45f9-4896-913d-11fe02043e6e e637bb04-a8b7-4c77-809c-d58051494c52 c2c4ade6-049e-4159-a294-a0c151f4983d 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:05,698::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52
Thread-27::DEBUG::2014-04-27 12:42:05,727::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-29::DEBUG::2014-04-27 12:42:05,736::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-13::DEBUG::2014-04-27 12:42:05,739::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "7b083758-45f9-4896-913d-11fe02043e6e" not found\n Skipping volume group 7b083758-45f9-4896-913d-11fe02043e6e\n Volume group "e637bb04-a8b7-4c77-809c-d58051494c52" not found\n Skipping volume group e637bb04-a8b7-4c77-809c-d58051494c52\n Volume group "c2c4ade6-049e-4159-a294-a0c151f4983d" not found\n Skipping volume group c2c4ade6-049e-4159-a294-a0c151f4983d\n Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-22::DEBUG::2014-04-27 12:42:05,763::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:42:05,774::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e
Thread-22::DEBUG::2014-04-27 12:42:05,789::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.0139692 s, 47.6 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:05,802::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 changed its status to Valid
Thread-22::DEBUG::2014-04-27 12:42:05,802::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-33::DEBUG::2014-04-27 12:42:05,803::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-34::INFO::2014-04-27 12:42:05,804::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-27::DEBUG::2014-04-27 12:42:05,830::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-18::DEBUG::2014-04-27 12:42:05,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-13::WARNING::2014-04-27 12:42:05,855::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "7b083758-45f9-4896-913d-11fe02043e6e" not found', ' Skipping volume group 7b083758-45f9-4896-913d-11fe02043e6e', ' Volume group "e637bb04-a8b7-4c77-809c-d58051494c52" not found', ' Skipping volume group e637bb04-a8b7-4c77-809c-d58051494c52', ' Volume group "c2c4ade6-049e-4159-a294-a0c151f4983d" not found', ' Skipping volume group c2c4ade6-049e-4159-a294-a0c151f4983d', ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::WARNING::2014-04-27 12:42:05,855::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:42:05,858::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-18::DEBUG::2014-04-27 12:42:05,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-18::DEBUG::2014-04-27 12:42:05,967::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-18::DEBUG::2014-04-27 12:42:05,967::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e637bb04-a8b7-4c77-809c-d58051494c52_imageNS'
Thread-18::DEBUG::2014-04-27 12:42:05,967::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'e637bb04-a8b7-4c77-809c-d58051494c52_volumeNS'
Thread-18::DEBUG::2014-04-27 12:42:05,978::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:05,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-18::DEBUG::2014-04-27 12:42:05,985::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000404616 s, 912 kB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:05,986::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain e637bb04-a8b7-4c77-809c-d58051494c52 changed its status to Valid
Thread-48::DEBUG::2014-04-27 12:42:05,986::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-18::DEBUG::2014-04-27 12:42:05,986::clusterlock::337::LocalLock::(acquireHostId) Host id for domain e637bb04-a8b7-4c77-809c-d58051494c52 successfully acquired (id: 1)
Thread-48::DEBUG::2014-04-27 12:42:05,987::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-48::DEBUG::2014-04-27 12:42:05,987::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-27::DEBUG::2014-04-27 12:42:05,988::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-49::INFO::2014-04-27 12:42:05,988::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-27::DEBUG::2014-04-27 12:42:05,989::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'c2c4ade6-049e-4159-a294-a0c151f4983d_imageNS'
Thread-27::DEBUG::2014-04-27 12:42:05,989::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace 'c2c4ade6-049e-4159-a294-a0c151f4983d_volumeNS'
Thread-27::DEBUG::2014-04-27 12:42:06,002::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:06,006::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 8.4325e-05 s, 4.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:06,007::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain c2c4ade6-049e-4159-a294-a0c151f4983d changed its status to Valid
Thread-50::DEBUG::2014-04-27 12:42:06,007::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-27::DEBUG::2014-04-27 12:42:06,007::clusterlock::337::LocalLock::(acquireHostId) Host id for domain c2c4ade6-049e-4159-a294-a0c151f4983d successfully acquired (id: 1)
Thread-50::DEBUG::2014-04-27 12:42:06,008::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-50::DEBUG::2014-04-27 12:42:06,009::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-51::INFO::2014-04-27 12:42:06,009::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-22::DEBUG::2014-04-27 12:42:15,812::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:15,816::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000128207 s, 5.2 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:16,001::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:16,005::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000115492 s, 3.2 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:16,021::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:16,025::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 8.102e-05 s, 4.5 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:25,828::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:25,832::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000124018 s, 5.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:26,020::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:26,023::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 8.4621e-05 s, 4.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:26,039::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:26,043::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.2273e-05 s, 5.9 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:35,843::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:35,847::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000112682 s, 5.9 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:36,035::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:36,039::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000100668 s, 3.7 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:36,057::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:36,061::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.1211e-05 s, 6.0 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:45,859::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:45,862::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000105352 s, 6.3 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:46,054::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:46,058::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000130353 s, 2.8 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:46,075::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:46,078::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 9.7641e-05 s, 3.8 MB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:42:55,874::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:42:55,878::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000101233 s, 6.6 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:42:56,073::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:42:56,077::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000103071 s, 3.6 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:42:56,093::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:42:56,096::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 6.2305e-05 s, 5.9 MB/s\n'; <rc> = 0
Thread-17::ERROR::2014-04-27 12:43:05,778::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 7b083758-45f9-4896-913d-11fe02043e6e monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/nfsSD.py", line 132, in findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/fileSD.py", line 147, in __init__
validateFileSystemFeatures(sdUUID, self.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 76, in validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
*args, **kwargs)
File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
raise Timeout()
Timeout
Thread-13::INFO::2014-04-27 12:43:05,781::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/7b083758-45f9-4896-913d-11fe02043e6e
Thread-29::ERROR::2014-04-27 12:43:05,781::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:05,824::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 7b083758-45f9-4896-913d-11fe02043e6e changed its status to Invalid
Thread-13::INFO::2014-04-27 12:43:05,825::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/e637bb04-a8b7-4c77-809c-d58051494c52
Thread-29::ERROR::2014-04-27 12:43:05,825::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-52::DEBUG::2014-04-27 12:43:05,826::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-13::INFO::2014-04-27 12:43:05,826::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-17::DEBUG::2014-04-27 12:43:05,827::remoteFileHandler::260::RepoFileHelper.PoolHandler::(stop) Pool handler existed, OUT: '' ERR: ''
Thread-29::DEBUG::2014-04-27 12:43:05,827::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 9a4a80a1-5377-4a94-ade3-e58183e916ae changed its status to Invalid
Thread-52::DEBUG::2014-04-27 12:43:05,828::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-13::INFO::2014-04-27 12:43:05,829::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/0a897f2e-1b01-4577-9f91-cd136ef4a978
Thread-53::DEBUG::2014-04-27 12:43:05,829::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-52::DEBUG::2014-04-27 12:43:05,830::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-13::INFO::2014-04-27 12:43:05,831::sp::1104::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978 to /rhev/data-center/c713062f-300f-4256-9ac8-2d3fcfcdb002/mastersd
Thread-53::DEBUG::2014-04-27 12:43:05,831::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-53::DEBUG::2014-04-27 12:43:05,832::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:05,833::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:05,834::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-13::INFO::2014-04-27 12:43:05,834::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True
Thread-13::DEBUG::2014-04-27 12:43:05,834::task::1185::TaskManager.Task::(prepare) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::finished: True
Thread-13::DEBUG::2014-04-27 12:43:05,834::task::595::TaskManager.Task::(_updateState) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:05,835::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:05,835::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:05,835::task::990::TaskManager.Task::(_decref) Task=`c1b1c3a6-5af5-48fb-8f48-3630f4150a16`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:05,881::task::595::TaskManager.Task::(_updateState) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:05,882::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:05,884::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000103071', 'lastCheck': '9.8', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000101233', 'lastCheck': '10.0', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '6.2305e-05', 'lastCheck': '9.8', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:05,885::task::1185::TaskManager.Task::(prepare) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000103071', 'lastCheck': '9.8', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000101233', 'lastCheck': '10.0', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '6.2305e-05', 'lastCheck': '9.8', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.1', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:05,886::task::595::TaskManager.Task::(_updateState) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:05,886::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:05,887::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:05,888::task::990::TaskManager.Task::(_decref) Task=`59e9198b-d2d9-4487-8fe7-47833a98b9a7`::ref 0 aborting False
Thread-22::DEBUG::2014-04-27 12:43:05,915::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-04-27 12:43:05,920::BindingXMLRPC::1067::vds::(wrapper) client [10.50.0.18]::call setMOMPolicyParameters with ({'balloonEnabled': False, 'ksmEnabled': True},) {} flowID [16b415b2]
Thread-22::DEBUG::2014-04-27 12:43:05,926::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000113663 s, 5.9 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:05,974::BindingXMLRPC::1074::vds::(wrapper) return setMOMPolicyParameters with {'status': {'message': 'Done', 'code': 0}}
Thread-18::DEBUG::2014-04-27 12:43:06,088::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:06,092::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000119876 s, 3.1 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:06,110::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:06,114::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000118739 s, 3.1 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:09,620::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:09,620::task::595::TaskManager.Task::(_updateState) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:09,620::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:09,620::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::1185::TaskManager.Task::(prepare) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::595::TaskManager.Task::(_updateState) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:09,621::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:09,621::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:09,621::task::990::TaskManager.Task::(_decref) Task=`c715c9ae-eae8-4725-b689-8442e1ae7f04`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:09,654::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:09,654::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:09,654::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:09,654::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`8d3b4f80-acf6-4f5a-950e-d2ba4e192644`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:09,655::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`8d3b4f80-acf6-4f5a-950e-d2ba4e192644`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::827::TaskManager.Task::(resourceAcquired) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:09,655::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::1180::TaskManager.Task::(prepare) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:09,655::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:09,656::taskManager::50::TaskManager::(_queueTask) queuing task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:09,656::taskManager::56::TaskManager::(_queueTask) task queued: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::1182::TaskManager.Task::(prepare) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::returning
Thread-13::DEBUG::2014-04-27 12:43:09,656::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 0 aborting False
Thread-6::DEBUG::2014-04-27 12:43:09,656::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,656::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4 running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00189d88>> with: None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::1194::TaskManager.Task::(commit) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::committing task: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state queued -> state running
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::905::TaskManager.Task::(_runJobs) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::task::331::TaskManager.Task::(run) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,657::misc::764::SamplingMethod::(__call__) Got in to sampling method
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::misc::772::SamplingMethod::(__call__) Returning last result
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,667::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,667::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,677::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,699::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1fde14b7-9079-4f20-aa21-442823d1f323`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`1fde14b7-9079-4f20-aa21-442823d1f323`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,700::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`96ca8e8b-6df3-42ff-8e8c-bcf6d7758878`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,700::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,701::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,701::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`96ca8e8b-6df3-42ff-8e8c-bcf6d7758878`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,702::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,722::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,722::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,722::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,723::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,732::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-60::DEBUG::2014-04-27 12:43:09,732::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-61::DEBUG::2014-04-27 12:43:09,733::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-62::ERROR::2014-04-27 12:43:09,733::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,733::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-60::DEBUG::2014-04-27 12:43:09,734::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`3ab79dd8-4826-4364-9f0e-8c6ad5de90c1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-62::ERROR::2014-04-27 12:43:09,735::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-61::DEBUG::2014-04-27 12:43:09,735::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5263a1de-de00-433a-ac28-b871630d4576`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,735::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,735::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-62::DEBUG::2014-04-27 12:43:09,736::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-61::DEBUG::2014-04-27 12:43:09,736::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,736::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-62::DEBUG::2014-04-27 12:43:09,737::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,738::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-61::DEBUG::2014-04-27 12:43:09,742::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-61::DEBUG::2014-04-27 12:43:09,744::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5263a1de-de00-433a-ac28-b871630d4576`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,744::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-60::DEBUG::2014-04-27 12:43:09,745::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,747::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`3ab79dd8-4826-4364-9f0e-8c6ad5de90c1`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,747::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-61::DEBUG::2014-04-27 12:43:09,747::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`8a50d7b4-ad5a-4a85-967e-3f95dfcb379b`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-59::DEBUG::2014-04-27 12:43:09,749::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e
Thread-61::DEBUG::2014-04-27 12:43:09,749::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-60::DEBUG::2014-04-27 12:43:09,750::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ddcb64fe-fe94-4c2a-abb4-c708ec085d4d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-61::DEBUG::2014-04-27 12:43:09,750::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,754::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-61::DEBUG::2014-04-27 12:43:09,755::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`8a50d7b4-ad5a-4a85-967e-3f95dfcb379b`::Granted request
Thread-60::DEBUG::2014-04-27 12:43:09,757::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-60::DEBUG::2014-04-27 12:43:09,758::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ddcb64fe-fe94-4c2a-abb4-c708ec085d4d`::Granted request
Thread-62::DEBUG::2014-04-27 12:43:09,758::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,762::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-62::WARNING::2014-04-27 12:43:09,762::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-62::DEBUG::2014-04-27 12:43:09,764::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-61::DEBUG::2014-04-27 12:43:09,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-61::DEBUG::2014-04-27 12:43:09,773::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-61::DEBUG::2014-04-27 12:43:09,773::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-61::DEBUG::2014-04-27 12:43:09,774::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-61::DEBUG::2014-04-27 12:43:09,775::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-61::DEBUG::2014-04-27 12:43:09,776::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-62::ERROR::2014-04-27 12:43:09,782::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-60::DEBUG::2014-04-27 12:43:09,798::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-60::DEBUG::2014-04-27 12:43:09,798::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-60::DEBUG::2014-04-27 12:43:09,798::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-60::DEBUG::2014-04-27 12:43:09,799::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-59::DEBUG::2014-04-27 12:43:09,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
Thread-59::DEBUG::2014-04-27 12:43:09,862::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO1_ZFS', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=f027ec99-913f-4f00-ac95-ad484c9c6a4b,c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=ovirt-noded:/iso', 'ROLE=Regular', 'SDUUID=7b083758-45f9-4896-913d-11fe02043e6e', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=25bcbb3617da81289b741097860834da2ed0040e']
Thread-59::DEBUG::2014-04-27 12:43:09,864::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
Thread-59::DEBUG::2014-04-27 12:43:09,865::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '7b083758-45f9-4896-913d-11fe02043e6e_imageNS'
Thread-59::DEBUG::2014-04-27 12:43:09,865::resourceManager::421::ResourceManager::(registerNamespace) Registering namespace '7b083758-45f9-4896-913d-11fe02043e6e_volumeNS'
Thread-59::DEBUG::2014-04-27 12:43:09,865::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,866::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-59::DEBUG::2014-04-27 12:43:09,867::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`5ba44554-9c56-4fc8-a462-76d23f3c72bb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,867::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-59::DEBUG::2014-04-27 12:43:09,867::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,868::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-59::DEBUG::2014-04-27 12:43:09,869::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-59::DEBUG::2014-04-27 12:43:09,869::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`5ba44554-9c56-4fc8-a462-76d23f3c72bb`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,871::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-59::DEBUG::2014-04-27 12:43:09,871::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`416fc1f9-3f25-4d6a-9d51-2f72f48f86e0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-59::DEBUG::2014-04-27 12:43:09,875::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-59::DEBUG::2014-04-27 12:43:09,876::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-59::DEBUG::2014-04-27 12:43:09,876::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`416fc1f9-3f25-4d6a-9d51-2f72f48f86e0`::Granted request
Thread-59::DEBUG::2014-04-27 12:43:09,876::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-59::DEBUG::2014-04-27 12:43:09,877::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,885::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::WARNING::2014-04-27 12:43:09,886::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,886::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,895::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,895::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
self.checkBackupDomain(__securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,896::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,896::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,896::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,896::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,909::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::WARNING::2014-04-27 12:43:09,910::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,910::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,920::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,921::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,921::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f38ae238-755c-4e45-8cfc-906f765396a0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,921::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f38ae238-755c-4e45-8cfc-906f765396a0`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`5e8feec4-83e0-4266-b943-c260f00f1136`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,922::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`5e8feec4-83e0-4266-b943-c260f00f1136`::Granted request
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,923::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,924::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,925::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::INFO::2014-04-27 12:43:09,925::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,925::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::ERROR::2014-04-27 12:43:09,926::task::866::TaskManager.Task::(_setError) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 334, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::885::TaskManager.Task::(_run) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task._run: feedf3e9-23d6-40b8-9545-7c3a4dba8be4 () {} failed - stopping task
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::1211::TaskManager.Task::(stop) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::stopping in state running (force False)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 1 aborting True
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::916::TaskManager.Task::(_runJobs) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::990::TaskManager.Task::(_decref) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::ref 0 aborting True
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::task::925::TaskManager.Task::(_doAbort) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Task._doAbort: force False
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,937::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state running -> state aborting
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::550::TaskManager.Task::(__state_aborting) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_aborting: recover policy auto
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state aborting -> state racquiring
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state racquiring -> state recovering
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::798::TaskManager.Task::(_recover) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::_recover
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::805::TaskManager.Task::(_recover) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::running recovery None
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::786::TaskManager.Task::(_recoverDone) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::Recover Done: state recovering
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::task::595::TaskManager.Task::(_updateState) Task=`feedf3e9-23d6-40b8-9545-7c3a4dba8be4`::moving from state recovering -> state recovered
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,938::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
feedf3e9-23d6-40b8-9545-7c3a4dba8be4::DEBUG::2014-04-27 12:43:09,939::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:10,701::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,701::task::595::TaskManager.Task::(_updateState) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,701::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='feedf3e9-23d6-40b8-9545-7c3a4dba8be4', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:10,701::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:10,701::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}
Thread-13::INFO::2014-04-27 12:43:10,701::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::1185::TaskManager.Task::(prepare) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'feedf3e9-23d6-40b8-9545-7c3a4dba8be4'}}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::595::TaskManager.Task::(_updateState) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,702::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,702::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,702::task::990::TaskManager.Task::(_decref) Task=`0b53520c-5fe2-49ee-b444-bdb73cd931eb`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:10,708::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,708::task::595::TaskManager.Task::(_updateState) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,708::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:10,709::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::1185::TaskManager.Task::(prepare) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::595::TaskManager.Task::(_updateState) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,709::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,709::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,709::task::990::TaskManager.Task::(_decref) Task=`26c9ec22-c511-4351-acee-6413a58f04a9`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:10,755::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:10,755::task::595::TaskManager.Task::(_updateState) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:10,755::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='feedf3e9-23d6-40b8-9545-7c3a4dba8be4', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:10,755::taskManager::161::TaskManager::(clearTask) Entry. taskID: feedf3e9-23d6-40b8-9545-7c3a4dba8be4
Thread-13::DEBUG::2014-04-27 12:43:10,755::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:10,756::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::1185::TaskManager.Task::(prepare) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::595::TaskManager.Task::(_updateState) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:10,756::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:10,756::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:10,756::task::990::TaskManager.Task::(_decref) Task=`fcb6f6f1-4a70-4d93-a027-76c26d3041f0`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:15,830::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:15,831::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:15,832::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:15,833::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:15,849::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:15,851::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:15,852::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:15,869::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:15,870::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:15,877::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:15,882::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000689305 s, 541 kB/s\n'; <rc> = 0
Thread-17::DEBUG::2014-04-27 12:43:15,884::domainMonitor::247::Storage.DomainMonitorThread::(_monitorDomain) Domain 7b083758-45f9-4896-913d-11fe02043e6e changed its status to Valid
Thread-100::DEBUG::2014-04-27 12:43:15,884::misc::884::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event
Thread-100::DEBUG::2014-04-27 12:43:15,884::misc::894::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms`
Thread-100::DEBUG::2014-04-27 12:43:15,885::misc::904::Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted
Thread-101::INFO::2014-04-27 12:43:15,885::clientIF::126::vds::(contEIOVms) vmContainerLock acquired
Thread-22::DEBUG::2014-04-27 12:43:15,944::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:15,948::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.00013004 s, 5.1 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:16,107::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:16,111::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000113178 s, 3.3 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:16,128::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:16,131::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000142305 s, 2.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:20,849::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,850::task::595::TaskManager.Task::(_updateState) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,851::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:20,851::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::1185::TaskManager.Task::(prepare) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::595::TaskManager.Task::(_updateState) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:20,851::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,851::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,851::task::990::TaskManager.Task::(_decref) Task=`7fac00bf-a24a-48bc-999d-08b8d7b9f489`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:20,856::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,856::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,856::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:20,856::task::866::TaskManager.Task::(_setError) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::885::TaskManager.Task::(_run) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Task._run: 47a9679a-408e-489d-83fa-2387e4c1d64d () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::1211::TaskManager.Task::(stop) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:20,922::task::990::TaskManager.Task::(_decref) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:20,922::task::1168::TaskManager.Task::(prepare) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::1173::TaskManager.Task::(prepare) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::990::TaskManager.Task::(_decref) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::925::TaskManager.Task::(_doAbort) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::550::TaskManager.Task::(__state_aborting) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:20,923::task::595::TaskManager.Task::(_updateState) Task=`47a9679a-408e-489d-83fa-2387e4c1d64d`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,923::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:20,923::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:20,972::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:20,972::task::595::TaskManager.Task::(_updateState) Task=`a7250895-430c-4166-9062-e4ee751bc621`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:20,972::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:20,973::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::1185::TaskManager.Task::(prepare) Task=`a7250895-430c-4166-9062-e4ee751bc621`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::595::TaskManager.Task::(_updateState) Task=`a7250895-430c-4166-9062-e4ee751bc621`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:20,973::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:20,973::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:20,973::task::990::TaskManager.Task::(_decref) Task=`a7250895-430c-4166-9062-e4ee751bc621`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:21,046::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:21,046::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:21,046::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`574424cb-3cfa-45c3-8286-065fcb6b22e2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:21,047::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`574424cb-3cfa-45c3-8286-065fcb6b22e2`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::827::TaskManager.Task::(resourceAcquired) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:21,047::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:21,047::task::1180::TaskManager.Task::(prepare) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:21,048::taskManager::50::TaskManager::(_queueTask) queuing task: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:21,048::taskManager::56::TaskManager::(_queueTask) task queued: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::1182::TaskManager.Task::(prepare) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::returning
Thread-13::DEBUG::2014-04-27 12:43:21,048::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 0 aborting False
Thread-1::DEBUG::2014-04-27 12:43:21,048::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 6c81094c-b1cf-4565-a4b3-c11042363b34 running: <bound method Task.commit of <storage.task.Task instance at 0x1c29a70>> with: None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::1194::TaskManager.Task::(commit) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::committing task: 6c81094c-b1cf-4565-a4b3-c11042363b34
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state queued -> state running
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::905::TaskManager.Task::(_runJobs) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::task::331::TaskManager.Task::(run) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,049::misc::764::SamplingMethod::(__call__) Got in to sampling method
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::misc::772::SamplingMethod::(__call__) Returning last result
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,059::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,059::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,060::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,069::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,070::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`198baa3b-e695-4027-ad2e-2922433555f1`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`198baa3b-e695-4027-ad2e-2922433555f1`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,071::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,071::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`16f8dba2-c2e9-47b8-a592-46eab358c97d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,072::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`16f8dba2-c2e9-47b8-a592-46eab358c97d`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,074::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,084::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,085::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,085::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,085::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,094::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,095::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,095::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-105::DEBUG::2014-04-27 12:43:21,095::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-105::DEBUG::2014-04-27 12:43:21,095::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a5ca1a94-a356-455e-b82c-808408e53ab8`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,096::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-107::DEBUG::2014-04-27 12:43:21,096::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-105::DEBUG::2014-04-27 12:43:21,096::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a5ca1a94-a356-455e-b82c-808408e53ab8`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,097::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-108::ERROR::2014-04-27 12:43:21,097::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-107::DEBUG::2014-04-27 12:43:21,097::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`2260c82e-6d6e-4b4b-8648-9e6a29b1a9d0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-106::DEBUG::2014-04-27 12:43:21,097::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ad4dd919-e918-4226-9e10-8bcdbaadbafc`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,098::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`ddd5e1a7-4937-458f-b7cb-68c7ddeab8f4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,098::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-108::ERROR::2014-04-27 12:43:21,098::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-107::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-106::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,098::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,098::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-108::DEBUG::2014-04-27 12:43:21,098::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,099::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-108::DEBUG::2014-04-27 12:43:21,099::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-107::DEBUG::2014-04-27 12:43:21,099::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,103::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-107::DEBUG::2014-04-27 12:43:21,103::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`2260c82e-6d6e-4b4b-8648-9e6a29b1a9d0`::Granted request
Thread-106::DEBUG::2014-04-27 12:43:21,103::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,104::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ad4dd919-e918-4226-9e10-8bcdbaadbafc`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,104::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-105::DEBUG::2014-04-27 12:43:21,104::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-107::DEBUG::2014-04-27 12:43:21,104::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`68c7edb8-45f3-4423-ba0b-61b654e480b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-106::DEBUG::2014-04-27 12:43:21,105::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ebf341b1-9f45-494e-9067-5ef60b7df58d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-105::DEBUG::2014-04-27 12:43:21,105::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`ddd5e1a7-4937-458f-b7cb-68c7ddeab8f4`::Granted request
Thread-107::DEBUG::2014-04-27 12:43:21,105::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-105::DEBUG::2014-04-27 12:43:21,106::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-106::DEBUG::2014-04-27 12:43:21,106::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-107::DEBUG::2014-04-27 12:43:21,106::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-105::DEBUG::2014-04-27 12:43:21,106::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-107::DEBUG::2014-04-27 12:43:21,107::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`68c7edb8-45f3-4423-ba0b-61b654e480b4`::Granted request
Thread-106::DEBUG::2014-04-27 12:43:21,107::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-106::DEBUG::2014-04-27 12:43:21,107::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ebf341b1-9f45-494e-9067-5ef60b7df58d`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-107::DEBUG::2014-04-27 12:43:21,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-105::DEBUG::2014-04-27 12:43:21,118::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-107::DEBUG::2014-04-27 12:43:21,119::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-105::DEBUG::2014-04-27 12:43:21,120::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-105::DEBUG::2014-04-27 12:43:21,122::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-106::DEBUG::2014-04-27 12:43:21,123::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-108::DEBUG::2014-04-27 12:43:21,123::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-107::DEBUG::2014-04-27 12:43:21,124::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-107::DEBUG::2014-04-27 12:43:21,127::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-105::DEBUG::2014-04-27 12:43:21,129::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-106::DEBUG::2014-04-27 12:43:21,130::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-107::DEBUG::2014-04-27 12:43:21,131::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-108::WARNING::2014-04-27 12:43:21,134::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-105::DEBUG::2014-04-27 12:43:21,134::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,137::task::595::TaskManager.Task::(_updateState) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::moving from state init -> state preparing
Thread-105::DEBUG::2014-04-27 12:43:21,137::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-107::DEBUG::2014-04-27 12:43:21,138::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-108::DEBUG::2014-04-27 12:43:21,138::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-13::INFO::2014-04-27 12:43:21,139::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-105::DEBUG::2014-04-27 12:43:21,139::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-106::DEBUG::2014-04-27 12:43:21,141::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-106::DEBUG::2014-04-27 12:43:21,144::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-13::INFO::2014-04-27 12:43:21,144::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000689305', 'lastCheck': '5.3', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000113178', 'lastCheck': '5.0', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00013004', 'lastCheck': '5.2', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000142305', 'lastCheck': '5.0', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.3', 'valid': False}}
Thread-106::DEBUG::2014-04-27 12:43:21,146::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:21,147::task::1185::TaskManager.Task::(prepare) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000689305', 'lastCheck': '5.3', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000113178', 'lastCheck': '5.0', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00013004', 'lastCheck': '5.2', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000142305', 'lastCheck': '5.0', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.3', 'valid': False}}
Thread-107::DEBUG::2014-04-27 12:43:21,150::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,151::task::595::TaskManager.Task::(_updateState) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::moving from state preparing -> state finished
Thread-107::DEBUG::2014-04-27 12:43:21,154::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-13::DEBUG::2014-04-27 12:43:21,154::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-106::DEBUG::2014-04-27 12:43:21,154::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-107::DEBUG::2014-04-27 12:43:21,156::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-13::DEBUG::2014-04-27 12:43:21,157::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-106::DEBUG::2014-04-27 12:43:21,159::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-13::DEBUG::2014-04-27 12:43:21,160::task::990::TaskManager.Task::(_decref) Task=`83dc6ada-7640-4c59-bb1e-cb5daba3effa`::ref 0 aborting False
Thread-106::DEBUG::2014-04-27 12:43:21,161::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-106::DEBUG::2014-04-27 12:43:21,164::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-108::ERROR::2014-04-27 12:43:21,170::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,171::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,172::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,173::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,177::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,190::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
6c81094c-b1cf-4565-a4b3-c11042363b34::WARNING::2014-04-27 12:43:21,191::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,191::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,204::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,204::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
self.checkBackupDomain(__securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,205::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,205::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,205::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,206::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,219::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
6c81094c-b1cf-4565-a4b3-c11042363b34::WARNING::2014-04-27 12:43:21,219::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,220::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,230::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,230::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`224f5b16-8485-4f1d-bcd3-0fc6ae0b618f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`224f5b16-8485-4f1d-bcd3-0fc6ae0b618f`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`c6289201-7b36-4cc7-a69d-5fc62a0240f3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,231::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`c6289201-7b36-4cc7-a69d-5fc62a0240f3`::Granted request
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,232::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,233::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,234::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,234::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
6c81094c-b1cf-4565-a4b3-c11042363b34::INFO::2014-04-27 12:43:21,235::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
6c81094c-b1cf-4565-a4b3-c11042363b34::ERROR::2014-04-27 12:43:21,235::task::866::TaskManager.Task::(_setError) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 334, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::885::TaskManager.Task::(_run) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task._run: 6c81094c-b1cf-4565-a4b3-c11042363b34 () {} failed - stopping task
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::1211::TaskManager.Task::(stop) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::stopping in state running (force False)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,235::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 1 aborting True
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::916::TaskManager.Task::(_runJobs) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::990::TaskManager.Task::(_decref) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::ref 0 aborting True
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::925::TaskManager.Task::(_doAbort) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Task._doAbort: force False
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state running -> state aborting
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::550::TaskManager.Task::(__state_aborting) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_aborting: recover policy auto
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state aborting -> state racquiring
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state racquiring -> state recovering
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,236::task::798::TaskManager.Task::(_recover) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::_recover
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::805::TaskManager.Task::(_recover) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::running recovery None
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::786::TaskManager.Task::(_recoverDone) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::Recover Done: state recovering
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::task::595::TaskManager.Task::(_updateState) Task=`6c81094c-b1cf-4565-a4b3-c11042363b34`::moving from state recovering -> state recovered
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,237::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,238::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:21,238::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:22,094::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,094::task::595::TaskManager.Task::(_updateState) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,094::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='6c81094c-b1cf-4565-a4b3-c11042363b34', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:22,094::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:22,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}
Thread-13::INFO::2014-04-27 12:43:22,095::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::1185::TaskManager.Task::(prepare) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '6c81094c-b1cf-4565-a4b3-c11042363b34'}}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::595::TaskManager.Task::(_updateState) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,095::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,095::task::990::TaskManager.Task::(_decref) Task=`eedcd99b-08d1-4635-a7be-395030d75f81`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:22,101::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::595::TaskManager.Task::(_updateState) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,101::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:22,101::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::1185::TaskManager.Task::(prepare) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:22,101::task::595::TaskManager.Task::(_updateState) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,101::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,102::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,102::task::990::TaskManager.Task::(_decref) Task=`195b6fb7-b45a-4dcb-bff3-1e93ba499748`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:22,146::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::595::TaskManager.Task::(_updateState) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:22,147::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='6c81094c-b1cf-4565-a4b3-c11042363b34', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:22,147::taskManager::161::TaskManager::(clearTask) Entry. taskID: 6c81094c-b1cf-4565-a4b3-c11042363b34
Thread-13::DEBUG::2014-04-27 12:43:22,147::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:22,147::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::1185::TaskManager.Task::(prepare) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:22,147::task::595::TaskManager.Task::(_updateState) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:22,147::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:22,147::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:22,148::task::990::TaskManager.Task::(_decref) Task=`aba2cadc-9faa-490d-9dbb-bf15bdc51f61`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:25,872::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:25,872::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:25,872::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:25,872::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:25,888::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:25,890::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:25,892::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:25,909::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:25,910::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:25,931::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:25,935::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000781446 s, 477 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:25,962::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:25,965::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000128537 s, 5.2 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:26,126::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:26,130::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000108243 s, 3.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:26,145::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:26,149::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000100579 s, 3.6 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:32,220::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,220::task::595::TaskManager.Task::(_updateState) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,221::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:32,221::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::1185::TaskManager.Task::(prepare) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::595::TaskManager.Task::(_updateState) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:32,221::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,221::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,221::task::990::TaskManager.Task::(_decref) Task=`37563c71-eb28-43c0-ac80-9913a8b284d6`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:32,225::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,226::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:32,226::task::866::TaskManager.Task::(_setError) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::885::TaskManager.Task::(_run) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Task._run: 96ad43f1-38cb-4249-a2f9-8c02a5abe445 () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::1211::TaskManager.Task::(stop) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:32,226::task::990::TaskManager.Task::(_decref) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:32,226::task::1168::TaskManager.Task::(prepare) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::1173::TaskManager.Task::(prepare) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::990::TaskManager.Task::(_decref) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::925::TaskManager.Task::(_doAbort) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::550::TaskManager.Task::(__state_aborting) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:32,227::task::595::TaskManager.Task::(_updateState) Task=`96ad43f1-38cb-4249-a2f9-8c02a5abe445`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,227::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:32,227::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:32,276::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::595::TaskManager.Task::(_updateState) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,277::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:32,277::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::1185::TaskManager.Task::(prepare) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:32,277::task::595::TaskManager.Task::(_updateState) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:32,278::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:32,278::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:32,278::task::990::TaskManager.Task::(_decref) Task=`11aba587-a2ec-4e77-b4cd-181aff5fcfa9`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:32,349::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:32,349::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:32,350::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`fe669ea3-2285-48e5-9a1f-7f40c1ad7abf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:32,350::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`fe669ea3-2285-48e5-9a1f-7f40c1ad7abf`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::827::TaskManager.Task::(resourceAcquired) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:32,351::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::1180::TaskManager.Task::(prepare) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:32,351::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:32,351::taskManager::50::TaskManager::(_queueTask) queuing task: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:32,351::taskManager::56::TaskManager::(_queueTask) task queued: 5a22ef66-e6a9-4d46-88be-70a384dc5442
6c81094c-b1cf-4565-a4b3-c11042363b34::DEBUG::2014-04-27 12:43:32,352::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:32,352::task::1182::TaskManager.Task::(prepare) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::returning
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,352::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 5a22ef66-e6a9-4d46-88be-70a384dc5442 running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00188ef0>> with: None
Thread-13::DEBUG::2014-04-27 12:43:32,353::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 0 aborting False
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,353::task::1194::TaskManager.Task::(commit) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::committing task: 5a22ef66-e6a9-4d46-88be-70a384dc5442
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,354::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state queued -> state running
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::task::905::TaskManager.Task::(_runJobs) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::task::331::TaskManager.Task::(run) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,355::misc::764::SamplingMethod::(__call__) Got in to sampling method
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::misc::772::SamplingMethod::(__call__) Returning last result
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,365::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,365::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,366::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,375::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,376::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`6ad8f73a-54d4-4840-9e50-f3a5b0892163`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,377::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`6ad8f73a-54d4-4840-9e50-f3a5b0892163`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,377::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`743af4fb-133f-4c84-9102-d44bc6260e8f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,378::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`743af4fb-133f-4c84-9102-d44bc6260e8f`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,382::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,383::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,392::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,393::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,393::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,393::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,403::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-144::DEBUG::2014-04-27 12:43:32,404::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-145::DEBUG::2014-04-27 12:43:32,404::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-146::DEBUG::2014-04-27 12:43:32,405::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-147::ERROR::2014-04-27 12:43:32,407::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,407::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-147::ERROR::2014-04-27 12:43:32,408::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-144::DEBUG::2014-04-27 12:43:32,409::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`7ab40a6b-b38b-4487-b919-d09f70b94309`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-145::DEBUG::2014-04-27 12:43:32,410::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`0c85280a-218d-4933-8ffc-b6a91684614a`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,411::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,411::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`58397b55-c600-4331-a969-c0fd4a439554`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-144::DEBUG::2014-04-27 12:43:32,411::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-147::DEBUG::2014-04-27 12:43:32,412::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-145::DEBUG::2014-04-27 12:43:32,412::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,412::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-146::DEBUG::2014-04-27 12:43:32,413::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,414::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-144::DEBUG::2014-04-27 12:43:32,415::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,416::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`7ab40a6b-b38b-4487-b919-d09f70b94309`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,418::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-145::DEBUG::2014-04-27 12:43:32,418::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-147::DEBUG::2014-04-27 12:43:32,419::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-145::DEBUG::2014-04-27 12:43:32,420::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`0c85280a-218d-4933-8ffc-b6a91684614a`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,425::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-144::DEBUG::2014-04-27 12:43:32,425::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`67ac383c-c84b-4309-bc44-f341f8c490e3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-146::DEBUG::2014-04-27 12:43:32,427::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-146::DEBUG::2014-04-27 12:43:32,428::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`58397b55-c600-4331-a969-c0fd4a439554`::Granted request
Thread-144::DEBUG::2014-04-27 12:43:32,428::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-145::DEBUG::2014-04-27 12:43:32,431::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ef073285-3ea7-4f61-81c3-5593454f1797`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-145::DEBUG::2014-04-27 12:43:32,432::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-146::DEBUG::2014-04-27 12:43:32,434::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5ca6f208-4b81-4db4-b3f3-66ed05bdc66d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-146::DEBUG::2014-04-27 12:43:32,434::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-147::DEBUG::2014-04-27 12:43:32,435::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-147::WARNING::2014-04-27 12:43:32,436::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-147::DEBUG::2014-04-27 12:43:32,436::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-144::DEBUG::2014-04-27 12:43:32,431::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,443::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`67ac383c-c84b-4309-bc44-f341f8c490e3`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-145::DEBUG::2014-04-27 12:43:32,448::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-145::DEBUG::2014-04-27 12:43:32,450::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ef073285-3ea7-4f61-81c3-5593454f1797`::Granted request
Thread-144::DEBUG::2014-04-27 12:43:32,451::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-147::ERROR::2014-04-27 12:43:32,451::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-146::DEBUG::2014-04-27 12:43:32,453::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-144::DEBUG::2014-04-27 12:43:32,454::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,455::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-146::DEBUG::2014-04-27 12:43:32,456::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`5ca6f208-4b81-4db4-b3f3-66ed05bdc66d`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,456::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-144::DEBUG::2014-04-27 12:43:32,458::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,458::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-144::DEBUG::2014-04-27 12:43:32,459::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-144::DEBUG::2014-04-27 12:43:32,459::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-144::DEBUG::2014-04-27 12:43:32,461::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-144::DEBUG::2014-04-27 12:43:32,461::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,462::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-144::DEBUG::2014-04-27 12:43:32,462::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-144::DEBUG::2014-04-27 12:43:32,466::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-145::DEBUG::2014-04-27 12:43:32,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-145::DEBUG::2014-04-27 12:43:32,476::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-145::DEBUG::2014-04-27 12:43:32,477::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,477::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-145::DEBUG::2014-04-27 12:43:32,478::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-145::DEBUG::2014-04-27 12:43:32,479::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::WARNING::2014-04-27 12:43:32,480::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-146::DEBUG::2014-04-27 12:43:32,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-145::DEBUG::2014-04-27 12:43:32,481::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,482::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-145::DEBUG::2014-04-27 12:43:32,484::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,484::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-145::DEBUG::2014-04-27 12:43:32,485::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-145::DEBUG::2014-04-27 12:43:32,486::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-146::DEBUG::2014-04-27 12:43:32,488::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,490::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-146::DEBUG::2014-04-27 12:43:32,491::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-146::DEBUG::2014-04-27 12:43:32,494::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-146::DEBUG::2014-04-27 12:43:32,497::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-146::DEBUG::2014-04-27 12:43:32,499::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,500::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-146::DEBUG::2014-04-27 12:43:32,500::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,501::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
self.checkBackupDomain(__securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,502::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,502::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,502::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,503::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,516::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
5a22ef66-e6a9-4d46-88be-70a384dc5442::WARNING::2014-04-27 12:43:32,517::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,517::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,527::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`467354ae-fac1-4cb3-8462-107de32251d3`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,528::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`467354ae-fac1-4cb3-8462-107de32251d3`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`0dd6852a-2ca2-4348-8d47-9166ee323b5d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`0dd6852a-2ca2-4348-8d47-9166ee323b5d`::Granted request
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,529::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,530::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,531::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,531::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,532::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
5a22ef66-e6a9-4d46-88be-70a384dc5442::INFO::2014-04-27 12:43:32,532::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,532::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
5a22ef66-e6a9-4d46-88be-70a384dc5442::ERROR::2014-04-27 12:43:32,532::task::866::TaskManager.Task::(_setError) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 334, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::885::TaskManager.Task::(_run) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task._run: 5a22ef66-e6a9-4d46-88be-70a384dc5442 () {} failed - stopping task
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::1211::TaskManager.Task::(stop) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::stopping in state running (force False)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 1 aborting True
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::916::TaskManager.Task::(_runJobs) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::990::TaskManager.Task::(_decref) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::ref 0 aborting True
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::925::TaskManager.Task::(_doAbort) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Task._doAbort: force False
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,533::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state running -> state aborting
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::550::TaskManager.Task::(__state_aborting) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_aborting: recover policy auto
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state aborting -> state racquiring
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state racquiring -> state recovering
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::798::TaskManager.Task::(_recover) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::_recover
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::805::TaskManager.Task::(_recover) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::running recovery None
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::786::TaskManager.Task::(_recoverDone) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::Recover Done: state recovering
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::task::595::TaskManager.Task::(_updateState) Task=`5a22ef66-e6a9-4d46-88be-70a384dc5442`::moving from state recovering -> state recovered
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,534::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:32,535::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:33,397::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::595::TaskManager.Task::(_updateState) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,398::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='5a22ef66-e6a9-4d46-88be-70a384dc5442', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:33,398::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:33,398::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}
Thread-13::INFO::2014-04-27 12:43:33,398::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::1185::TaskManager.Task::(prepare) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '5a22ef66-e6a9-4d46-88be-70a384dc5442'}}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::595::TaskManager.Task::(_updateState) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,398::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,398::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,398::task::990::TaskManager.Task::(_decref) Task=`fb7e0702-ec0e-4958-88bc-42673d8e902e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:33,404::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::595::TaskManager.Task::(_updateState) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,404::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:33,404::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::1185::TaskManager.Task::(prepare) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:33,404::task::595::TaskManager.Task::(_updateState) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,405::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,405::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,405::task::990::TaskManager.Task::(_decref) Task=`1842f59f-8c04-4334-b294-9f3a765b7403`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:33,449::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:33,449::task::595::TaskManager.Task::(_updateState) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:33,449::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='5a22ef66-e6a9-4d46-88be-70a384dc5442', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:33,450::taskManager::161::TaskManager::(clearTask) Entry. taskID: 5a22ef66-e6a9-4d46-88be-70a384dc5442
Thread-13::DEBUG::2014-04-27 12:43:33,450::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:33,450::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::1185::TaskManager.Task::(prepare) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::595::TaskManager.Task::(_updateState) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:33,450::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:33,450::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:33,450::task::990::TaskManager.Task::(_decref) Task=`0e428ee7-4823-4ce8-806e-b649069d7ece`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:35,911::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:35,912::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:35,912::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:35,912::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:35,926::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:35,926::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:35,926::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:35,937::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:35,937::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:35,960::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:35,965::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000713226 s, 523 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:35,979::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:35,983::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 7.9599e-05 s, 8.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:36,142::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:36,145::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000109191 s, 3.4 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:36,163::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:36,167::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.000137089 s, 2.7 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:36,329::task::595::TaskManager.Task::(_updateState) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:36,330::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:36,330::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000713226', 'lastCheck': '0.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000109191', 'lastCheck': '0.2', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '7.9599e-05', 'lastCheck': '0.3', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000137089', 'lastCheck': '0.2', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:36,330::task::1185::TaskManager.Task::(prepare) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000713226', 'lastCheck': '0.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000109191', 'lastCheck': '0.2', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '7.9599e-05', 'lastCheck': '0.3', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000137089', 'lastCheck': '0.2', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '0.4', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:36,330::task::595::TaskManager.Task::(_updateState) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:36,330::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:36,330::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:36,331::task::990::TaskManager.Task::(_decref) Task=`73124206-1162-4ca0-837e-c3ed6fcd66aa`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,550::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::595::TaskManager.Task::(_updateState) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,551::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:43,551::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::1185::TaskManager.Task::(prepare) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,551::task::595::TaskManager.Task::(_updateState) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:43,551::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,551::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,552::task::990::TaskManager.Task::(_decref) Task=`29529d81-d985-4f9f-bbb5-5661be8bd496`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,556::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,556::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,556::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:43,556::task::866::TaskManager.Task::(_setError) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::885::TaskManager.Task::(_run) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Task._run: 2e2e74ad-be80-486e-adac-011cadf3a713 () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::1211::TaskManager.Task::(stop) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::990::TaskManager.Task::(_decref) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:43,557::task::1168::TaskManager.Task::(prepare) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::1173::TaskManager.Task::(prepare) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::990::TaskManager.Task::(_decref) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::925::TaskManager.Task::(_doAbort) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:43,557::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,557::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:43,558::task::550::TaskManager.Task::(__state_aborting) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:43,558::task::595::TaskManager.Task::(_updateState) Task=`2e2e74ad-be80-486e-adac-011cadf3a713`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:43,558::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,558::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:43,558::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:43,607::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::595::TaskManager.Task::(_updateState) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,607::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:43,607::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::1185::TaskManager.Task::(prepare) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:43,607::task::595::TaskManager.Task::(_updateState) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:43,607::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:43,608::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:43,608::task::990::TaskManager.Task::(_decref) Task=`e18d8a62-c40d-472a-86e0-64b8b4457df1`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:43,677::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:43,677::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:43,677::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`68802a3e-fc25-4bae-a693-81462470ecdf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:43,678::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`68802a3e-fc25-4bae-a693-81462470ecdf`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:43,678::task::827::TaskManager.Task::(resourceAcquired) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:43,679::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::1180::TaskManager.Task::(prepare) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:43,679::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:43,679::taskManager::50::TaskManager::(_queueTask) queuing task: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:43,679::taskManager::56::TaskManager::(_queueTask) task queued: 65d28cfc-d761-4f97-94f1-48886843966f
5a22ef66-e6a9-4d46-88be-70a384dc5442::DEBUG::2014-04-27 12:43:43,680::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:43,680::task::1182::TaskManager.Task::(prepare) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::returning
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,680::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 65d28cfc-d761-4f97-94f1-48886843966f running: <bound method Task.commit of <storage.task.Task instance at 0x7fcd00169638>> with: None
Thread-13::DEBUG::2014-04-27 12:43:43,681::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 0 aborting False
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,681::task::1194::TaskManager.Task::(commit) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::committing task: 65d28cfc-d761-4f97-94f1-48886843966f
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,682::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state queued -> state running
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::task::905::TaskManager.Task::(_runJobs) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::task::331::TaskManager.Task::(run) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,683::misc::764::SamplingMethod::(__call__) Got in to sampling method
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,692::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::misc::772::SamplingMethod::(__call__) Returning last result
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,693::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,693::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,703::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,704::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`5df371d5-0cfa-467c-855e-66c9b8db67db`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`5df371d5-0cfa-467c-855e-66c9b8db67db`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,705::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`34b26d27-705f-4f25-80e8-b2bc2f3b0d66`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,705::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,706::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,706::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`34b26d27-705f-4f25-80e8-b2bc2f3b0d66`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,710::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,720::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,720::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,721::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,721::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,731::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-183::DEBUG::2014-04-27 12:43:43,731::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-184::DEBUG::2014-04-27 12:43:43,732::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-185::DEBUG::2014-04-27 12:43:43,733::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-186::ERROR::2014-04-27 12:43:43,734::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,735::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-186::ERROR::2014-04-27 12:43:43,736::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,736::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`fe18f291-25c0-405b-8f1d-6bde9dc0d3c0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-184::DEBUG::2014-04-27 12:43:43,737::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`6fd2488a-0158-4ade-9833-bece0980ac9d`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,738::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,738::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`6b0db895-87c9-48dd-b121-5cd800612ccb`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-183::DEBUG::2014-04-27 12:43:43,738::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-186::DEBUG::2014-04-27 12:43:43,739::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-184::DEBUG::2014-04-27 12:43:43,739::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,740::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,740::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,742::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-183::DEBUG::2014-04-27 12:43:43,742::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,743::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`fe18f291-25c0-405b-8f1d-6bde9dc0d3c0`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,745::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-184::DEBUG::2014-04-27 12:43:43,745::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-184::DEBUG::2014-04-27 12:43:43,747::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`6fd2488a-0158-4ade-9833-bece0980ac9d`::Granted request
Thread-186::DEBUG::2014-04-27 12:43:43,747::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,752::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-183::DEBUG::2014-04-27 12:43:43,753::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`3dbc4063-c6d1-4115-a6dc-3d4d08a97da8`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-185::DEBUG::2014-04-27 12:43:43,753::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,755::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-185::DEBUG::2014-04-27 12:43:43,756::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`6b0db895-87c9-48dd-b121-5cd800612ccb`::Granted request
Thread-184::DEBUG::2014-04-27 12:43:43,758::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`83868e7a-a661-4661-ac4e-92b09e4bb711`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-183::DEBUG::2014-04-27 12:43:43,759::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-185::DEBUG::2014-04-27 12:43:43,760::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`3051cfcb-c306-4e5d-a9ad-7e836f551a94`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-186::DEBUG::2014-04-27 12:43:43,761::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-186::WARNING::2014-04-27 12:43:43,763::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-186::DEBUG::2014-04-27 12:43:43,763::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-184::DEBUG::2014-04-27 12:43:43,768::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-183::DEBUG::2014-04-27 12:43:43,768::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`3dbc4063-c6d1-4115-a6dc-3d4d08a97da8`::Granted request
Thread-185::DEBUG::2014-04-27 12:43:43,769::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-184::DEBUG::2014-04-27 12:43:43,771::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-184::DEBUG::2014-04-27 12:43:43,774::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`83868e7a-a661-4661-ac4e-92b09e4bb711`::Granted request
Thread-183::DEBUG::2014-04-27 12:43:43,774::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-185::DEBUG::2014-04-27 12:43:43,778::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-183::DEBUG::2014-04-27 12:43:43,778::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-185::DEBUG::2014-04-27 12:43:43,781::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`3051cfcb-c306-4e5d-a9ad-7e836f551a94`::Granted request
Thread-183::DEBUG::2014-04-27 12:43:43,783::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-183::DEBUG::2014-04-27 12:43:43,785::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-183::DEBUG::2014-04-27 12:43:43,787::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-185::DEBUG::2014-04-27 12:43:43,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-183::DEBUG::2014-04-27 12:43:43,803::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
Thread-184::DEBUG::2014-04-27 12:43:43,804::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-186::ERROR::2014-04-27 12:43:43,807::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-183::DEBUG::2014-04-27 12:43:43,807::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,807::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-184::DEBUG::2014-04-27 12:43:43,808::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,809::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,809::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,810::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-183::DEBUG::2014-04-27 12:43:43,811::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,812::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-185::DEBUG::2014-04-27 12:43:43,812::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,813::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,813::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-184::DEBUG::2014-04-27 12:43:43,814::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-184::DEBUG::2014-04-27 12:43:43,815::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-185::DEBUG::2014-04-27 12:43:43,815::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-184::DEBUG::2014-04-27 12:43:43,816::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,817::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-185::DEBUG::2014-04-27 12:43:43,817::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-185::DEBUG::2014-04-27 12:43:43,821::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-184::DEBUG::2014-04-27 12:43:43,822::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-185::DEBUG::2014-04-27 12:43:43,822::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-184::DEBUG::2014-04-27 12:43:43,823::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-184::DEBUG::2014-04-27 12:43:43,823::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-184::DEBUG::2014-04-27 12:43:43,824::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,831::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
65d28cfc-d761-4f97-94f1-48886843966f::WARNING::2014-04-27 12:43:43,831::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,831::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,841::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,841::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
self.checkBackupDomain(__securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,842::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,842::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,842::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,843::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,856::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
65d28cfc-d761-4f97-94f1-48886843966f::WARNING::2014-04-27 12:43:43,856::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,857::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,867::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,867::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,868::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`e0ccd375-ac40-42cb-bd7f-a00809cf4eb0`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,868::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`e0ccd375-ac40-42cb-bd7f-a00809cf4eb0`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`f265244e-fe3e-4433-86a4-15a2873ab7b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`f265244e-fe3e-4433-86a4-15a2873ab7b4`::Granted request
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,869::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,870::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,871::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,872::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
65d28cfc-d761-4f97-94f1-48886843966f::INFO::2014-04-27 12:43:43,872::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,872::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
65d28cfc-d761-4f97-94f1-48886843966f::ERROR::2014-04-27 12:43:43,872::task::866::TaskManager.Task::(_setError) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 334, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::885::TaskManager.Task::(_run) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task._run: 65d28cfc-d761-4f97-94f1-48886843966f () {} failed - stopping task
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::1211::TaskManager.Task::(stop) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::stopping in state running (force False)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 1 aborting True
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::916::TaskManager.Task::(_runJobs) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::990::TaskManager.Task::(_decref) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::ref 0 aborting True
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::task::925::TaskManager.Task::(_doAbort) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Task._doAbort: force False
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,873::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state running -> state aborting
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::550::TaskManager.Task::(__state_aborting) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_aborting: recover policy auto
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state aborting -> state racquiring
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state racquiring -> state recovering
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::798::TaskManager.Task::(_recover) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::_recover
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::805::TaskManager.Task::(_recover) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::running recovery None
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::786::TaskManager.Task::(_recoverDone) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::Recover Done: state recovering
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::task::595::TaskManager.Task::(_updateState) Task=`65d28cfc-d761-4f97-94f1-48886843966f`::moving from state recovering -> state recovered
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,874::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:43,875::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-13::DEBUG::2014-04-27 12:43:44,725::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::595::TaskManager.Task::(_updateState) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,726::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='65d28cfc-d761-4f97-94f1-48886843966f', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:44,726::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:44,726::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}
Thread-13::INFO::2014-04-27 12:43:44,726::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}}
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::1185::TaskManager.Task::(prepare) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '65d28cfc-d761-4f97-94f1-48886843966f'}}
Thread-13::DEBUG::2014-04-27 12:43:44,726::task::595::TaskManager.Task::(_updateState) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,727::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,727::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,727::task::990::TaskManager.Task::(_decref) Task=`3f658fef-8aa5-4b7b-8c61-8f0d60642e9d`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:44,733::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::595::TaskManager.Task::(_updateState) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,733::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:44,733::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::1185::TaskManager.Task::(prepare) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:44,733::task::595::TaskManager.Task::(_updateState) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,734::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,734::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,734::task::990::TaskManager.Task::(_decref) Task=`1c0046a5-d361-4b82-b524-b7c22fffdd1e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:44,779::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::595::TaskManager.Task::(_updateState) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:44,780::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='65d28cfc-d761-4f97-94f1-48886843966f', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:44,780::taskManager::161::TaskManager::(clearTask) Entry. taskID: 65d28cfc-d761-4f97-94f1-48886843966f
Thread-13::DEBUG::2014-04-27 12:43:44,780::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:44,780::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::1185::TaskManager.Task::(prepare) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:44,780::task::595::TaskManager.Task::(_updateState) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:44,780::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:44,781::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:44,781::task::990::TaskManager.Task::(_decref) Task=`fbeecab0-bbac-46f6-94cb-649e9e1b9e23`::ref 0 aborting False
Thread-29::ERROR::2014-04-27 12:43:45,938::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:45,938::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:45,938::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:45,938::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:45,952::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:45,952::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:45,952::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:45,963::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:45,963::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-17::DEBUG::2014-04-27 12:43:46,019::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:46,029::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.000653768 s, 571 kB/s\n'; <rc> = 0
Thread-22::DEBUG::2014-04-27 12:43:46,038::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2014-04-27 12:43:46,042::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 0.000108903 s, 6.1 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:46,160::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:46,164::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 0.000115133 s, 3.2 MB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:46,181::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:46,184::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.00010829 s, 3.4 MB/s\n'; <rc> = 0
Thread-13::DEBUG::2014-04-27 12:43:51,471::task::595::TaskManager.Task::(_updateState) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:51,472::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-13::INFO::2014-04-27 12:43:51,472::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653768', 'lastCheck': '5.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000115133', 'lastCheck': '5.3', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000108903', 'lastCheck': '5.4', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00010829', 'lastCheck': '5.3', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::1185::TaskManager.Task::(prepare) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::finished: {'7b083758-45f9-4896-913d-11fe02043e6e': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653768', 'lastCheck': '5.4', 'valid': True}, 'e637bb04-a8b7-4c77-809c-d58051494c52': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000115133', 'lastCheck': '5.3', 'valid': True}, '0a897f2e-1b01-4577-9f91-cd136ef4a978': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000108903', 'lastCheck': '5.4', 'valid': True}, 'c2c4ade6-049e-4159-a294-a0c151f4983d': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.00010829', 'lastCheck': '5.3', 'valid': True}, '9a4a80a1-5377-4a94-ade3-e58183e916ae': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.5', 'valid': False}}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::595::TaskManager.Task::(_updateState) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:51,472::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:51,472::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:51,472::task::990::TaskManager.Task::(_decref) Task=`0a211ef6-c372-4271-980e-44cd2c215d62`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:54,869::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::595::TaskManager.Task::(_updateState) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,869::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:54,869::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::1185::TaskManager.Task::(prepare) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,869::task::595::TaskManager.Task::(_updateState) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:54,870::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,870::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,870::task::990::TaskManager.Task::(_decref) Task=`8550392b-80ea-43c6-81e8-5dfa5d212295`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:54,874::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,874::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,874::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-13::ERROR::2014-04-27 12:43:54,874::task::866::TaskManager.Task::(_setError) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2114, in getAllTasksStatuses
allTasksStatus = sp.getAllTasksStatuses()
File "/usr/share/vdsm/storage/securable.py", line 73, in wrapper
raise SecureError("Secured object is not in safe state")
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::885::TaskManager.Task::(_run) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Task._run: b40fc62c-69e4-4198-8688-fe0a29553a3d () {} failed - stopping task
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::1211::TaskManager.Task::(stop) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::stopping in state preparing (force False)
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::990::TaskManager.Task::(_decref) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::ref 1 aborting True
Thread-13::INFO::2014-04-27 12:43:54,875::task::1168::TaskManager.Task::(prepare) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::aborting: Task is aborted: u'Secured object is not in safe state' - code 100
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::1173::TaskManager.Task::(prepare) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Prepare: aborted: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::990::TaskManager.Task::(_decref) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::ref 0 aborting True
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::925::TaskManager.Task::(_doAbort) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::Task._doAbort: force False
Thread-13::DEBUG::2014-04-27 12:43:54,875::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state preparing -> state aborting
Thread-13::DEBUG::2014-04-27 12:43:54,875::task::550::TaskManager.Task::(__state_aborting) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::_aborting: recover policy none
Thread-13::DEBUG::2014-04-27 12:43:54,876::task::595::TaskManager.Task::(_updateState) Task=`b40fc62c-69e4-4198-8688-fe0a29553a3d`::moving from state aborting -> state failed
Thread-13::DEBUG::2014-04-27 12:43:54,876::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,876::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::ERROR::2014-04-27 12:43:54,876::dispatcher::68::Storage.Dispatcher.Protect::(run) Secured object is not in safe state
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
SecureError: Secured object is not in safe state
Thread-13::DEBUG::2014-04-27 12:43:54,925::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::595::TaskManager.Task::(_updateState) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:54,925::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:54,925::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::1185::TaskManager.Task::(prepare) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:54,925::task::595::TaskManager.Task::(_updateState) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:54,926::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:54,926::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:54,926::task::990::TaskManager.Task::(_decref) Task=`c90bcbfc-116d-4f3d-a5b0-3580dcf95b24`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:55,006::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:55,006::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:55,007::logUtils::44::dispatcher::(wrapper) Run and protect: spmStart(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', prevID=-1, prevLVER='-1', maxHostID=250, domVersion='3', options=None)
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`41387658-59c1-4051-89e0-99b6b07ecde9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '577' at 'spmStart'
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
Thread-13::DEBUG::2014-04-27 12:43:55,007::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`41387658-59c1-4051-89e0-99b6b07ecde9`::Granted request
Thread-13::DEBUG::2014-04-27 12:43:55,007::task::827::TaskManager.Task::(resourceAcquired) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_resourcesAcquired: Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002 (exclusive)
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 1 aborting False
Thread-13::INFO::2014-04-27 12:43:55,008::logUtils::47::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::1180::TaskManager.Task::(prepare) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Prepare: 1 jobs exist, move to acquiring
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state preparing -> state acquiring
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state acquiring -> state queued
Thread-13::DEBUG::2014-04-27 12:43:55,008::taskManager::50::TaskManager::(_queueTask) queuing task: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:55,008::taskManager::56::TaskManager::(_queueTask) task queued: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:55,008::task::1182::TaskManager.Task::(prepare) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::returning
65d28cfc-d761-4f97-94f1-48886843966f::DEBUG::2014-04-27 12:43:55,008::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
Thread-13::DEBUG::2014-04-27 12:43:55,009::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 0 aborting False
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::threadPool::200::Misc.ThreadPool.WorkerThread::(_processNextTask) Task: 007f8290-5a3f-44ed-8802-c9a26c3718ae running: <bound method Task.commit of <storage.task.Task instance at 0x1c29a70>> with: None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::1194::TaskManager.Task::(commit) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::committing task: 007f8290-5a3f-44ed-8802-c9a26c3718ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state queued -> state running
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::905::TaskManager.Task::(_runJobs) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {})
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,009::task::331::TaskManager.Task::(run) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fcd000eac50>> (args: (-1, '-1', 250, 3) kwargs: {}) callback None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,010::misc::762::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,010::misc::764::SamplingMethod::(__call__) Got in to sampling method
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,019::misc::772::SamplingMethod::(__call__) Returning last result
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::clusterlock::337::LocalLock::(acquireHostId) Host id for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,020::clusterlock::365::LocalLock::(acquire) Acquiring local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::clusterlock::400::LocalLock::(acquire) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully acquired (id: 1)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,020::sp::266::Storage.StoragePool::(startSpm) spm lock acquired successfully
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,029::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,030::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,030::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`bc5cd286-6e68-45d3-9220-a09aa59a3c27`::Request was made in '/usr/share/vdsm/storage/sp.py' line '425' at '_upgradePool'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,031::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`bc5cd286-6e68-45d3-9220-a09aa59a3c27`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,031::sp::431::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `0a897f2e-1b01-4577-9f91-cd136ef4a978`
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`ea94ba76-06d3-49b9-9a52-a6f4ac6f5f53`::Request was made in '/usr/share/vdsm/storage/sp.py' line '433' at '_upgradePool'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,032::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978`ReqID=`ea94ba76-06d3-49b9-9a52-a6f4ac6f5f53`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,034::fileSD::140::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,043::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::fileSD::575::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,044::sd::374::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_imageNS already registered
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,044::sd::382::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 0a897f2e-1b01-4577-9f91-cd136ef4a978_volumeNS already registered
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,044::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,045::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0a897f2e-1b01-4577-9f91-cd136ef4a978', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,045::sp::436::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::sp::443::Storage.StoragePool::(_upgradePool) Registering with state change event
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,054::sp::446::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
Thread-222::DEBUG::2014-04-27 12:43:55,054::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 7b083758-45f9-4896-913d-11fe02043e6e
Thread-223::DEBUG::2014-04-27 12:43:55,055::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain e637bb04-a8b7-4c77-809c-d58051494c52
Thread-224::DEBUG::2014-04-27 12:43:55,055::sp::152::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain c2c4ade6-049e-4159-a294-a0c151f4983d
Thread-222::DEBUG::2014-04-27 12:43:55,055::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`1b4af6e7-372b-459e-bc09-9e1743d8340f`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-225::ERROR::2014-04-27 12:43:55,056::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,056::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
Thread-223::DEBUG::2014-04-27 12:43:55,056::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`43f758a8-c60c-4991-8849-f1f5281c034e`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-224::DEBUG::2014-04-27 12:43:55,056::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`7a4c6bbc-a127-4b64-8ffd-8cde731c5746`::Request was made in '/usr/share/vdsm/storage/sp.py' line '163' at '_upgradePoolDomain'
Thread-222::DEBUG::2014-04-27 12:43:55,056::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-225::ERROR::2014-04-27 12:43:55,057::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,057::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,057::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-224::DEBUG::2014-04-27 12:43:55,057::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-225::DEBUG::2014-04-27 12:43:55,057::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,057::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
Thread-225::DEBUG::2014-04-27 12:43:55,058::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,058::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
Thread-222::DEBUG::2014-04-27 12:43:55,062::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,062::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`1b4af6e7-372b-459e-bc09-9e1743d8340f`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,062::sd::623::Storage.StorageDomain::(createMasterTree) creating vms dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/vms
Thread-223::DEBUG::2014-04-27 12:43:55,062::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-223::DEBUG::2014-04-27 12:43:55,063::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`43f758a8-c60c-4991-8849-f1f5281c034e`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,063::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a87d0814-57f4-4980-8f30-ad2c26dd120c`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,063::sd::626::Storage.StorageDomain::(createMasterTree) creating task dir: /rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/master/tasks
Thread-223::DEBUG::2014-04-27 12:43:55,064::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ae9bcb49-96e6-4285-8254-21172289e844`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-224::DEBUG::2014-04-27 12:43:55,064::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,064::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' for lock type 'exclusive'
Thread-223::DEBUG::2014-04-27 12:43:55,064::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' for lock type 'exclusive'
Thread-224::DEBUG::2014-04-27 12:43:55,064::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`7a4c6bbc-a127-4b64-8ffd-8cde731c5746`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,065::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free. Now locking as 'exclusive' (1 active user)
Thread-224::DEBUG::2014-04-27 12:43:55,065::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`f620074f-c736-415b-acbd-35639f6451b4`::Request was made in '/usr/share/vdsm/storage/sp.py' line '165' at '_upgradePoolDomain'
Thread-222::DEBUG::2014-04-27 12:43:55,065::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.7b083758-45f9-4896-913d-11fe02043e6e`ReqID=`a87d0814-57f4-4980-8f30-ad2c26dd120c`::Granted request
Thread-224::DEBUG::2014-04-27 12:43:55,066::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' for lock type 'exclusive'
Thread-223::DEBUG::2014-04-27 12:43:55,066::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free. Now locking as 'exclusive' (1 active user)
Thread-223::DEBUG::2014-04-27 12:43:55,066::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.e637bb04-a8b7-4c77-809c-d58051494c52`ReqID=`ae9bcb49-96e6-4285-8254-21172289e844`::Granted request
Thread-222::DEBUG::2014-04-27 12:43:55,066::sp::175::Storage.StoragePool::(_upgradePoolDomain) Domain `7b083758-45f9-4896-913d-11fe02043e6e` is not a data domain it is an 2 domain, not upgrading
Thread-224::DEBUG::2014-04-27 12:43:55,066::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free. Now locking as 'exclusive' (1 active user)
Thread-222::DEBUG::2014-04-27 12:43:55,067::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e'
Thread-224::DEBUG::2014-04-27 12:43:55,067::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.c2c4ade6-049e-4159-a294-a0c151f4983d`ReqID=`f620074f-c736-415b-acbd-35639f6451b4`::Granted request
Thread-225::DEBUG::2014-04-27 12:43:55,072::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-225::WARNING::2014-04-27 12:43:55,073::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-225::DEBUG::2014-04-27 12:43:55,073::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-222::DEBUG::2014-04-27 12:43:55,078::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-222::DEBUG::2014-04-27 12:43:55,082::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=68a337432f9f1794a057deb08eb27f6bb3513470']
Thread-224::DEBUG::2014-04-27 12:43:55,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_app1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_app1', 'ROLE=Regular', 'SDUUID=c2c4ade6-049e-4159-a294-a0c151f4983d', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=5eda40da414dcb977d9d94d0ed6ad1162a55d69f']
Thread-223::DEBUG::2014-04-27 12:43:55,086::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2usr_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2usr_boot1', 'ROLE=Regular', 'SDUUID=e637bb04-a8b7-4c77-809c-d58051494c52', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=1f8ddad2bfec7f562b2e20b339649fe8c088a3d9']
Thread-222::DEBUG::2014-04-27 12:43:55,085::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-224::DEBUG::2014-04-27 12:43:55,090::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-225::ERROR::2014-04-27 12:43:55,091::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-223::DEBUG::2014-04-27 12:43:55,092::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52'
Thread-222::DEBUG::2014-04-27 12:43:55,092::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e'
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,093::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-224::DEBUG::2014-04-27 12:43:55,093::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,095::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-224::DEBUG::2014-04-27 12:43:55,096::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,096::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-224::DEBUG::2014-04-27 12:43:55,096::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-223::DEBUG::2014-04-27 12:43:55,097::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,098::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-224::DEBUG::2014-04-27 12:43:55,098::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d'
Thread-223::DEBUG::2014-04-27 12:43:55,099::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
Thread-222::DEBUG::2014-04-27 12:43:55,100::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' (0 active users)
Thread-222::DEBUG::2014-04-27 12:43:55,101::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e' is free, finding out if anyone is waiting for it.
Thread-223::DEBUG::2014-04-27 12:43:55,101::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,102::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-222::DEBUG::2014-04-27 12:43:55,102::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_7b083758-45f9-4896-913d-11fe02043e6e', Clearing records.
Thread-224::DEBUG::2014-04-27 12:43:55,106::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' (0 active users)
Thread-224::DEBUG::2014-04-27 12:43:55,107::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d' is free, finding out if anyone is waiting for it.
Thread-224::DEBUG::2014-04-27 12:43:55,107::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c2c4ade6-049e-4159-a294-a0c151f4983d', Clearing records.
Thread-223::DEBUG::2014-04-27 12:43:55,107::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' (0 active users)
Thread-223::DEBUG::2014-04-27 12:43:55,108::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52' is free, finding out if anyone is waiting for it.
Thread-223::DEBUG::2014-04-27 12:43:55,108::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_e637bb04-a8b7-4c77-809c-d58051494c52', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,116::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
007f8290-5a3f-44ed-8802-c9a26c3718ae::WARNING::2014-04-27 12:43:55,116::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,117::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 285, in startSpm
self.checkBackupDomain(__securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1409, in checkBackupDomain
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,127::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,128::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,128::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,142::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
007f8290-5a3f-44ed-8802-c9a26c3718ae::WARNING::2014-04-27 12:43:55,142::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,142::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sp::329::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,153::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,153::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f2cbe0e5-6641-4b51-a705-d9e9935275b6`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002`ReqID=`f2cbe0e5-6641-4b51-a705-d9e9935275b6`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`46da9ab1-a47d-4017-91d9-223eb4614818`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,154::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' for lock type 'exclusive'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free. Now locking as 'exclusive' (1 active user)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae`ReqID=`46da9ab1-a47d-4017-91d9-223eb4614818`::Granted request
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,155::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_9a4a80a1-5377-4a94-ade3-e58183e916ae', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,156::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,157::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=s2data1_s2mgt_boot1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=BDS_DataCentre2', 'POOL_DOMAINS=7b083758-45f9-4896-913d-11fe02043e6e:Active,e637bb04-a8b7-4c77-809c-d58051494c52:Active,0a897f2e-1b01-4577-9f91-cd136ef4a978:Active,c2c4ade6-049e-4159-a294-a0c151f4983d:Active,9a4a80a1-5377-4a94-ade3-e58183e916ae:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=c713062f-300f-4256-9ac8-2d3fcfcdb002', 'REMOTE_PATH=/vdsm_store/s2data1_s2mgt_boot1', 'ROLE=Master', 'SDUUID=0a897f2e-1b01-4577-9f91-cd136ef4a978', 'TYPE=LOCALFS', 'VERSION=3', '_SHA_CKSUM=afe618d7596d75d0fb96453bcdd34a1255534454']
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
007f8290-5a3f-44ed-8802-c9a26c3718ae::INFO::2014-04-27 12:43:55,158::clusterlock::409::LocalLock::(release) Releasing local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::clusterlock::422::LocalLock::(release) Local lock for domain 0a897f2e-1b01-4577-9f91-cd136ef4a978 successfully released
007f8290-5a3f-44ed-8802-c9a26c3718ae::ERROR::2014-04-27 12:43:55,158::task::866::TaskManager.Task::(_setError) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 334, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm
self._updateDomainsRole()
File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole
domain = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::885::TaskManager.Task::(_run) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task._run: 007f8290-5a3f-44ed-8802-c9a26c3718ae () {} failed - stopping task
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::1211::TaskManager.Task::(stop) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::stopping in state running (force False)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,158::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 1 aborting True
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::916::TaskManager.Task::(_runJobs) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::aborting: Task is aborted: 'Storage domain does not exist' - code 358
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::990::TaskManager.Task::(_decref) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::ref 0 aborting True
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::925::TaskManager.Task::(_doAbort) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Task._doAbort: force False
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state running -> state aborting
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::550::TaskManager.Task::(__state_aborting) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_aborting: recover policy auto
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state aborting -> state racquiring
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state racquiring -> state recovering
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,159::task::798::TaskManager.Task::(_recover) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::_recover
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::805::TaskManager.Task::(_recover) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::running recovery None
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::786::TaskManager.Task::(_recoverDone) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::Recover Done: state recovering
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::task::595::TaskManager.Task::(_updateState) Task=`007f8290-5a3f-44ed-8802-c9a26c3718ae`::moving from state recovering -> state recovered
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002': < ResourceRef 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', isValid: 'True' obj: 'None'>}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002'
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' (0 active users)
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,160::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002' is free, finding out if anyone is waiting for it.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,161::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c713062f-300f-4256-9ac8-2d3fcfcdb002', Clearing records.
007f8290-5a3f-44ed-8802-c9a26c3718ae::DEBUG::2014-04-27 12:43:55,161::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
Thread-29::ERROR::2014-04-27 12:43:55,963::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::ERROR::2014-04-27 12:43:55,964::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 9a4a80a1-5377-4a94-ade3-e58183e916ae
Thread-29::DEBUG::2014-04-27 12:43:55,964::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2014-04-27 12:43:55,964::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 9a4a80a1-5377-4a94-ade3-e58183e916ae' (cwd None)
Thread-29::DEBUG::2014-04-27 12:43:55,977::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found\n Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae\n'; <rc> = 5
Thread-29::WARNING::2014-04-27 12:43:55,978::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "9a4a80a1-5377-4a94-ade3-e58183e916ae" not found', ' Skipping volume group 9a4a80a1-5377-4a94-ade3-e58183e916ae']
Thread-29::DEBUG::2014-04-27 12:43:55,978::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-29::ERROR::2014-04-27 12:43:55,988::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 9a4a80a1-5377-4a94-ade3-e58183e916ae not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-29::ERROR::2014-04-27 12:43:55,989::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 9a4a80a1-5377-4a94-ade3-e58183e916ae monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('9a4a80a1-5377-4a94-ade3-e58183e916ae',)
Thread-13::DEBUG::2014-04-27 12:43:56,055::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,057::task::595::TaskManager.Task::(_updateState) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,059::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='007f8290-5a3f-44ed-8802-c9a26c3718ae', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:56,060::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:56,062::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}
Thread-13::INFO::2014-04-27 12:43:56,063::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}}
Thread-13::DEBUG::2014-04-27 12:43:56,065::task::1185::TaskManager.Task::(prepare) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '007f8290-5a3f-44ed-8802-c9a26c3718ae'}}
Thread-13::DEBUG::2014-04-27 12:43:56,066::task::595::TaskManager.Task::(_updateState) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,068::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,069::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,071::task::990::TaskManager.Task::(_decref) Task=`9d7af804-2ba0-40c0-8273-aca34bd0a23a`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:56,083::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,085::task::595::TaskManager.Task::(_updateState) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,087::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='c713062f-300f-4256-9ac8-2d3fcfcdb002', options=None)
Thread-13::INFO::2014-04-27 12:43:56,089::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:56,091::task::1185::TaskManager.Task::(prepare) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}}
Thread-13::DEBUG::2014-04-27 12:43:56,092::task::595::TaskManager.Task::(_updateState) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,094::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,095::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,097::task::990::TaskManager.Task::(_decref) Task=`6d49640d-f1de-456d-9290-95f7a978f12e`::ref 0 aborting False
Thread-13::DEBUG::2014-04-27 12:43:56,146::BindingXMLRPC::251::vds::(wrapper) client [10.50.0.18]
Thread-13::DEBUG::2014-04-27 12:43:56,149::task::595::TaskManager.Task::(_updateState) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::moving from state init -> state preparing
Thread-13::INFO::2014-04-27 12:43:56,151::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='007f8290-5a3f-44ed-8802-c9a26c3718ae', spUUID=None, options=None)
Thread-13::DEBUG::2014-04-27 12:43:56,152::taskManager::161::TaskManager::(clearTask) Entry. taskID: 007f8290-5a3f-44ed-8802-c9a26c3718ae
Thread-13::DEBUG::2014-04-27 12:43:56,153::taskManager::166::TaskManager::(clearTask) Return.
Thread-13::INFO::2014-04-27 12:43:56,154::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
Thread-22::DEBUG::2014-04-27 12:43:56,155::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__boot1/0a897f2e-1b01-4577-9f91-cd136ef4a978/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-13::DEBUG::2014-04-27 12:43:56,159::task::1185::TaskManager.Task::(prepare) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::finished: None
Thread-13::DEBUG::2014-04-27 12:43:56,161::task::595::TaskManager.Task::(_updateState) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::moving from state preparing -> state finished
Thread-13::DEBUG::2014-04-27 12:43:56,162::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-13::DEBUG::2014-04-27 12:43:56,163::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-13::DEBUG::2014-04-27 12:43:56,166::task::990::TaskManager.Task::(_decref) Task=`de2f0e09-eb41-421e-a06c-22c92b424e38`::ref 0 aborting False
Thread-22::DEBUG::2014-04-27 12:43:56,175::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n665 bytes (665 B) copied, 8.9833e-05 s, 7.4 MB/s\n'; <rc> = 0
Thread-17::DEBUG::2014-04-27 12:43:56,179::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt-noded:_iso/7b083758-45f9-4896-913d-11fe02043e6e/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-17::DEBUG::2014-04-27 12:43:56,198::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n373 bytes (373 B) copied, 0.00062052 s, 601 kB/s\n'; <rc> = 0
Thread-27::DEBUG::2014-04-27 12:43:56,290::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2mgt__app1/c2c4ade6-049e-4159-a294-a0c151f4983d/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-18::DEBUG::2014-04-27 12:43:56,295::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/_vdsm__store_s2data1__s2usr__boot1/e637bb04-a8b7-4c77-809c-d58051494c52/dom_md/metadata bs=4096 count=1' (cwd None)
Thread-27::DEBUG::2014-04-27 12:43:56,302::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n367 bytes (367 B) copied, 0.00015041 s, 2.4 MB/s\n'; <rc> = 0
Thread-18::DEBUG::2014-04-27 12:43:56,304::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n369 bytes (369 B) copied, 7.0876e-05 s, 5.2 MB/s\n'; <rc> = 0
More information about the Users
mailing list