MainThread::INFO::2017-02-19 13:39:57,971::vdsm::135::vds::(run) (PID: 30122) I am the actual vdsm 4.18.21-1.el7.centos node-01.eurooil.ru (3.10.0-514.6.1.el7.x86_64) MainThread::INFO::2017-02-19 13:39:57,971::vdsm::235::vds::(__set_cpu_affinity) VDSM will run with cpu affinity: frozenset([1]) MainThread::DEBUG::2017-02-19 13:39:57,972::commands::68::root::(execCmd) /usr/bin/taskset --all-tasks --pid --cpu-list 1 30122 (cwd None) MainThread::DEBUG::2017-02-19 13:39:57,974::commands::86::root::(execCmd) SUCCESS: = ''; = 0 MainThread::DEBUG::2017-02-19 13:39:57,975::resourceManager::426::Storage.ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2017-02-19 13:39:57,975::threadPool::33::Storage.ThreadPool::(__init__) Enter - numThreads: 10, waitTimeout: 3, maxTasks: 500 MainThread::DEBUG::2017-02-19 13:39:57,978::fileUtils::209::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt mode: None MainThread::DEBUG::2017-02-19 13:39:57,978::fileUtils::218::Storage.fileUtils::(createdir) Using existing directory: /rhev/data-center/mnt MainThread::DEBUG::2017-02-19 13:39:57,981::sp::379::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/c7c466c5-983d-40b2-8f8c-d569755281f5/master` is not mounted, skipping MainThread::DEBUG::2017-02-19 13:39:58,016::hsm::413::Storage.Misc.excCmd::(__validateLvmLockingType) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm dumpconfig global/locking_type (cwd None) MainThread::DEBUG::2017-02-19 13:39:58,033::hsm::413::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: = ''; = 0 MainThread::DEBUG::2017-02-19 13:39:58,033::hsm::437::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' MainThread::DEBUG::2017-02-19 13:39:58,034::hsm::469::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] MainThread::DEBUG::2017-02-19 13:39:58,034::hsm::470::Storage.HSM::(__cleanStorageRepository) Mount list: [] MainThread::DEBUG::2017-02-19 13:39:58,035::hsm::472::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers MainThread::DEBUG::2017-02-19 13:39:58,036::hsm::515::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' MainThread::INFO::2017-02-19 13:39:58,036::check::90::storage.check::(start) Starting check service storageRefresh::DEBUG::2017-02-19 13:39:58,036::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) storageRefresh::DEBUG::2017-02-19 13:39:58,037::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:39:58,037::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) storageRefresh::DEBUG::2017-02-19 13:39:58,037::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:39:58,037::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds storageRefresh::DEBUG::2017-02-19 13:39:58,037::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R (cwd None) MainThread::INFO::2017-02-19 13:39:58,042::dispatcher::47::Storage.Dispatcher::(__init__) Starting StorageDispatcher... MainThread::DEBUG::2017-02-19 13:39:58,042::schedule::98::Scheduler::(start) Starting scheduler vdsm.Scheduler Thread-12::INFO::2017-02-19 13:39:58,042::asyncevent::121::storage.asyncevent::(run_forever) Starting MainThread::DEBUG::2017-02-19 13:39:58,115::task::597::Storage.TaskManager.Task::(_updateState) Task=`9ca18c2c-bd6f-4667-949d-ab6954b5e64e`::moving from state init -> state preparing MainThread::INFO::2017-02-19 13:39:58,116::logUtils::49::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=) MainThread::INFO::2017-02-19 13:39:58,116::logUtils::52::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None MainThread::DEBUG::2017-02-19 13:39:58,116::task::1193::Storage.TaskManager.Task::(prepare) Task=`9ca18c2c-bd6f-4667-949d-ab6954b5e64e`::finished: None MainThread::DEBUG::2017-02-19 13:39:58,116::task::597::Storage.TaskManager.Task::(_updateState) Task=`9ca18c2c-bd6f-4667-949d-ab6954b5e64e`::moving from state preparing -> state finished MainThread::DEBUG::2017-02-19 13:39:58,116::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} MainThread::DEBUG::2017-02-19 13:39:58,116::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} MainThread::DEBUG::2017-02-19 13:39:58,116::task::995::Storage.TaskManager.Task::(_decref) Task=`9ca18c2c-bd6f-4667-949d-ab6954b5e64e`::ref 0 aborting False MainThread::INFO::2017-02-19 13:39:58,116::momIF::49::MOM::(__init__) Preparing MOM interface MainThread::INFO::2017-02-19 13:39:58,118::momIF::58::MOM::(__init__) Using named unix socket /var/run/vdsm/mom-vdsm.sock MainThread::INFO::2017-02-19 13:39:58,119::secret::91::root::(clear) Unregistering all secrets MainThread::DEBUG::2017-02-19 13:39:58,119::libvirtconnection::160::root::(get) trying to connect libvirt storageRefresh::DEBUG::2017-02-19 13:39:58,120::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:39:58,120::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (vdsm.storage.hba.rescan) storageRefresh::DEBUG::2017-02-19 13:39:58,120::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:39:58,120::hba::58::Storage.HBA::(rescan) Starting scan storageRefresh::DEBUG::2017-02-19 13:39:58,121::supervdsm::79::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm vdsm.Scheduler::DEBUG::2017-02-19 13:39:58,122::schedule::140::Scheduler::(_run) started MainThread::INFO::2017-02-19 13:39:58,129::vmchannels::218::vds::(settimeout) Setting channels' timeout to 30 seconds. clientIFinit::DEBUG::2017-02-19 13:39:58,130::clientIF::468::vds::(_recoverExistingVms) recovery: started MainThread::INFO::2017-02-19 13:39:58,132::protocoldetector::186::vds.MultiProtocolAcceptor::(__init__) Listening at :::54321 MainThread::DEBUG::2017-02-19 13:39:58,134::protocoldetector::213::vds.MultiProtocolAcceptor::(add_detector) Adding detector VM Channels Listener::DEBUG::2017-02-19 13:39:58,180::vmchannels::200::vds::(run) Starting VM channels listener thread. clientIFinit::INFO::2017-02-19 13:39:58,182::clientIF::495::vds::(_recoverExistingVms) recovery: completed in 0s storageRefresh::DEBUG::2017-02-19 13:39:58,194::hba::64::Storage.HBA::(rescan) Scan finished storageRefresh::DEBUG::2017-02-19 13:39:58,194::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:39:58,195::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/udevadm settle --timeout=5 (cwd None) storageRefresh::DEBUG::2017-02-19 13:39:58,223::commands::86::root::(execCmd) SUCCESS: = ''; = 0 storageRefresh::DEBUG::2017-02-19 13:39:58,224::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:39:58,225::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm pvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size (cwd None) storageRefresh::DEBUG::2017-02-19 13:39:58,263::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:39:58,264::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) storageRefresh::DEBUG::2017-02-19 13:39:58,308::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:39:58,308::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags (cwd None) storageRefresh::DEBUG::2017-02-19 13:39:58,350::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:39:58,351::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) storageRefresh::DEBUG::2017-02-19 13:39:58,384::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:39:58,384::hsm::383::Storage.HSM::(storageRefresh) HSM is ready MainThread::DEBUG::2017-02-19 13:39:58,399::protocoldetector::213::vds.MultiProtocolAcceptor::(add_detector) Adding detector MainThread::DEBUG::2017-02-19 13:39:58,400::executor::71::Executor::(start) Starting executor MainThread::DEBUG::2017-02-19 13:39:58,400::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/0 jsonrpc.Executor/0::DEBUG::2017-02-19 13:39:58,400::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,400::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/1 MainThread::DEBUG::2017-02-19 13:39:58,401::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/2 jsonrpc.Executor/1::DEBUG::2017-02-19 13:39:58,401::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,401::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/3 MainThread::DEBUG::2017-02-19 13:39:58,401::vdsm::66::vds::(sigtermHandler) Received signal 15 jsonrpc.Executor/2::DEBUG::2017-02-19 13:39:58,402::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,402::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/4 jsonrpc.Executor/3::DEBUG::2017-02-19 13:39:58,402::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,402::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:39:58,402::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,403::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/6 jsonrpc.Executor/5::DEBUG::2017-02-19 13:39:58,403::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,403::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/7 jsonrpc.Executor/6::DEBUG::2017-02-19 13:39:58,404::executor::214::Executor::(_run) Worker started jsonrpc.Executor/7::DEBUG::2017-02-19 13:39:58,404::executor::214::Executor::(_run) Worker started BindingXMLRPC::INFO::2017-02-19 13:39:58,404::bindingxmlrpc::63::vds::(threaded_start) XMLRPC server running MainThread::DEBUG::2017-02-19 13:39:58,405::executor::71::Executor::(start) Starting executor MainThread::DEBUG::2017-02-19 13:39:58,406::executor::196::Executor::(__init__) Starting worker periodic/0 MainThread::DEBUG::2017-02-19 13:39:58,406::executor::196::Executor::(__init__) Starting worker periodic/1 periodic/0::DEBUG::2017-02-19 13:39:58,406::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,406::executor::196::Executor::(__init__) Starting worker periodic/2 periodic/1::DEBUG::2017-02-19 13:39:58,407::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,407::executor::196::Executor::(__init__) Starting worker periodic/3 periodic/2::DEBUG::2017-02-19 13:39:58,407::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,408::libvirtconnection::160::root::(get) trying to connect libvirt periodic/3::DEBUG::2017-02-19 13:39:58,409::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:39:58,414::periodic::167::virt.periodic.Operation::(start) starting operation at 0x3289a90> MainThread::DEBUG::2017-02-19 13:39:58,414::periodic::167::virt.periodic.Operation::(start) starting operation at 0x329c790> MainThread::DEBUG::2017-02-19 13:39:58,414::periodic::167::virt.periodic.Operation::(start) starting operation at 0x329c810> MainThread::DEBUG::2017-02-19 13:39:58,414::periodic::167::virt.periodic.Operation::(start) starting operation MainThread::DEBUG::2017-02-19 13:39:58,414::periodic::167::virt.periodic.Operation::(start) starting operation at 0x329cb50> MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::167::virt.periodic.Operation::(start) starting operation MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation at 0x3289a90> MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation at 0x329c790> MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation at 0x329c810> MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation at 0x329cb50> MainThread::DEBUG::2017-02-19 13:39:58,415::periodic::176::virt.periodic.Operation::(stop) stopping operation MainThread::DEBUG::2017-02-19 13:39:58,415::executor::80::Executor::(stop) Stopping executor MainThread::DEBUG::2017-02-19 13:39:58,416::protocoldetector::217::vds.MultiProtocolAcceptor::(stop) Stopping Acceptor MainThread::INFO::2017-02-19 13:39:58,416::__init__::636::jsonrpc.JsonRpcServer::(stop) Stopping JsonRPC Server MainThread::DEBUG::2017-02-19 13:39:58,416::executor::80::Executor::(stop) Stopping executor MainThread::DEBUG::2017-02-19 13:39:58,416::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/6 jsonrpc.Executor/0::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/1::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/2::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/3::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/4::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/5::DEBUG::2017-02-19 13:39:58,417::executor::219::Executor::(_run) Worker stopped jsonrpc.Executor/6::DEBUG::2017-02-19 13:39:58,418::executor::219::Executor::(_run) Worker stopped MainThread::DEBUG::2017-02-19 13:39:58,418::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/4 MainThread::DEBUG::2017-02-19 13:39:58,418::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/2 MainThread::DEBUG::2017-02-19 13:39:58,418::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/7 jsonrpc.Executor/7::DEBUG::2017-02-19 13:39:58,418::executor::219::Executor::(_run) Worker stopped MainThread::DEBUG::2017-02-19 13:39:58,418::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/0 MainThread::DEBUG::2017-02-19 13:39:58,418::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/5 MainThread::DEBUG::2017-02-19 13:39:58,419::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/3 MainThread::DEBUG::2017-02-19 13:39:58,419::executor::205::Executor::(join) Waiting for worker jsonrpc.Executor/1 MainThread::INFO::2017-02-19 13:39:58,419::bindingxmlrpc::85::vds::(stop) Stopping XMLRPC server BindingXMLRPC::INFO::2017-02-19 13:39:58,419::bindingxmlrpc::75::vds::(threaded_start) XMLRPC server stopped MainThread::INFO::2017-02-19 13:39:58,419::secret::91::root::(clear) Unregistering all secrets MainThread::DEBUG::2017-02-19 13:39:58,420::vmchannels::214::vds::(stop) VM channels listener was stopped. MainThread::DEBUG::2017-02-19 13:39:58,420::task::597::Storage.TaskManager.Task::(_updateState) Task=`4107249f-b7e2-4312-acd5-2c52f2db433c`::moving from state init -> state preparing MainThread::INFO::2017-02-19 13:39:58,420::logUtils::49::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) Thread-11::DEBUG::2017-02-19 13:39:58,421::storageServer::873::Storage.ConnectionMonitor::(_monitorConnections) Monitoring stopped MainThread::DEBUG::2017-02-19 13:39:58,422::sp::379::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/c7c466c5-983d-40b2-8f8c-d569755281f5/master` is not mounted, skipping MainThread::INFO::2017-02-19 13:39:58,454::monitor::222::Storage.Monitor::(shutdown) Shutting down domain monitors MainThread::INFO::2017-02-19 13:39:58,454::check::103::storage.check::(stop) Stopping check service Thread-12::INFO::2017-02-19 13:39:58,454::asyncevent::193::storage.asyncevent::(stop) Stopping Thread-12::DEBUG::2017-02-19 13:39:58,454::asyncevent::129::storage.asyncevent::(run_forever) stopped MainThread::DEBUG::2017-02-19 13:39:58,455::asyncevent::214::storage.asyncevent::(close) Closing MainThread::DEBUG::2017-02-19 13:39:58,455::taskManager::91::Storage.TaskManager::(prepareForShutdown) Request to stop all tasks MainThread::INFO::2017-02-19 13:39:58,455::logUtils::52::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None MainThread::DEBUG::2017-02-19 13:39:58,455::task::1193::Storage.TaskManager.Task::(prepare) Task=`4107249f-b7e2-4312-acd5-2c52f2db433c`::finished: None MainThread::DEBUG::2017-02-19 13:39:58,455::task::597::Storage.TaskManager.Task::(_updateState) Task=`4107249f-b7e2-4312-acd5-2c52f2db433c`::moving from state preparing -> state finished MainThread::DEBUG::2017-02-19 13:39:58,455::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} MainThread::DEBUG::2017-02-19 13:39:58,455::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} MainThread::DEBUG::2017-02-19 13:39:58,455::task::995::Storage.TaskManager.Task::(_decref) Task=`4107249f-b7e2-4312-acd5-2c52f2db433c`::ref 0 aborting False MainThread::DEBUG::2017-02-19 13:39:58,456::schedule::110::Scheduler::(stop) Stopping scheduler vdsm.Scheduler vdsm.Scheduler::DEBUG::2017-02-19 13:39:58,456::schedule::143::Scheduler::(_run) stopped MainThread::INFO::2017-02-19 13:39:58,456::vdsm::147::vds::(run) VDSM main thread ended. Waiting for 16 other threads... MainThread::INFO::2017-02-19 13:39:58,456::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,456::vdsm::151::vds::(run) MainThread::DEBUG::2017-02-19 13:39:58,456::vmchannels::214::vds::(stop) VM channels listener was stopped. MainThread::INFO::2017-02-19 13:39:58,456::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) <_MainThread(MainThread, started 140270208096064)> MainThread::INFO::2017-02-19 13:39:58,457::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,458::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,458::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,458::vdsm::151::vds::(run) MainThread::INFO::2017-02-19 13:39:58,458::vdsm::151::vds::(run) periodic/0::DEBUG::2017-02-19 13:39:58,420::executor::219::Executor::(_run) Worker stopped periodic/1::DEBUG::2017-02-19 13:39:58,458::executor::219::Executor::(_run) Worker stopped periodic/2::DEBUG::2017-02-19 13:39:58,459::executor::219::Executor::(_run) Worker stopped periodic/3::DEBUG::2017-02-19 13:39:58,459::executor::219::Executor::(_run) Worker stopped MainThread::INFO::2017-02-19 13:40:00,670::vdsm::135::vds::(run) (PID: 30322) I am the actual vdsm 4.18.21-1.el7.centos node-01.eurooil.ru (3.10.0-514.6.1.el7.x86_64) MainThread::INFO::2017-02-19 13:40:00,670::vdsm::235::vds::(__set_cpu_affinity) VDSM will run with cpu affinity: frozenset([1]) MainThread::DEBUG::2017-02-19 13:40:00,670::commands::68::root::(execCmd) /usr/bin/taskset --all-tasks --pid --cpu-list 1 30322 (cwd None) MainThread::DEBUG::2017-02-19 13:40:00,673::commands::86::root::(execCmd) SUCCESS: = ''; = 0 MainThread::DEBUG::2017-02-19 13:40:00,674::resourceManager::426::Storage.ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2017-02-19 13:40:00,674::threadPool::33::Storage.ThreadPool::(__init__) Enter - numThreads: 10, waitTimeout: 3, maxTasks: 500 MainThread::DEBUG::2017-02-19 13:40:00,676::fileUtils::209::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt mode: None MainThread::DEBUG::2017-02-19 13:40:00,677::fileUtils::218::Storage.fileUtils::(createdir) Using existing directory: /rhev/data-center/mnt MainThread::DEBUG::2017-02-19 13:40:00,679::sp::379::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/c7c466c5-983d-40b2-8f8c-d569755281f5/master` is not mounted, skipping MainThread::DEBUG::2017-02-19 13:40:00,709::hsm::413::Storage.Misc.excCmd::(__validateLvmLockingType) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm dumpconfig global/locking_type (cwd None) MainThread::DEBUG::2017-02-19 13:40:00,723::hsm::413::Storage.Misc.excCmd::(__validateLvmLockingType) SUCCESS: = ''; = 0 MainThread::DEBUG::2017-02-19 13:40:00,723::hsm::437::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' MainThread::DEBUG::2017-02-19 13:40:00,725::hsm::469::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] MainThread::DEBUG::2017-02-19 13:40:00,725::hsm::470::Storage.HSM::(__cleanStorageRepository) Mount list: [] MainThread::DEBUG::2017-02-19 13:40:00,725::hsm::472::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers MainThread::DEBUG::2017-02-19 13:40:00,726::hsm::515::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' MainThread::INFO::2017-02-19 13:40:00,726::check::90::storage.check::(start) Starting check service storageRefresh::DEBUG::2017-02-19 13:40:00,727::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) storageRefresh::DEBUG::2017-02-19 13:40:00,727::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:40:00,727::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) storageRefresh::DEBUG::2017-02-19 13:40:00,727::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:40:00,727::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds storageRefresh::DEBUG::2017-02-19 13:40:00,727::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R (cwd None) MainThread::INFO::2017-02-19 13:40:00,731::dispatcher::47::Storage.Dispatcher::(__init__) Starting StorageDispatcher... MainThread::DEBUG::2017-02-19 13:40:00,731::schedule::98::Scheduler::(start) Starting scheduler vdsm.Scheduler Thread-12::INFO::2017-02-19 13:40:00,732::asyncevent::121::storage.asyncevent::(run_forever) Starting MainThread::DEBUG::2017-02-19 13:40:00,803::task::597::Storage.TaskManager.Task::(_updateState) Task=`9786532c-c4d6-4c7f-82ad-cb598ab18b86`::moving from state init -> state preparing MainThread::INFO::2017-02-19 13:40:00,804::logUtils::49::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback(callbackFunc=) MainThread::INFO::2017-02-19 13:40:00,804::logUtils::52::dispatcher::(wrapper) Run and protect: registerDomainStateChangeCallback, Return response: None MainThread::DEBUG::2017-02-19 13:40:00,804::task::1193::Storage.TaskManager.Task::(prepare) Task=`9786532c-c4d6-4c7f-82ad-cb598ab18b86`::finished: None MainThread::DEBUG::2017-02-19 13:40:00,804::task::597::Storage.TaskManager.Task::(_updateState) Task=`9786532c-c4d6-4c7f-82ad-cb598ab18b86`::moving from state preparing -> state finished MainThread::DEBUG::2017-02-19 13:40:00,804::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} MainThread::DEBUG::2017-02-19 13:40:00,804::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} MainThread::DEBUG::2017-02-19 13:40:00,804::task::995::Storage.TaskManager.Task::(_decref) Task=`9786532c-c4d6-4c7f-82ad-cb598ab18b86`::ref 0 aborting False MainThread::INFO::2017-02-19 13:40:00,805::momIF::49::MOM::(__init__) Preparing MOM interface MainThread::INFO::2017-02-19 13:40:00,806::momIF::58::MOM::(__init__) Using named unix socket /var/run/vdsm/mom-vdsm.sock MainThread::INFO::2017-02-19 13:40:00,807::secret::91::root::(clear) Unregistering all secrets MainThread::DEBUG::2017-02-19 13:40:00,807::libvirtconnection::160::root::(get) trying to connect libvirt storageRefresh::DEBUG::2017-02-19 13:40:00,808::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:40:00,808::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (vdsm.storage.hba.rescan) storageRefresh::DEBUG::2017-02-19 13:40:00,808::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method storageRefresh::DEBUG::2017-02-19 13:40:00,808::hba::58::Storage.HBA::(rescan) Starting scan storageRefresh::DEBUG::2017-02-19 13:40:00,809::supervdsm::79::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm vdsm.Scheduler::DEBUG::2017-02-19 13:40:00,810::schedule::140::Scheduler::(_run) started MainThread::INFO::2017-02-19 13:40:00,817::vmchannels::218::vds::(settimeout) Setting channels' timeout to 30 seconds. clientIFinit::DEBUG::2017-02-19 13:40:00,817::clientIF::468::vds::(_recoverExistingVms) recovery: started MainThread::INFO::2017-02-19 13:40:00,819::protocoldetector::186::vds.MultiProtocolAcceptor::(__init__) Listening at :::54321 MainThread::DEBUG::2017-02-19 13:40:00,821::protocoldetector::213::vds.MultiProtocolAcceptor::(add_detector) Adding detector VM Channels Listener::DEBUG::2017-02-19 13:40:00,863::vmchannels::200::vds::(run) Starting VM channels listener thread. clientIFinit::INFO::2017-02-19 13:40:00,877::clientIF::495::vds::(_recoverExistingVms) recovery: completed in 0s storageRefresh::DEBUG::2017-02-19 13:40:00,918::hba::64::Storage.HBA::(rescan) Scan finished storageRefresh::DEBUG::2017-02-19 13:40:00,918::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:40:00,918::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/udevadm settle --timeout=5 (cwd None) storageRefresh::DEBUG::2017-02-19 13:40:00,944::commands::86::root::(execCmd) SUCCESS: = ''; = 0 storageRefresh::DEBUG::2017-02-19 13:40:00,946::misc::566::Storage.SamplingMethod::(__call__) Returning last result storageRefresh::DEBUG::2017-02-19 13:40:00,947::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm pvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size (cwd None) storageRefresh::DEBUG::2017-02-19 13:40:00,982::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:40:00,982::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) storageRefresh::DEBUG::2017-02-19 13:40:01,013::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:40:01,013::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags (cwd None) storageRefresh::DEBUG::2017-02-19 13:40:01,052::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:40:01,053::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) MainThread::DEBUG::2017-02-19 13:40:01,079::protocoldetector::213::vds.MultiProtocolAcceptor::(add_detector) Adding detector MainThread::DEBUG::2017-02-19 13:40:01,079::executor::71::Executor::(start) Starting executor MainThread::DEBUG::2017-02-19 13:40:01,080::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/0 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:01,080::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,080::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/1 MainThread::DEBUG::2017-02-19 13:40:01,081::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/2 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:01,081::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,081::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/3 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:01,081::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,081::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/4 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:01,082::executor::214::Executor::(_run) Worker started storageRefresh::DEBUG::2017-02-19 13:40:01,082::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 storageRefresh::DEBUG::2017-02-19 13:40:01,082::hsm::383::Storage.HSM::(storageRefresh) HSM is ready MainThread::DEBUG::2017-02-19 13:40:01,083::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:01,083::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,083::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/6 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:01,083::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,083::executor::196::Executor::(__init__) Starting worker jsonrpc.Executor/7 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:01,084::executor::214::Executor::(_run) Worker started jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:01,084::executor::214::Executor::(_run) Worker started BindingXMLRPC::INFO::2017-02-19 13:40:01,085::bindingxmlrpc::63::vds::(threaded_start) XMLRPC server running MainThread::DEBUG::2017-02-19 13:40:01,086::executor::71::Executor::(start) Starting executor MainThread::DEBUG::2017-02-19 13:40:01,086::executor::196::Executor::(__init__) Starting worker periodic/0 MainThread::DEBUG::2017-02-19 13:40:01,086::executor::196::Executor::(__init__) Starting worker periodic/1 periodic/0::DEBUG::2017-02-19 13:40:01,086::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,087::executor::196::Executor::(__init__) Starting worker periodic/2 periodic/1::DEBUG::2017-02-19 13:40:01,087::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,087::executor::196::Executor::(__init__) Starting worker periodic/3 periodic/2::DEBUG::2017-02-19 13:40:01,087::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,087::libvirtconnection::160::root::(get) trying to connect libvirt periodic/3::DEBUG::2017-02-19 13:40:01,088::executor::214::Executor::(_run) Worker started MainThread::DEBUG::2017-02-19 13:40:01,093::periodic::167::virt.periodic.Operation::(start) starting operation at 0x2816910> MainThread::DEBUG::2017-02-19 13:40:01,093::periodic::167::virt.periodic.Operation::(start) starting operation at 0x282a610> MainThread::DEBUG::2017-02-19 13:40:01,094::periodic::167::virt.periodic.Operation::(start) starting operation at 0x282a690> MainThread::DEBUG::2017-02-19 13:40:01,094::periodic::167::virt.periodic.Operation::(start) starting operation MainThread::DEBUG::2017-02-19 13:40:01,094::periodic::167::virt.periodic.Operation::(start) starting operation at 0x282a9d0> MainThread::DEBUG::2017-02-19 13:40:01,094::periodic::167::virt.periodic.Operation::(start) starting operation Reactor thread::INFO::2017-02-19 13:40:01,872::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52834 Reactor thread::DEBUG::2017-02-19 13:40:01,879::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:01,880::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::1:52834 Reactor thread::INFO::2017-02-19 13:40:01,881::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request JsonRpc (StompReactor)::INFO::2017-02-19 13:40:01,881::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received Reactor thread::DEBUG::2017-02-19 13:40:01,882::stompreactor::492::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::1', 52834) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:02,106::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getHardwareInfo' in bridge with {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:02,109::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'X9DBU', 'systemSerialNumber': '0123456789', 'systemFamily': 'To be filled by O.E.M.', 'systemVersion': '0123456789', 'systemUUID': '00000000-0000-0000-0000-0CC47A3C0510', 'systemManufacturer': 'Supermicro'} jsonrpc.Executor/0::INFO::2017-02-19 13:40:02,109::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getHardwareInfo succeeded in 0.00 seconds JsonRpc (StompReactor)::ERROR::2017-02-19 13:40:02,112::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof Reactor thread::INFO::2017-02-19 13:40:03,080::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52836 Reactor thread::DEBUG::2017-02-19 13:40:03,087::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:03,089::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::1:52836 Reactor thread::INFO::2017-02-19 13:40:03,090::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2017-02-19 13:40:03,090::stompreactor::492::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::1', 52836) JsonRpc (StompReactor)::INFO::2017-02-19 13:40:03,097::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:03,317::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.ping' in bridge with {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:03,318::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.ping' in bridge with True jsonrpc.Executor/1::INFO::2017-02-19 13:40:03,318::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.ping succeeded in 0.00 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:03,324::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getAllVmStats' in bridge with {} jsonrpc.Executor/2::INFO::2017-02-19 13:40:03,325::throttledlog::105::throttled::(log) Current getAllVmStats: {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:03,325::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getAllVmStats' in bridge with (suppressed) jsonrpc.Executor/2::INFO::2017-02-19 13:40:03,326::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getAllVmStats succeeded in 0.00 seconds Reactor thread::INFO::2017-02-19 13:40:04,117::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52838 Reactor thread::DEBUG::2017-02-19 13:40:04,123::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:04,124::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::1:52838 Reactor thread::INFO::2017-02-19 13:40:04,124::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2017-02-19 13:40:04,125::stompreactor::492::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::1', 52838) JsonRpc (StompReactor)::INFO::2017-02-19 13:40:04,125::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:04,360::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getHardwareInfo' in bridge with {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:04,362::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'X9DBU', 'systemSerialNumber': '0123456789', 'systemFamily': 'To be filled by O.E.M.', 'systemVersion': '0123456789', 'systemUUID': '00000000-0000-0000-0000-0CC47A3C0510', 'systemManufacturer': 'Supermicro'} jsonrpc.Executor/3::INFO::2017-02-19 13:40:04,362::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getHardwareInfo succeeded in 0.00 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,367::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'5457319a-3123-4ec1-9965-af7d9378d72d', u'connection': u'172.16.0.1', u'iqn': u'iqn.2003-01.org.linux-iscsi.storage-01.x8664:sn.1d56416909ce', u'portal': u'1', u'user': u'', u'password': '********', u'port': u'3260'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 3} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,369::task::597::Storage.TaskManager.Task::(_updateState) Task=`9cb9bd1b-972c-48e7-af85-657e20c68c8f`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:04,369::logUtils::49::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'5457319a-3123-4ec1-9965-af7d9378d72d', u'connection': u'172.16.0.1', u'iqn': u'iqn.2003-01.org.linux-iscsi.storage-01.x8664:sn.1d56416909ce', u'portal': u'1', u'user': u'', u'password': '********', u'port': u'3260'}], options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,370::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage-01.x8664:sn.1d56416909ce -I default -p 172.16.0.1:3260,1 --op=new (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,389::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,389::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m iface -I default (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,403::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::INFO::2017-02-19 13:40:04,404::iscsi::567::Storage.ISCSI::(setRpFilterIfNeeded) iSCSI iface.net_ifacename not provided. Skipping. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,404::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage-01.x8664:sn.1d56416909ce -I default -p 172.16.0.1:3260,1 -l (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,418::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,418::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage-01.x8664:sn.1d56416909ce -I default -p 172.16.0.1:3260,1 -n node.startup -v manual --op=update (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,432::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,432::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/udevadm settle --timeout=5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,445::commands::86::root::(execCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,445::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,445::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,445::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,446::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,446::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,446::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,461::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,461::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (vdsm.storage.hba.rescan) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,462::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,462::hba::58::Storage.HBA::(rescan) Starting scan jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,525::hba::64::Storage.HBA::(rescan) Scan finished jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,526::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,526::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/udevadm settle --timeout=5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,535::commands::86::root::(execCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,537::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,538::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,568::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n'; = 0 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,569::hsm::2367::Storage.HSM::(__prefetchDomains) Found SD uuids: () jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,569::hsm::2427::Storage.HSM::(connectStorageServer) knownSDs: {} jsonrpc.Executor/4::INFO::2017-02-19 13:40:04,569::logUtils::52::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'5457319a-3123-4ec1-9965-af7d9378d72d'}]} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,569::task::1193::Storage.TaskManager.Task::(prepare) Task=`9cb9bd1b-972c-48e7-af85-657e20c68c8f`::finished: {'statuslist': [{'status': 0, 'id': u'5457319a-3123-4ec1-9965-af7d9378d72d'}]} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,570::task::597::Storage.TaskManager.Task::(_updateState) Task=`9cb9bd1b-972c-48e7-af85-657e20c68c8f`::moving from state preparing -> state finished jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,570::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,570::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,570::task::995::Storage.TaskManager.Task::(_decref) Task=`9cb9bd1b-972c-48e7-af85-657e20c68c8f`::ref 0 aborting False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:04,570::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'5457319a-3123-4ec1-9965-af7d9378d72d'}] jsonrpc.Executor/4::INFO::2017-02-19 13:40:04,570::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.connectStorageServer succeeded in 0.20 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,582::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'StorageDomain.getStats' in bridge with {u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,583::task::597::Storage.TaskManager.Task::(_updateState) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:04,583::logUtils::49::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,583::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0ead2c9a-0c13-49e8-b076-87525e240fa6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2737' at 'getStorageDomainStats' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,583::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,583::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0ead2c9a-0c13-49e8-b076-87525e240fa6`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::task::995::Storage.TaskManager.Task::(_decref) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::ref 1 aborting False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,584::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,585::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,585::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,585::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/iscsiadm -m session -R (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,600::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,600::misc::556::Storage.SamplingMethod::(__call__) Trying to enter sampling method (vdsm.storage.hba.rescan) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,601::misc::559::Storage.SamplingMethod::(__call__) Got in to sampling method jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,601::hba::58::Storage.HBA::(rescan) Starting scan jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,663::hba::64::Storage.HBA::(rescan) Scan finished jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,664::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,664::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/udevadm settle --timeout=5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,675::commands::86::root::(execCmd) SUCCESS: = ''; = 0 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,676::misc::566::Storage.SamplingMethod::(__call__) Returning last result jsonrpc.Executor/5::ERROR::2017-02-19 13:40:04,677::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:04,677::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,678::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,702::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:04,703::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-13::DEBUG::2017-02-19 13:40:04,705::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpkMiT9p Thread-13::INFO::2017-02-19 13:40:04,705::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-0 Thread-13::DEBUG::2017-02-19 13:40:04,705::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-0 Thread-13::DEBUG::2017-02-19 13:40:04,712::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-0 Thread-13::DEBUG::2017-02-19 13:40:04,712::__init__::398::IOProcessClient::(_startCommunication) Communication thread for client ioprocess-0 started Thread-14::DEBUG::2017-02-19 13:40:04,713::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/_var_lib_ovirt-hosted-engine-setup_tmpo0Nkzt Thread-14::INFO::2017-02-19 13:40:04,713::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-1 Thread-14::DEBUG::2017-02-19 13:40:04,713::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-1 Thread-14::DEBUG::2017-02-19 13:40:04,718::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-1 ioprocess communication (30442)::INFO::2017-02-19 13:40:04,719::__init__::447::IOProcess::(_processLogs) Starting ioprocess ioprocess communication (30442)::INFO::2017-02-19 13:40:04,719::__init__::447::IOProcess::(_processLogs) Starting ioprocess Thread-14::DEBUG::2017-02-19 13:40:04,720::__init__::398::IOProcessClient::(_startCommunication) Communication thread for client ioprocess-1 started ioprocess communication (30448)::INFO::2017-02-19 13:40:04,721::__init__::447::IOProcess::(_processLogs) Starting ioprocess ioprocess communication (30448)::INFO::2017-02-19 13:40:04,721::__init__::447::IOProcess::(_processLogs) Starting ioprocess Thread-17::DEBUG::2017-02-19 13:40:04,724::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/backup-01.eurooil.ru:_mnt_backup_nfs_iso Thread-17::INFO::2017-02-19 13:40:04,724::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-2 Thread-17::DEBUG::2017-02-19 13:40:04,724::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-2 Thread-17::DEBUG::2017-02-19 13:40:04,729::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-2 Thread-17::DEBUG::2017-02-19 13:40:04,730::__init__::398::IOProcessClient::(_startCommunication) Communication thread for client ioprocess-2 started Thread-18::DEBUG::2017-02-19 13:40:04,730::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/backup-01.eurooil.ru:_mnt_backup_nfs_ovirt-vm-backup Thread-18::INFO::2017-02-19 13:40:04,731::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-3 Thread-18::DEBUG::2017-02-19 13:40:04,731::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-3 Thread-18::DEBUG::2017-02-19 13:40:04,736::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-3 ioprocess communication (30461)::INFO::2017-02-19 13:40:04,736::__init__::447::IOProcess::(_processLogs) Starting ioprocess ioprocess communication (30461)::INFO::2017-02-19 13:40:04,736::__init__::447::IOProcess::(_processLogs) Starting ioprocess Thread-18::DEBUG::2017-02-19 13:40:04,738::__init__::398::IOProcessClient::(_startCommunication) Communication thread for client ioprocess-3 started ioprocess communication (30467)::INFO::2017-02-19 13:40:04,738::__init__::447::IOProcess::(_processLogs) Starting ioprocess ioprocess communication (30467)::INFO::2017-02-19 13:40:04,738::__init__::447::IOProcess::(_processLogs) Starting ioprocess jsonrpc.Executor/5::ERROR::2017-02-19 13:40:04,739::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:04,740::task::868::Storage.TaskManager.Task::(_setError) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2738, in getStorageDomainStats dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,741::task::887::Storage.TaskManager.Task::(_run) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::Task._run: facf3ed0-9e9a-4c25-a923-ad973f9a1990 (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,741::task::1248::Storage.TaskManager.Task::(stop) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,741::task::995::Storage.TaskManager.Task::(_decref) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:04,741::task::1173::Storage.TaskManager.Task::(prepare) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,741::task::1178::Storage.TaskManager.Task::(prepare) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::task::995::Storage.TaskManager.Task::(_decref) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::task::930::Storage.TaskManager.Task::(_doAbort) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::task::597::Storage.TaskManager.Task::(_updateState) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::task::597::Storage.TaskManager.Task::(_updateState) Task=`facf3ed0-9e9a-4c25-a923-ad973f9a1990`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,742::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,743::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,743::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:04,743::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:04,743::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:04,743::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StorageDomain.getStats failed (error 358) in 0.16 seconds JsonRpc (StompReactor)::ERROR::2017-02-19 13:40:04,745::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof Reactor thread::INFO::2017-02-19 13:40:06,750::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52840 Reactor thread::DEBUG::2017-02-19 13:40:06,755::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:06,756::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::1:52840 Reactor thread::INFO::2017-02-19 13:40:06,757::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2017-02-19 13:40:06,757::stompreactor::492::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::1', 52840) JsonRpc (StompReactor)::INFO::2017-02-19 13:40:06,757::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:06,960::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getHardwareInfo' in bridge with {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:06,962::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'X9DBU', 'systemSerialNumber': '0123456789', 'systemFamily': 'To be filled by O.E.M.', 'systemVersion': '0123456789', 'systemUUID': '00000000-0000-0000-0000-0CC47A3C0510', 'systemManufacturer': 'Supermicro'} jsonrpc.Executor/6::INFO::2017-02-19 13:40:06,962::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getHardwareInfo succeeded in 0.00 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,967::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'StorageDomain.getImages' in bridge with {u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,968::task::597::Storage.TaskManager.Task::(_updateState) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:06,968::logUtils::49::dispatcher::(wrapper) Run and protect: getImagesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,968::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`c1920dda-0731-439c-a9ef-1bbfa045a397`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3216' at 'getImagesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,968::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,969::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,969::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`c1920dda-0731-439c-a9ef-1bbfa045a397`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,969::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,969::task::995::Storage.TaskManager.Task::(_decref) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:06,969::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:06,969::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,970::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:06,997::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:06,998::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,003::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,003::task::868::Storage.TaskManager.Task::(_setError) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3217, in getImagesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::887::Storage.TaskManager.Task::(_run) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::Task._run: 8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4 (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::1248::Storage.TaskManager.Task::(stop) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::995::Storage.TaskManager.Task::(_decref) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,004::task::1173::Storage.TaskManager.Task::(prepare) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::1178::Storage.TaskManager.Task::(prepare) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::995::Storage.TaskManager.Task::(_decref) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,004::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::task::597::Storage.TaskManager.Task::(_updateState) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::task::597::Storage.TaskManager.Task::(_updateState) Task=`8dfbf6ce-bf21-492d-8148-cb75d8aeb3d4`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,005::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,006::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,006::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,006::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StorageDomain.getImages failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,033::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,033::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,034::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,034::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6bc8877f-b360-474d-a915-08afd15e4d3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,034::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,034::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,034::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6bc8877f-b360-474d-a915-08afd15e4d3a`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,035::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,035::task::995::Storage.TaskManager.Task::(_decref) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,035::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,035::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,035::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,059::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:07,060::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,065::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,065::task::868::Storage.TaskManager.Task::(_setError) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::887::Storage.TaskManager.Task::(_run) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::Task._run: 8ae844ff-afc1-46df-bd67-886e7d62c723 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::1248::Storage.TaskManager.Task::(stop) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::995::Storage.TaskManager.Task::(_decref) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,066::task::1173::Storage.TaskManager.Task::(prepare) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::1178::Storage.TaskManager.Task::(prepare) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::995::Storage.TaskManager.Task::(_decref) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,066::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ae844ff-afc1-46df-bd67-886e7d62c723`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,067::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,068::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,068::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,068::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,068::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,097::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'2fac8cc8-107e-42aa-b522-acde956558d1'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,098::task::597::Storage.TaskManager.Task::(_updateState) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,098::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'2fac8cc8-107e-42aa-b522-acde956558d1', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,098::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`91a50cfe-14b6-48f1-82ac-407811e606d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,099::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,099::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,099::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`91a50cfe-14b6-48f1-82ac-407811e606d8`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,099::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,099::task::995::Storage.TaskManager.Task::(_decref) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,099::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,100::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,100::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,125::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:07,126::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,130::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,131::task::868::Storage.TaskManager.Task::(_setError) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,131::task::887::Storage.TaskManager.Task::(_run) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::Task._run: cdb17bfe-ffb0-4ee5-920c-f42fb2388af5 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'2fac8cc8-107e-42aa-b522-acde956558d1') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,131::task::1248::Storage.TaskManager.Task::(stop) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,131::task::995::Storage.TaskManager.Task::(_decref) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,131::task::1173::Storage.TaskManager.Task::(prepare) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,131::task::1178::Storage.TaskManager.Task::(prepare) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,131::task::995::Storage.TaskManager.Task::(_decref) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::task::930::Storage.TaskManager.Task::(_doAbort) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::task::597::Storage.TaskManager.Task::(_updateState) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::task::597::Storage.TaskManager.Task::(_updateState) Task=`cdb17bfe-ffb0-4ee5-920c-f42fb2388af5`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,132::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,133::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,133::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,133::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,133::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,161::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'41645a90-fe57-4395-ade7-652e2fa1b684'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,162::task::597::Storage.TaskManager.Task::(_updateState) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,162::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'41645a90-fe57-4395-ade7-652e2fa1b684', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0561b0c2-cf4e-4373-93f9-729dd614c961`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0561b0c2-cf4e-4373-93f9-729dd614c961`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,163::task::995::Storage.TaskManager.Task::(_decref) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,163::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,164::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,164::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,188::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:07,189::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,193::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,194::task::868::Storage.TaskManager.Task::(_setError) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,194::task::887::Storage.TaskManager.Task::(_run) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::Task._run: 8454d09e-2346-4c3d-865e-5a5f88697149 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'41645a90-fe57-4395-ade7-652e2fa1b684') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,194::task::1248::Storage.TaskManager.Task::(stop) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,194::task::995::Storage.TaskManager.Task::(_decref) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,194::task::1173::Storage.TaskManager.Task::(prepare) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,194::task::1178::Storage.TaskManager.Task::(prepare) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::task::995::Storage.TaskManager.Task::(_decref) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::task::597::Storage.TaskManager.Task::(_updateState) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::task::597::Storage.TaskManager.Task::(_updateState) Task=`8454d09e-2346-4c3d-865e-5a5f88697149`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,195::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,196::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,196::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,196::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,196::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,196::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,225::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'9fb8fb9f-2709-435f-94e7-45ecf52bfd7a'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,226::task::597::Storage.TaskManager.Task::(_updateState) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,226::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'9fb8fb9f-2709-435f-94e7-45ecf52bfd7a', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,227::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eeb2d68a-c38b-4202-b0cd-9f8e53c7da95`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,227::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,227::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,227::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eeb2d68a-c38b-4202-b0cd-9f8e53c7da95`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,227::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,228::task::995::Storage.TaskManager.Task::(_decref) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,228::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,228::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,228::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,252::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:07,253::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,258::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,258::task::868::Storage.TaskManager.Task::(_setError) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,258::task::887::Storage.TaskManager.Task::(_run) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::Task._run: 504dfb70-4f33-46ba-ac92-d8567ea6dec7 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'9fb8fb9f-2709-435f-94e7-45ecf52bfd7a') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,258::task::1248::Storage.TaskManager.Task::(stop) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,258::task::995::Storage.TaskManager.Task::(_decref) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,258::task::1173::Storage.TaskManager.Task::(prepare) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::1178::Storage.TaskManager.Task::(prepare) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::995::Storage.TaskManager.Task::(_decref) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::930::Storage.TaskManager.Task::(_doAbort) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::597::Storage.TaskManager.Task::(_updateState) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::task::597::Storage.TaskManager.Task::(_updateState) Task=`504dfb70-4f33-46ba-ac92-d8567ea6dec7`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,259::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,260::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,260::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,260::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,260::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,260::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,260::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,290::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'353571ba-9e52-4bc8-b31b-60459d9e6421'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,290::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,290::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'353571ba-9e52-4bc8-b31b-60459d9e6421', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,291::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9e616104-76e9-4b72-98b9-8aeac23ad942`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,291::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,291::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,291::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9e616104-76e9-4b72-98b9-8aeac23ad942`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,292::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,292::task::995::Storage.TaskManager.Task::(_decref) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,292::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,292::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,292::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,316::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:07,317::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,321::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,322::task::868::Storage.TaskManager.Task::(_setError) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,322::task::887::Storage.TaskManager.Task::(_run) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::Task._run: 4c8e4823-092c-454e-9a52-adc174ca550c (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'353571ba-9e52-4bc8-b31b-60459d9e6421') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,322::task::1248::Storage.TaskManager.Task::(stop) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,322::task::995::Storage.TaskManager.Task::(_decref) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,322::task::1173::Storage.TaskManager.Task::(prepare) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,322::task::1178::Storage.TaskManager.Task::(prepare) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,322::task::995::Storage.TaskManager.Task::(_decref) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::task::930::Storage.TaskManager.Task::(_doAbort) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c8e4823-092c-454e-9a52-adc174ca550c`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,323::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,324::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,324::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,324::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,324::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,324::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,354::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'491a9ab6-1666-4bdd-8bcb-b203864ffe99'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,354::task::597::Storage.TaskManager.Task::(_updateState) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,355::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'491a9ab6-1666-4bdd-8bcb-b203864ffe99', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,355::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`da41affd-dddf-4c83-b838-079797260d07`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,355::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,355::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,355::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`da41affd-dddf-4c83-b838-079797260d07`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,356::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,356::task::995::Storage.TaskManager.Task::(_decref) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,356::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,356::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,356::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,380::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:07,381::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,386::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,386::task::868::Storage.TaskManager.Task::(_setError) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::887::Storage.TaskManager.Task::(_run) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::Task._run: 36da4203-b2c2-4a55-86b3-71470495239e (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'491a9ab6-1666-4bdd-8bcb-b203864ffe99') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::1248::Storage.TaskManager.Task::(stop) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::995::Storage.TaskManager.Task::(_decref) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,387::task::1173::Storage.TaskManager.Task::(prepare) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::1178::Storage.TaskManager.Task::(prepare) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::995::Storage.TaskManager.Task::(_decref) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::930::Storage.TaskManager.Task::(_doAbort) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,387::task::597::Storage.TaskManager.Task::(_updateState) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::task::597::Storage.TaskManager.Task::(_updateState) Task=`36da4203-b2c2-4a55-86b3-71470495239e`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,388::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,389::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,389::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,418::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'1cebd37c-fcf4-435c-a061-7ec60f3435e7'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,418::task::597::Storage.TaskManager.Task::(_updateState) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,419::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'1cebd37c-fcf4-435c-a061-7ec60f3435e7', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,419::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e18aa6bd-ef8b-4f92-bc20-8423fcbebe5c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,419::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,419::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,419::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e18aa6bd-ef8b-4f92-bc20-8423fcbebe5c`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,420::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,420::task::995::Storage.TaskManager.Task::(_decref) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,420::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,420::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,420::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,444::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:07,445::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,450::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,450::task::868::Storage.TaskManager.Task::(_setError) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::887::Storage.TaskManager.Task::(_run) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::Task._run: 65afd9f2-775b-4887-988d-0596db4a20ed (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'1cebd37c-fcf4-435c-a061-7ec60f3435e7') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::1248::Storage.TaskManager.Task::(stop) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::995::Storage.TaskManager.Task::(_decref) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,451::task::1173::Storage.TaskManager.Task::(prepare) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::1178::Storage.TaskManager.Task::(prepare) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::995::Storage.TaskManager.Task::(_decref) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::task::930::Storage.TaskManager.Task::(_doAbort) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,451::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::task::597::Storage.TaskManager.Task::(_updateState) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::task::597::Storage.TaskManager.Task::(_updateState) Task=`65afd9f2-775b-4887-988d-0596db4a20ed`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,452::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,453::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,453::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,453::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,482::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'4fc108de-838f-4acc-9688-c8d45739d19b'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,483::task::597::Storage.TaskManager.Task::(_updateState) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,483::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'4fc108de-838f-4acc-9688-c8d45739d19b', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,483::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`60ac6ce5-a0a4-4542-9f44-5cf65bbf2421`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,483::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,484::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,484::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`60ac6ce5-a0a4-4542-9f44-5cf65bbf2421`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,484::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,484::task::995::Storage.TaskManager.Task::(_decref) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,484::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,484::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,485::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,508::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:07,509::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,514::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,514::task::868::Storage.TaskManager.Task::(_setError) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::887::Storage.TaskManager.Task::(_run) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::Task._run: ea451c30-6360-4ea1-9104-0ac43609897f (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'4fc108de-838f-4acc-9688-c8d45739d19b') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::1248::Storage.TaskManager.Task::(stop) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::995::Storage.TaskManager.Task::(_decref) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,515::task::1173::Storage.TaskManager.Task::(prepare) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::1178::Storage.TaskManager.Task::(prepare) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::995::Storage.TaskManager.Task::(_decref) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,515::task::930::Storage.TaskManager.Task::(_doAbort) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::task::597::Storage.TaskManager.Task::(_updateState) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::task::597::Storage.TaskManager.Task::(_updateState) Task=`ea451c30-6360-4ea1-9104-0ac43609897f`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,516::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,517::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,517::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,517::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,546::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'1487b39c-0d0d-4afe-ac31-bc7c0cdb524c'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,547::task::597::Storage.TaskManager.Task::(_updateState) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,547::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'1487b39c-0d0d-4afe-ac31-bc7c0cdb524c', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,547::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d9047ecd-8a23-47bb-a952-3c6eeeb98a8b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,547::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,548::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,548::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d9047ecd-8a23-47bb-a952-3c6eeeb98a8b`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,548::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,548::task::995::Storage.TaskManager.Task::(_decref) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,548::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,548::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,549::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,572::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:07,573::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,578::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,578::task::868::Storage.TaskManager.Task::(_setError) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::887::Storage.TaskManager.Task::(_run) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::Task._run: f44d9b53-9346-4297-beab-f82a3c1d5e34 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'1487b39c-0d0d-4afe-ac31-bc7c0cdb524c') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::1248::Storage.TaskManager.Task::(stop) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::995::Storage.TaskManager.Task::(_decref) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,579::task::1173::Storage.TaskManager.Task::(prepare) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::1178::Storage.TaskManager.Task::(prepare) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::995::Storage.TaskManager.Task::(_decref) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::930::Storage.TaskManager.Task::(_doAbort) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,579::task::597::Storage.TaskManager.Task::(_updateState) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::task::597::Storage.TaskManager.Task::(_updateState) Task=`f44d9b53-9346-4297-beab-f82a3c1d5e34`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:07,580::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:07,581::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:07,581::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,610::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'8c0a0e39-075f-4a35-9e43-75d7c3805783'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,611::task::597::Storage.TaskManager.Task::(_updateState) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,611::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'8c0a0e39-075f-4a35-9e43-75d7c3805783', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,611::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`31e02278-6939-46fe-a17d-cabec4ab87e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,612::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,612::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,612::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`31e02278-6939-46fe-a17d-cabec4ab87e7`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,612::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,612::task::995::Storage.TaskManager.Task::(_decref) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,612::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,613::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,613::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,636::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:07,637::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,642::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,642::task::868::Storage.TaskManager.Task::(_setError) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::887::Storage.TaskManager.Task::(_run) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::Task._run: 73149b6a-ddbb-4daf-82cd-849581ebf754 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'8c0a0e39-075f-4a35-9e43-75d7c3805783') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::1248::Storage.TaskManager.Task::(stop) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::995::Storage.TaskManager.Task::(_decref) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,643::task::1173::Storage.TaskManager.Task::(prepare) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::1178::Storage.TaskManager.Task::(prepare) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::995::Storage.TaskManager.Task::(_decref) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,643::task::930::Storage.TaskManager.Task::(_doAbort) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::task::597::Storage.TaskManager.Task::(_updateState) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::task::597::Storage.TaskManager.Task::(_updateState) Task=`73149b6a-ddbb-4daf-82cd-849581ebf754`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,644::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,645::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:07,645::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:07,645::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:07,645::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,674::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'0221fd74-69db-4c72-9a88-6f210dfa3aa9'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,675::task::597::Storage.TaskManager.Task::(_updateState) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,675::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'0221fd74-69db-4c72-9a88-6f210dfa3aa9', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3619e77a-586b-4f66-b235-d83d3d262a64`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3619e77a-586b-4f66-b235-d83d3d262a64`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,676::task::995::Storage.TaskManager.Task::(_decref) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,677::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,677::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,677::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,701::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:07,702::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,706::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,707::task::868::Storage.TaskManager.Task::(_setError) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,707::task::887::Storage.TaskManager.Task::(_run) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::Task._run: 2085163a-52bf-49c2-9f83-238545a1dbdc (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'0221fd74-69db-4c72-9a88-6f210dfa3aa9') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,707::task::1248::Storage.TaskManager.Task::(stop) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,707::task::995::Storage.TaskManager.Task::(_decref) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,707::task::1173::Storage.TaskManager.Task::(prepare) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,707::task::1178::Storage.TaskManager.Task::(prepare) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::task::995::Storage.TaskManager.Task::(_decref) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::task::930::Storage.TaskManager.Task::(_doAbort) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::task::597::Storage.TaskManager.Task::(_updateState) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::task::597::Storage.TaskManager.Task::(_updateState) Task=`2085163a-52bf-49c2-9f83-238545a1dbdc`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,708::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,709::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,709::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:07,709::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:07,709::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:07,709::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,739::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'61cea3fe-4dd7-45b7-b0a3-ed5a500801e0'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,739::task::597::Storage.TaskManager.Task::(_updateState) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,739::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'61cea3fe-4dd7-45b7-b0a3-ed5a500801e0', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,740::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`345b230a-de4d-4119-8383-afa8eafd43e8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,740::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,740::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,740::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`345b230a-de4d-4119-8383-afa8eafd43e8`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,740::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,741::task::995::Storage.TaskManager.Task::(_decref) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,741::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,741::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,741::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,765::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:07,766::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,771::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,771::task::868::Storage.TaskManager.Task::(_setError) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,771::task::887::Storage.TaskManager.Task::(_run) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::Task._run: 295e7bff-0ee2-4672-95e0-142fb142ba19 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'61cea3fe-4dd7-45b7-b0a3-ed5a500801e0') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,771::task::1248::Storage.TaskManager.Task::(stop) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,771::task::995::Storage.TaskManager.Task::(_decref) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,772::task::1173::Storage.TaskManager.Task::(prepare) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::1178::Storage.TaskManager.Task::(prepare) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::995::Storage.TaskManager.Task::(_decref) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::930::Storage.TaskManager.Task::(_doAbort) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::597::Storage.TaskManager.Task::(_updateState) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::task::597::Storage.TaskManager.Task::(_updateState) Task=`295e7bff-0ee2-4672-95e0-142fb142ba19`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,772::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,773::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,773::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,773::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:07,773::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:07,773::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:07,773::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,803::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'3cdd1519-9956-4177-ad7f-2a262de50932'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,803::task::597::Storage.TaskManager.Task::(_updateState) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,804::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'3cdd1519-9956-4177-ad7f-2a262de50932', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,804::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1ca854b1-01dc-4077-b2cf-62a369a75c8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,804::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,804::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,804::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1ca854b1-01dc-4077-b2cf-62a369a75c8f`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,805::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,805::task::995::Storage.TaskManager.Task::(_decref) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,805::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,805::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,805::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,829::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:07,830::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,834::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,835::task::868::Storage.TaskManager.Task::(_setError) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,835::task::887::Storage.TaskManager.Task::(_run) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::Task._run: c852cd7c-1f8c-46bc-802a-a68da28a4e4b (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'3cdd1519-9956-4177-ad7f-2a262de50932') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,835::task::1248::Storage.TaskManager.Task::(stop) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,835::task::995::Storage.TaskManager.Task::(_decref) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,835::task::1173::Storage.TaskManager.Task::(prepare) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::1178::Storage.TaskManager.Task::(prepare) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::995::Storage.TaskManager.Task::(_decref) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::597::Storage.TaskManager.Task::(_updateState) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::task::597::Storage.TaskManager.Task::(_updateState) Task=`c852cd7c-1f8c-46bc-802a-a68da28a4e4b`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,836::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,837::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,837::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,837::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:07,837::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:07,837::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:07,837::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,867::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'41c693e7-164b-438d-ab8b-435a74103f83'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,868::task::597::Storage.TaskManager.Task::(_updateState) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,868::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'41c693e7-164b-438d-ab8b-435a74103f83', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,868::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6584f891-7873-4bf4-aa0c-44419af95712`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,868::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,868::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,869::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6584f891-7873-4bf4-aa0c-44419af95712`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,869::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,869::task::995::Storage.TaskManager.Task::(_decref) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,869::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,869::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,870::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,894::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:07,895::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,899::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,900::task::868::Storage.TaskManager.Task::(_setError) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,900::task::887::Storage.TaskManager.Task::(_run) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::Task._run: 0b0dde24-dc70-417e-89b3-ff2ef79630ef (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'41c693e7-164b-438d-ab8b-435a74103f83') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,900::task::1248::Storage.TaskManager.Task::(stop) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,900::task::995::Storage.TaskManager.Task::(_decref) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,900::task::1173::Storage.TaskManager.Task::(prepare) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,900::task::1178::Storage.TaskManager.Task::(prepare) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,900::task::995::Storage.TaskManager.Task::(_decref) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::task::930::Storage.TaskManager.Task::(_doAbort) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::task::597::Storage.TaskManager.Task::(_updateState) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::task::597::Storage.TaskManager.Task::(_updateState) Task=`0b0dde24-dc70-417e-89b3-ff2ef79630ef`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,901::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,902::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,902::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:07,902::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:07,902::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:07,902::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,931::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'e5f92371-8ba2-4199-a363-e9e85938dc14'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,932::task::597::Storage.TaskManager.Task::(_updateState) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,932::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e5f92371-8ba2-4199-a363-e9e85938dc14', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,932::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`4b7f720c-8b6c-4f57-b045-52fe6b4b2aec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,932::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,933::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,933::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`4b7f720c-8b6c-4f57-b045-52fe6b4b2aec`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,933::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,933::task::995::Storage.TaskManager.Task::(_decref) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,933::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,933::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,934::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,957::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:07,958::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,963::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,963::task::868::Storage.TaskManager.Task::(_setError) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::887::Storage.TaskManager.Task::(_run) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::Task._run: 57732d9b-7125-4adb-988a-d915aafe36c8 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'e5f92371-8ba2-4199-a363-e9e85938dc14') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::1248::Storage.TaskManager.Task::(stop) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::995::Storage.TaskManager.Task::(_decref) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,964::task::1173::Storage.TaskManager.Task::(prepare) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::1178::Storage.TaskManager.Task::(prepare) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::995::Storage.TaskManager.Task::(_decref) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,964::task::930::Storage.TaskManager.Task::(_doAbort) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::task::597::Storage.TaskManager.Task::(_updateState) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::task::597::Storage.TaskManager.Task::(_updateState) Task=`57732d9b-7125-4adb-988a-d915aafe36c8`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,965::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,966::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:07,966::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:07,966::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:07,966::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,995::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'bfe662dd-7f46-4a0e-a0d0-ec68b895b6ab'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,996::task::597::Storage.TaskManager.Task::(_updateState) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:07,996::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'bfe662dd-7f46-4a0e-a0d0-ec68b895b6ab', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,996::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b6bc61fc-3c10-4543-8f46-f492734f6755`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,997::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,997::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,997::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b6bc61fc-3c10-4543-8f46-f492734f6755`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,997::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,997::task::995::Storage.TaskManager.Task::(_decref) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,997::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:07,998::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:07,998::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,021::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:08,022::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,027::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,028::task::868::Storage.TaskManager.Task::(_setError) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,028::task::887::Storage.TaskManager.Task::(_run) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::Task._run: 1c7e62e7-e175-4620-8cad-128cb7646fac (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'bfe662dd-7f46-4a0e-a0d0-ec68b895b6ab') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,028::task::1248::Storage.TaskManager.Task::(stop) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,028::task::995::Storage.TaskManager.Task::(_decref) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:08,028::task::1173::Storage.TaskManager.Task::(prepare) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,028::task::1178::Storage.TaskManager.Task::(prepare) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,028::task::995::Storage.TaskManager.Task::(_decref) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::task::930::Storage.TaskManager.Task::(_doAbort) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::task::597::Storage.TaskManager.Task::(_updateState) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::task::597::Storage.TaskManager.Task::(_updateState) Task=`1c7e62e7-e175-4620-8cad-128cb7646fac`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,029::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,030::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,030::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,030::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:08,030::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,059::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'c58f2c71-5055-4644-bc3f-e4d708b9bc3c'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,060::task::597::Storage.TaskManager.Task::(_updateState) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,060::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'c58f2c71-5055-4644-bc3f-e4d708b9bc3c', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf0ff696-afba-4ce6-9d90-d208d2f1c496`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf0ff696-afba-4ce6-9d90-d208d2f1c496`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,061::task::995::Storage.TaskManager.Task::(_decref) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,062::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,062::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,062::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,085::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:08,086::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,091::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,091::task::868::Storage.TaskManager.Task::(_setError) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::887::Storage.TaskManager.Task::(_run) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::Task._run: 7bebb54b-205c-453d-8a14-7937f20be97b (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'c58f2c71-5055-4644-bc3f-e4d708b9bc3c') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::1248::Storage.TaskManager.Task::(stop) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::995::Storage.TaskManager.Task::(_decref) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,092::task::1173::Storage.TaskManager.Task::(prepare) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::1178::Storage.TaskManager.Task::(prepare) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::995::Storage.TaskManager.Task::(_decref) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::task::930::Storage.TaskManager.Task::(_doAbort) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,092::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::task::597::Storage.TaskManager.Task::(_updateState) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::task::597::Storage.TaskManager.Task::(_updateState) Task=`7bebb54b-205c-453d-8a14-7937f20be97b`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,093::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,094::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,094::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,094::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,124::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'de0a9fd7-3fcb-4747-b698-fa13e5073928'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,124::task::597::Storage.TaskManager.Task::(_updateState) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,124::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'de0a9fd7-3fcb-4747-b698-fa13e5073928', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,125::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`75b7c200-506d-4726-83d3-3dd18fca8c59`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,125::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,125::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,125::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`75b7c200-506d-4726-83d3-3dd18fca8c59`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,126::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,126::task::995::Storage.TaskManager.Task::(_decref) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,126::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,126::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,126::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,150::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:08,151::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,156::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,156::task::868::Storage.TaskManager.Task::(_setError) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::887::Storage.TaskManager.Task::(_run) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::Task._run: 0ea88ccf-3da3-4d25-98f8-9d84430b2813 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'de0a9fd7-3fcb-4747-b698-fa13e5073928') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::1248::Storage.TaskManager.Task::(stop) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::995::Storage.TaskManager.Task::(_decref) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,157::task::1173::Storage.TaskManager.Task::(prepare) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::1178::Storage.TaskManager.Task::(prepare) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::995::Storage.TaskManager.Task::(_decref) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::task::930::Storage.TaskManager.Task::(_doAbort) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,157::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::task::597::Storage.TaskManager.Task::(_updateState) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::task::597::Storage.TaskManager.Task::(_updateState) Task=`0ea88ccf-3da3-4d25-98f8-9d84430b2813`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,158::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,159::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,159::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,159::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,188::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'88d34b61-1023-462b-86ea-92809153737f'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,188::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,189::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'88d34b61-1023-462b-86ea-92809153737f', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,189::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f4484f2c-514a-401e-a18b-8cc74ced51dd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,189::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,189::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,189::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f4484f2c-514a-401e-a18b-8cc74ced51dd`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,190::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,190::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,190::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,190::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,190::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,214::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:08,215::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,220::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,220::task::868::Storage.TaskManager.Task::(_setError) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,220::task::887::Storage.TaskManager.Task::(_run) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::Task._run: 8ad0ba47-fee0-49ca-a91b-955205012259 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'88d34b61-1023-462b-86ea-92809153737f') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,220::task::1248::Storage.TaskManager.Task::(stop) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,221::task::1173::Storage.TaskManager.Task::(prepare) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::1178::Storage.TaskManager.Task::(prepare) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,221::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad0ba47-fee0-49ca-a91b-955205012259`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,222::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,222::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,223::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,252::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd16ba498-f044-498d-b7ee-8252d88da5a6'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,253::task::597::Storage.TaskManager.Task::(_updateState) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,253::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd16ba498-f044-498d-b7ee-8252d88da5a6', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,253::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d76918f2-45b6-4a40-be30-e623c180ad4a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,253::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,253::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,253::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d76918f2-45b6-4a40-be30-e623c180ad4a`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,254::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,254::task::995::Storage.TaskManager.Task::(_decref) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,254::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,254::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,254::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,278::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:08,279::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,284::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,284::task::868::Storage.TaskManager.Task::(_setError) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::887::Storage.TaskManager.Task::(_run) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::Task._run: 5ce72a79-e628-4c55-b1ec-dd1a86482260 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd16ba498-f044-498d-b7ee-8252d88da5a6') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::1248::Storage.TaskManager.Task::(stop) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::995::Storage.TaskManager.Task::(_decref) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,285::task::1173::Storage.TaskManager.Task::(prepare) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::1178::Storage.TaskManager.Task::(prepare) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::995::Storage.TaskManager.Task::(_decref) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::task::930::Storage.TaskManager.Task::(_doAbort) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,285::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::task::597::Storage.TaskManager.Task::(_updateState) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::task::597::Storage.TaskManager.Task::(_updateState) Task=`5ce72a79-e628-4c55-b1ec-dd1a86482260`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,286::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,287::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,287::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,287::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,316::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'61676fd3-fd45-4818-bffc-e6c95ebcf747'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,317::task::597::Storage.TaskManager.Task::(_updateState) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,317::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'61676fd3-fd45-4818-bffc-e6c95ebcf747', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,317::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eafb5103-e73a-4040-8922-6312af525b3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,317::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,318::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,318::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eafb5103-e73a-4040-8922-6312af525b3b`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,318::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,318::task::995::Storage.TaskManager.Task::(_decref) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,318::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,318::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,319::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,342::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:08,343::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,348::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,348::task::868::Storage.TaskManager.Task::(_setError) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::887::Storage.TaskManager.Task::(_run) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::Task._run: a0138104-5fa9-42bb-baa0-dec2766c2026 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'61676fd3-fd45-4818-bffc-e6c95ebcf747') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::1248::Storage.TaskManager.Task::(stop) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::995::Storage.TaskManager.Task::(_decref) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,349::task::1173::Storage.TaskManager.Task::(prepare) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::1178::Storage.TaskManager.Task::(prepare) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::995::Storage.TaskManager.Task::(_decref) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::task::930::Storage.TaskManager.Task::(_doAbort) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,349::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::task::597::Storage.TaskManager.Task::(_updateState) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::task::597::Storage.TaskManager.Task::(_updateState) Task=`a0138104-5fa9-42bb-baa0-dec2766c2026`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,350::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,351::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,351::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,351::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,380::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'e90f6df7-d8f9-49ea-9f7c-4fd0a7263e15'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,381::task::597::Storage.TaskManager.Task::(_updateState) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,381::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e90f6df7-d8f9-49ea-9f7c-4fd0a7263e15', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,381::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d9bd3660-77ca-4f61-b0d2-5bb233f6959e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,382::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,382::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,382::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d9bd3660-77ca-4f61-b0d2-5bb233f6959e`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,382::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,382::task::995::Storage.TaskManager.Task::(_decref) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,382::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,383::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,383::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,406::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:08,407::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,412::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,412::task::868::Storage.TaskManager.Task::(_setError) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::887::Storage.TaskManager.Task::(_run) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::Task._run: e7ee6779-dbec-42db-98e7-0ce853edfd86 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'e90f6df7-d8f9-49ea-9f7c-4fd0a7263e15') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::1248::Storage.TaskManager.Task::(stop) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::995::Storage.TaskManager.Task::(_decref) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,413::task::1173::Storage.TaskManager.Task::(prepare) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::1178::Storage.TaskManager.Task::(prepare) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::995::Storage.TaskManager.Task::(_decref) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,413::task::930::Storage.TaskManager.Task::(_doAbort) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::task::597::Storage.TaskManager.Task::(_updateState) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::task::597::Storage.TaskManager.Task::(_updateState) Task=`e7ee6779-dbec-42db-98e7-0ce853edfd86`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,414::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,415::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,415::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,415::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,415::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,444::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd3f1ee18-c68a-4e1c-946c-eb09b6ef616c'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,445::task::597::Storage.TaskManager.Task::(_updateState) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,445::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd3f1ee18-c68a-4e1c-946c-eb09b6ef616c', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0777df87-0201-4a94-8e9d-fc1d747ec867`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0777df87-0201-4a94-8e9d-fc1d747ec867`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,446::task::995::Storage.TaskManager.Task::(_decref) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,447::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,447::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,447::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,471::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:08,472::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,477::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,477::task::868::Storage.TaskManager.Task::(_setError) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::887::Storage.TaskManager.Task::(_run) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::Task._run: 674ec7dd-a32e-424e-8cfc-88a37f22a598 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd3f1ee18-c68a-4e1c-946c-eb09b6ef616c') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::1248::Storage.TaskManager.Task::(stop) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::995::Storage.TaskManager.Task::(_decref) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,478::task::1173::Storage.TaskManager.Task::(prepare) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::1178::Storage.TaskManager.Task::(prepare) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::995::Storage.TaskManager.Task::(_decref) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::930::Storage.TaskManager.Task::(_doAbort) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,478::task::597::Storage.TaskManager.Task::(_updateState) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::task::597::Storage.TaskManager.Task::(_updateState) Task=`674ec7dd-a32e-424e-8cfc-88a37f22a598`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,479::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,480::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,480::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,509::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'211b99e4-27b5-4f5a-920d-e0f8cae4bd37'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,509::task::597::Storage.TaskManager.Task::(_updateState) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:08,509::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'211b99e4-27b5-4f5a-920d-e0f8cae4bd37', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,510::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d212cc4a-0439-4f27-8e6d-df653dc4544a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,510::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,510::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,510::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d212cc4a-0439-4f27-8e6d-df653dc4544a`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,511::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,511::task::995::Storage.TaskManager.Task::(_decref) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,511::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,511::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,511::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,535::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:08,536::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,541::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,541::task::868::Storage.TaskManager.Task::(_setError) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::887::Storage.TaskManager.Task::(_run) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::Task._run: b0685bd6-d482-4da5-9585-a380834b1cac (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'211b99e4-27b5-4f5a-920d-e0f8cae4bd37') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::1248::Storage.TaskManager.Task::(stop) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::995::Storage.TaskManager.Task::(_decref) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:08,542::task::1173::Storage.TaskManager.Task::(prepare) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::1178::Storage.TaskManager.Task::(prepare) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::995::Storage.TaskManager.Task::(_decref) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,542::task::930::Storage.TaskManager.Task::(_doAbort) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::task::597::Storage.TaskManager.Task::(_updateState) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::task::597::Storage.TaskManager.Task::(_updateState) Task=`b0685bd6-d482-4da5-9585-a380834b1cac`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,543::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,544::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:08,544::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:08,544::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:08,544::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,573::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'fbc49765-6dee-4af7-a6cc-f1f198a22e70'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,573::task::597::Storage.TaskManager.Task::(_updateState) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,574::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'fbc49765-6dee-4af7-a6cc-f1f198a22e70', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,574::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b38ec4c2-35f0-4e93-8dfb-75362af97d8d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,574::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,574::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,574::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b38ec4c2-35f0-4e93-8dfb-75362af97d8d`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,575::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,575::task::995::Storage.TaskManager.Task::(_decref) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,575::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,575::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,575::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,599::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:08,600::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,604::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,605::task::868::Storage.TaskManager.Task::(_setError) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,605::task::887::Storage.TaskManager.Task::(_run) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::Task._run: 30b76d5a-b659-4b8b-87c9-9abacfcda587 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'fbc49765-6dee-4af7-a6cc-f1f198a22e70') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,605::task::1248::Storage.TaskManager.Task::(stop) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,605::task::995::Storage.TaskManager.Task::(_decref) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,605::task::1173::Storage.TaskManager.Task::(prepare) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,605::task::1178::Storage.TaskManager.Task::(prepare) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::task::995::Storage.TaskManager.Task::(_decref) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::task::930::Storage.TaskManager.Task::(_doAbort) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::task::597::Storage.TaskManager.Task::(_updateState) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::task::597::Storage.TaskManager.Task::(_updateState) Task=`30b76d5a-b659-4b8b-87c9-9abacfcda587`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,606::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,607::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,607::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:08,607::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:08,607::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:08,607::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,637::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'110dd2ee-b56f-438f-b262-40b75f46f794'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,638::task::597::Storage.TaskManager.Task::(_updateState) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,638::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'110dd2ee-b56f-438f-b262-40b75f46f794', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,638::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`edabf411-07b0-4c63-8eeb-b6b074c9f5e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,638::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,638::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,639::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`edabf411-07b0-4c63-8eeb-b6b074c9f5e9`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,639::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,639::task::995::Storage.TaskManager.Task::(_decref) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,639::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,639::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,639::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,663::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:08,664::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,669::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,669::task::868::Storage.TaskManager.Task::(_setError) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::887::Storage.TaskManager.Task::(_run) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::Task._run: 99cb2aa6-59ec-42f7-a71b-00e03c61beeb (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'110dd2ee-b56f-438f-b262-40b75f46f794') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::1248::Storage.TaskManager.Task::(stop) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::995::Storage.TaskManager.Task::(_decref) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,670::task::1173::Storage.TaskManager.Task::(prepare) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::1178::Storage.TaskManager.Task::(prepare) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::995::Storage.TaskManager.Task::(_decref) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::task::930::Storage.TaskManager.Task::(_doAbort) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,670::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::task::597::Storage.TaskManager.Task::(_updateState) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::task::597::Storage.TaskManager.Task::(_updateState) Task=`99cb2aa6-59ec-42f7-a71b-00e03c61beeb`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,671::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:08,672::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:08,672::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:08,672::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,701::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'66644fb8-230b-46ba-953c-16dd611f9b65'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,702::task::597::Storage.TaskManager.Task::(_updateState) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,702::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'66644fb8-230b-46ba-953c-16dd611f9b65', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,702::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`628958be-6862-4c6a-b35e-f757b6f2e293`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,702::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,703::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,703::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`628958be-6862-4c6a-b35e-f757b6f2e293`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,703::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,703::task::995::Storage.TaskManager.Task::(_decref) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,703::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,703::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,704::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,727::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:08,728::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,733::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,733::task::868::Storage.TaskManager.Task::(_setError) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::887::Storage.TaskManager.Task::(_run) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::Task._run: b4d0309d-7b11-4327-a321-c8501cd0fba8 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'66644fb8-230b-46ba-953c-16dd611f9b65') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::1248::Storage.TaskManager.Task::(stop) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::995::Storage.TaskManager.Task::(_decref) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,734::task::1173::Storage.TaskManager.Task::(prepare) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::1178::Storage.TaskManager.Task::(prepare) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::995::Storage.TaskManager.Task::(_decref) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,734::task::930::Storage.TaskManager.Task::(_doAbort) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::task::597::Storage.TaskManager.Task::(_updateState) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::task::597::Storage.TaskManager.Task::(_updateState) Task=`b4d0309d-7b11-4327-a321-c8501cd0fba8`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,735::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:08,736::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:08,736::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:08,736::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,765::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'dae606ab-cf39-4546-a06e-a81f08b27ac7'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,766::task::597::Storage.TaskManager.Task::(_updateState) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,766::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'dae606ab-cf39-4546-a06e-a81f08b27ac7', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,766::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`a6ec311b-cbd4-4a99-9ae7-c4fc7b80118c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,766::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,767::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,767::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`a6ec311b-cbd4-4a99-9ae7-c4fc7b80118c`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,767::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,767::task::995::Storage.TaskManager.Task::(_decref) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,767::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,767::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,768::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,792::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:08,793::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,797::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,797::task::868::Storage.TaskManager.Task::(_setError) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::887::Storage.TaskManager.Task::(_run) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::Task._run: 69e344c3-81dd-4ee5-affa-c8ec6b1bb755 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'dae606ab-cf39-4546-a06e-a81f08b27ac7') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::1248::Storage.TaskManager.Task::(stop) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::995::Storage.TaskManager.Task::(_decref) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,798::task::1173::Storage.TaskManager.Task::(prepare) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::1178::Storage.TaskManager.Task::(prepare) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::995::Storage.TaskManager.Task::(_decref) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,798::task::930::Storage.TaskManager.Task::(_doAbort) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::task::597::Storage.TaskManager.Task::(_updateState) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::task::597::Storage.TaskManager.Task::(_updateState) Task=`69e344c3-81dd-4ee5-affa-c8ec6b1bb755`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,799::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,800::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:08,800::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:08,800::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:08,800::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,829::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'4bc19695-ff84-48a3-8dde-68c4b6dc35ce'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,830::task::597::Storage.TaskManager.Task::(_updateState) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,830::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'4bc19695-ff84-48a3-8dde-68c4b6dc35ce', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,830::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e0b64093-e3cf-41af-a977-a2b6e4feca4f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,831::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,831::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,831::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e0b64093-e3cf-41af-a977-a2b6e4feca4f`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,831::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,831::task::995::Storage.TaskManager.Task::(_decref) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,831::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,832::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,832::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,856::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:08,857::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,862::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,862::task::868::Storage.TaskManager.Task::(_setError) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,862::task::887::Storage.TaskManager.Task::(_run) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::Task._run: d72f8278-c9ae-4a98-822f-c2658ac407dd (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'4bc19695-ff84-48a3-8dde-68c4b6dc35ce') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,862::task::1248::Storage.TaskManager.Task::(stop) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::995::Storage.TaskManager.Task::(_decref) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,863::task::1173::Storage.TaskManager.Task::(prepare) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::1178::Storage.TaskManager.Task::(prepare) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::995::Storage.TaskManager.Task::(_decref) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::930::Storage.TaskManager.Task::(_doAbort) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::597::Storage.TaskManager.Task::(_updateState) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,863::task::597::Storage.TaskManager.Task::(_updateState) Task=`d72f8278-c9ae-4a98-822f-c2658ac407dd`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:08,864::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:08,864::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:08,865::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,893::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'ef89222c-bfa7-44d0-80ca-bfab87058e15'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,894::task::597::Storage.TaskManager.Task::(_updateState) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,894::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'ef89222c-bfa7-44d0-80ca-bfab87058e15', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`66949b2e-9900-4a7b-86f9-15331924483a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`66949b2e-9900-4a7b-86f9-15331924483a`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,895::task::995::Storage.TaskManager.Task::(_decref) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,896::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,896::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,896::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,920::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:08,921::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,925::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,926::task::868::Storage.TaskManager.Task::(_setError) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,926::task::887::Storage.TaskManager.Task::(_run) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::Task._run: 85c22fad-b7e0-4cfc-9b2f-f91289e5c302 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'ef89222c-bfa7-44d0-80ca-bfab87058e15') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,926::task::1248::Storage.TaskManager.Task::(stop) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,926::task::995::Storage.TaskManager.Task::(_decref) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,926::task::1173::Storage.TaskManager.Task::(prepare) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::1178::Storage.TaskManager.Task::(prepare) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::995::Storage.TaskManager.Task::(_decref) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::930::Storage.TaskManager.Task::(_doAbort) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::597::Storage.TaskManager.Task::(_updateState) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::task::597::Storage.TaskManager.Task::(_updateState) Task=`85c22fad-b7e0-4cfc-9b2f-f91289e5c302`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,927::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,928::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,928::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,928::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:08,928::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:08,928::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:08,928::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,957::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'cf43ddef-d222-49e1-a308-3d018b879fd2'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,958::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,958::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'cf43ddef-d222-49e1-a308-3d018b879fd2', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,959::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9bc93424-8a24-49a9-a93a-662517ba9d2c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,959::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,959::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,959::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9bc93424-8a24-49a9-a93a-662517ba9d2c`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,959::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,960::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,960::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,960::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,960::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,984::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:08,985::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,990::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,990::task::868::Storage.TaskManager.Task::(_setError) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,990::task::887::Storage.TaskManager.Task::(_run) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::Task._run: 8ad07bff-141f-40f8-a3f3-504ffadcec53 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'cf43ddef-d222-49e1-a308-3d018b879fd2') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,990::task::1248::Storage.TaskManager.Task::(stop) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,990::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,991::task::1173::Storage.TaskManager.Task::(prepare) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::1178::Storage.TaskManager.Task::(prepare) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::995::Storage.TaskManager.Task::(_decref) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::task::597::Storage.TaskManager.Task::(_updateState) Task=`8ad07bff-141f-40f8-a3f3-504ffadcec53`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,991::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,992::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,992::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,992::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,992::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:08,992::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:08,992::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:08,993::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,022::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'ab07dce5-9dbf-41b2-b1a0-f2ba95aac6aa'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,022::task::597::Storage.TaskManager.Task::(_updateState) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,023::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'ab07dce5-9dbf-41b2-b1a0-f2ba95aac6aa', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,023::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`644edc26-cdcc-427b-a672-c924565e58cd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,023::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,023::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,023::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`644edc26-cdcc-427b-a672-c924565e58cd`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,024::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,024::task::995::Storage.TaskManager.Task::(_decref) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,024::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,024::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,024::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,048::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:09,049::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,054::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,055::task::868::Storage.TaskManager.Task::(_setError) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,055::task::887::Storage.TaskManager.Task::(_run) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::Task._run: 961efa81-713e-4c87-87a3-7d8b10398568 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'ab07dce5-9dbf-41b2-b1a0-f2ba95aac6aa') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,055::task::1248::Storage.TaskManager.Task::(stop) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,055::task::995::Storage.TaskManager.Task::(_decref) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,055::task::1173::Storage.TaskManager.Task::(prepare) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,055::task::1178::Storage.TaskManager.Task::(prepare) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,055::task::995::Storage.TaskManager.Task::(_decref) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::task::930::Storage.TaskManager.Task::(_doAbort) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::task::597::Storage.TaskManager.Task::(_updateState) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::task::597::Storage.TaskManager.Task::(_updateState) Task=`961efa81-713e-4c87-87a3-7d8b10398568`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,056::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,057::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,057::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,057::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,057::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,057::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,086::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'961efb32-5817-4ddc-9d2d-261cb3c2c2dc'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,086::task::597::Storage.TaskManager.Task::(_updateState) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,087::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'961efb32-5817-4ddc-9d2d-261cb3c2c2dc', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,087::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`7977a233-ceb2-4de2-876b-984ae035f389`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,087::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,087::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,088::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`7977a233-ceb2-4de2-876b-984ae035f389`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,088::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,088::task::995::Storage.TaskManager.Task::(_decref) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,088::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,088::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,088::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,112::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:09,113::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,118::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,118::task::868::Storage.TaskManager.Task::(_setError) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,118::task::887::Storage.TaskManager.Task::(_run) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::Task._run: 9dac1a29-4f8a-4523-9813-55985cf7e0e0 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'961efb32-5817-4ddc-9d2d-261cb3c2c2dc') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,118::task::1248::Storage.TaskManager.Task::(stop) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::995::Storage.TaskManager.Task::(_decref) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,119::task::1173::Storage.TaskManager.Task::(prepare) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::1178::Storage.TaskManager.Task::(prepare) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::995::Storage.TaskManager.Task::(_decref) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::930::Storage.TaskManager.Task::(_doAbort) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::597::Storage.TaskManager.Task::(_updateState) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,119::task::597::Storage.TaskManager.Task::(_updateState) Task=`9dac1a29-4f8a-4523-9813-55985cf7e0e0`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,120::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,120::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,121::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,150::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'e051f215-c98a-4621-aa71-a1d7bc5f7b7d'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,151::task::597::Storage.TaskManager.Task::(_updateState) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,151::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'e051f215-c98a-4621-aa71-a1d7bc5f7b7d', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,151::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`054ad653-b576-4a53-bb23-2bb1944711d5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,151::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,151::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,152::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`054ad653-b576-4a53-bb23-2bb1944711d5`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,152::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,152::task::995::Storage.TaskManager.Task::(_decref) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,152::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,152::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,153::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,176::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:09,177::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,182::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,182::task::868::Storage.TaskManager.Task::(_setError) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::887::Storage.TaskManager.Task::(_run) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::Task._run: ef6306bb-ef3a-4a4c-8c11-a6ab10bded23 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'e051f215-c98a-4621-aa71-a1d7bc5f7b7d') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::1248::Storage.TaskManager.Task::(stop) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::995::Storage.TaskManager.Task::(_decref) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,183::task::1173::Storage.TaskManager.Task::(prepare) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::1178::Storage.TaskManager.Task::(prepare) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::995::Storage.TaskManager.Task::(_decref) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::task::930::Storage.TaskManager.Task::(_doAbort) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,183::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::task::597::Storage.TaskManager.Task::(_updateState) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::task::597::Storage.TaskManager.Task::(_updateState) Task=`ef6306bb-ef3a-4a4c-8c11-a6ab10bded23`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,184::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,185::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,185::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,185::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,214::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'521aa4dc-25d4-4d97-af9b-7424edc2b939'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,215::task::597::Storage.TaskManager.Task::(_updateState) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,215::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'521aa4dc-25d4-4d97-af9b-7424edc2b939', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,215::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`49967646-7efe-4674-8908-a51fc9acb2ba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,215::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,216::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,216::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`49967646-7efe-4674-8908-a51fc9acb2ba`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,216::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,216::task::995::Storage.TaskManager.Task::(_decref) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,216::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,216::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,217::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,241::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:09,242::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,246::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,247::task::868::Storage.TaskManager.Task::(_setError) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,247::task::887::Storage.TaskManager.Task::(_run) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::Task._run: 642e5dcf-fba3-4d49-afff-350a60f3083a (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'521aa4dc-25d4-4d97-af9b-7424edc2b939') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,247::task::1248::Storage.TaskManager.Task::(stop) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,247::task::995::Storage.TaskManager.Task::(_decref) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,247::task::1173::Storage.TaskManager.Task::(prepare) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,247::task::1178::Storage.TaskManager.Task::(prepare) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,247::task::995::Storage.TaskManager.Task::(_decref) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::task::930::Storage.TaskManager.Task::(_doAbort) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::task::597::Storage.TaskManager.Task::(_updateState) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::task::597::Storage.TaskManager.Task::(_updateState) Task=`642e5dcf-fba3-4d49-afff-350a60f3083a`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,248::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,249::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,249::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,249::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,249::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,249::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,278::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'f61362c1-054d-4084-8e44-2a1bfae19157'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,279::task::597::Storage.TaskManager.Task::(_updateState) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,279::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'f61362c1-054d-4084-8e44-2a1bfae19157', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,279::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ed718964-8be3-46b4-bbca-4d92e1361160`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,280::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,280::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,280::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ed718964-8be3-46b4-bbca-4d92e1361160`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,280::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,280::task::995::Storage.TaskManager.Task::(_decref) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,280::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,281::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,281::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,305::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:09,306::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,310::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,311::task::868::Storage.TaskManager.Task::(_setError) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,311::task::887::Storage.TaskManager.Task::(_run) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::Task._run: fe8fced7-9937-4612-a216-2595d72f3bac (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'f61362c1-054d-4084-8e44-2a1bfae19157') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,311::task::1248::Storage.TaskManager.Task::(stop) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,311::task::995::Storage.TaskManager.Task::(_decref) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,311::task::1173::Storage.TaskManager.Task::(prepare) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,311::task::1178::Storage.TaskManager.Task::(prepare) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,311::task::995::Storage.TaskManager.Task::(_decref) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::task::930::Storage.TaskManager.Task::(_doAbort) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::task::597::Storage.TaskManager.Task::(_updateState) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::task::597::Storage.TaskManager.Task::(_updateState) Task=`fe8fced7-9937-4612-a216-2595d72f3bac`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,312::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,313::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,313::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,313::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,313::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,313::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,342::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'51799ca6-6ade-4f7c-b8e5-db8c4c768255'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,343::task::597::Storage.TaskManager.Task::(_updateState) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,343::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'51799ca6-6ade-4f7c-b8e5-db8c4c768255', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1d2d4509-e135-4c2f-bb30-17292f0fb6a8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1d2d4509-e135-4c2f-bb30-17292f0fb6a8`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,344::task::995::Storage.TaskManager.Task::(_decref) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,345::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,345::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,345::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,370::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:09,371::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,375::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,376::task::868::Storage.TaskManager.Task::(_setError) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,376::task::887::Storage.TaskManager.Task::(_run) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::Task._run: ab1bf7e6-daef-4efa-a7e4-1790f2588c3a (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'51799ca6-6ade-4f7c-b8e5-db8c4c768255') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,376::task::1248::Storage.TaskManager.Task::(stop) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,376::task::995::Storage.TaskManager.Task::(_decref) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,376::task::1173::Storage.TaskManager.Task::(prepare) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,376::task::1178::Storage.TaskManager.Task::(prepare) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::task::995::Storage.TaskManager.Task::(_decref) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::task::930::Storage.TaskManager.Task::(_doAbort) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::task::597::Storage.TaskManager.Task::(_updateState) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::task::597::Storage.TaskManager.Task::(_updateState) Task=`ab1bf7e6-daef-4efa-a7e4-1790f2588c3a`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,377::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,378::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,378::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,378::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,378::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,378::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,407::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'6bab565c-11dd-407e-97bd-d391fb07855c'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,407::task::597::Storage.TaskManager.Task::(_updateState) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,407::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'6bab565c-11dd-407e-97bd-d391fb07855c', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,408::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3c0813ac-8073-4f69-994b-b3ed1586e55d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,408::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,408::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,408::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3c0813ac-8073-4f69-994b-b3ed1586e55d`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,408::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,409::task::995::Storage.TaskManager.Task::(_decref) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,409::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,409::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,409::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,433::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:09,434::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,439::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,439::task::868::Storage.TaskManager.Task::(_setError) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::887::Storage.TaskManager.Task::(_run) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::Task._run: fecfb61f-f0e8-47a7-a334-cb89134d1c0e (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'6bab565c-11dd-407e-97bd-d391fb07855c') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::1248::Storage.TaskManager.Task::(stop) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::995::Storage.TaskManager.Task::(_decref) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,440::task::1173::Storage.TaskManager.Task::(prepare) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::1178::Storage.TaskManager.Task::(prepare) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::995::Storage.TaskManager.Task::(_decref) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,440::task::930::Storage.TaskManager.Task::(_doAbort) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::task::597::Storage.TaskManager.Task::(_updateState) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::task::597::Storage.TaskManager.Task::(_updateState) Task=`fecfb61f-f0e8-47a7-a334-cb89134d1c0e`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,441::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,442::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,442::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,442::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,471::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'079a34cc-1019-4916-91fa-c1e3be11e734'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,471::task::597::Storage.TaskManager.Task::(_updateState) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:09,472::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'079a34cc-1019-4916-91fa-c1e3be11e734', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,472::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`dc15398b-7c82-4362-a90e-75683295175d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,472::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,472::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,472::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`dc15398b-7c82-4362-a90e-75683295175d`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,473::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,473::task::995::Storage.TaskManager.Task::(_decref) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,473::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,473::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,473::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,497::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:09,498::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,503::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,503::task::868::Storage.TaskManager.Task::(_setError) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,503::task::887::Storage.TaskManager.Task::(_run) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::Task._run: 0c14ea70-ce40-4f1c-a575-60cadf7d0ba7 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'079a34cc-1019-4916-91fa-c1e3be11e734') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::1248::Storage.TaskManager.Task::(stop) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::995::Storage.TaskManager.Task::(_decref) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:09,504::task::1173::Storage.TaskManager.Task::(prepare) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::1178::Storage.TaskManager.Task::(prepare) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::995::Storage.TaskManager.Task::(_decref) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::930::Storage.TaskManager.Task::(_doAbort) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::597::Storage.TaskManager.Task::(_updateState) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,504::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::task::597::Storage.TaskManager.Task::(_updateState) Task=`0c14ea70-ce40-4f1c-a575-60cadf7d0ba7`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,505::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,506::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:09,506::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,535::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'f142ee2e-a247-4f0c-90d3-10ca1cd5dfaf'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,535::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,536::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'f142ee2e-a247-4f0c-90d3-10ca1cd5dfaf', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,536::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d6b0b1b7-922f-49c6-a305-1fec636436bb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,536::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,536::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,537::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`d6b0b1b7-922f-49c6-a305-1fec636436bb`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,537::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,537::task::995::Storage.TaskManager.Task::(_decref) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,537::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,537::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,537::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,561::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:09,562::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,567::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,567::task::868::Storage.TaskManager.Task::(_setError) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::887::Storage.TaskManager.Task::(_run) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::Task._run: f83509c2-8013-416c-92a1-e6a06c016e71 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'f142ee2e-a247-4f0c-90d3-10ca1cd5dfaf') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::1248::Storage.TaskManager.Task::(stop) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::995::Storage.TaskManager.Task::(_decref) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,568::task::1173::Storage.TaskManager.Task::(prepare) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::1178::Storage.TaskManager.Task::(prepare) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::995::Storage.TaskManager.Task::(_decref) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::930::Storage.TaskManager.Task::(_doAbort) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,568::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83509c2-8013-416c-92a1-e6a06c016e71`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:09,569::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:09,570::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:09,570::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,599::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'18c6297f-aaab-46ce-9ca5-5960d45644d8'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,600::task::597::Storage.TaskManager.Task::(_updateState) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,600::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'18c6297f-aaab-46ce-9ca5-5960d45644d8', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,600::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`317dddc8-4b67-4c9e-9e24-43b4ace7a846`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,600::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,601::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,601::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`317dddc8-4b67-4c9e-9e24-43b4ace7a846`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,601::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,601::task::995::Storage.TaskManager.Task::(_decref) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,601::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,601::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,602::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,626::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:09,627::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,631::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,632::task::868::Storage.TaskManager.Task::(_setError) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,632::task::887::Storage.TaskManager.Task::(_run) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::Task._run: e29d6518-9f5e-4dca-b160-183681c3d930 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'18c6297f-aaab-46ce-9ca5-5960d45644d8') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,632::task::1248::Storage.TaskManager.Task::(stop) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,632::task::995::Storage.TaskManager.Task::(_decref) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,632::task::1173::Storage.TaskManager.Task::(prepare) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,632::task::1178::Storage.TaskManager.Task::(prepare) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,632::task::995::Storage.TaskManager.Task::(_decref) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::task::930::Storage.TaskManager.Task::(_doAbort) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::task::597::Storage.TaskManager.Task::(_updateState) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::task::597::Storage.TaskManager.Task::(_updateState) Task=`e29d6518-9f5e-4dca-b160-183681c3d930`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,633::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,634::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,634::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:09,634::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:09,634::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:09,634::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,663::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'fe563820-7b32-4cf9-ae5c-ee626ae50035'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,664::task::597::Storage.TaskManager.Task::(_updateState) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,664::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'fe563820-7b32-4cf9-ae5c-ee626ae50035', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,664::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf28fc7c-de4f-4b27-9303-4298557adf01`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,664::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,665::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,665::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf28fc7c-de4f-4b27-9303-4298557adf01`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,665::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,665::task::995::Storage.TaskManager.Task::(_decref) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,665::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,665::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,666::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,690::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:09,690::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,695::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,695::task::868::Storage.TaskManager.Task::(_setError) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::887::Storage.TaskManager.Task::(_run) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::Task._run: 1f6dfc7c-f688-409e-999b-54166a3be0a5 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'fe563820-7b32-4cf9-ae5c-ee626ae50035') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::1248::Storage.TaskManager.Task::(stop) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::995::Storage.TaskManager.Task::(_decref) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,696::task::1173::Storage.TaskManager.Task::(prepare) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::1178::Storage.TaskManager.Task::(prepare) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::995::Storage.TaskManager.Task::(_decref) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,696::task::930::Storage.TaskManager.Task::(_doAbort) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::task::597::Storage.TaskManager.Task::(_updateState) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::task::597::Storage.TaskManager.Task::(_updateState) Task=`1f6dfc7c-f688-409e-999b-54166a3be0a5`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,697::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,698::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:09,698::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:09,698::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:09,698::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,727::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'030097c1-80f4-47a6-b40f-a881929d70ff'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,728::task::597::Storage.TaskManager.Task::(_updateState) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,728::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'030097c1-80f4-47a6-b40f-a881929d70ff', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,728::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3a543357-55b5-485d-870a-b94f10621057`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,729::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,729::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,729::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3a543357-55b5-485d-870a-b94f10621057`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,729::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,729::task::995::Storage.TaskManager.Task::(_decref) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,729::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,730::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,730::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,753::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:09,754::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,759::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,759::task::868::Storage.TaskManager.Task::(_setError) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::887::Storage.TaskManager.Task::(_run) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::Task._run: bc910013-a753-4b70-9c13-dca22dc71919 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'030097c1-80f4-47a6-b40f-a881929d70ff') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::1248::Storage.TaskManager.Task::(stop) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::995::Storage.TaskManager.Task::(_decref) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,760::task::1173::Storage.TaskManager.Task::(prepare) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::1178::Storage.TaskManager.Task::(prepare) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::995::Storage.TaskManager.Task::(_decref) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,760::task::930::Storage.TaskManager.Task::(_doAbort) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::task::597::Storage.TaskManager.Task::(_updateState) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::task::597::Storage.TaskManager.Task::(_updateState) Task=`bc910013-a753-4b70-9c13-dca22dc71919`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,761::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,762::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:09,762::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:09,762::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:09,762::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,791::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'3f282ee0-bc1e-4cbb-8592-88862d47d7f7'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,792::task::597::Storage.TaskManager.Task::(_updateState) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,792::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'3f282ee0-bc1e-4cbb-8592-88862d47d7f7', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`242248a0-4c52-415a-83db-4d20d74d8076`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`242248a0-4c52-415a-83db-4d20d74d8076`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,793::task::995::Storage.TaskManager.Task::(_decref) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,794::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,794::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,794::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,818::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:09,819::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,823::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,824::task::868::Storage.TaskManager.Task::(_setError) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,824::task::887::Storage.TaskManager.Task::(_run) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::Task._run: 8f503779-f8d9-4051-9f0e-ea61faa69bd6 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'3f282ee0-bc1e-4cbb-8592-88862d47d7f7') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,824::task::1248::Storage.TaskManager.Task::(stop) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,824::task::995::Storage.TaskManager.Task::(_decref) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,824::task::1173::Storage.TaskManager.Task::(prepare) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,824::task::1178::Storage.TaskManager.Task::(prepare) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::task::995::Storage.TaskManager.Task::(_decref) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::task::597::Storage.TaskManager.Task::(_updateState) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::task::597::Storage.TaskManager.Task::(_updateState) Task=`8f503779-f8d9-4051-9f0e-ea61faa69bd6`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,825::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,826::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,826::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:09,826::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:09,826::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:09,826::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,856::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd55038d0-8c92-44f3-b303-57ae9c22adde'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,856::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,856::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd55038d0-8c92-44f3-b303-57ae9c22adde', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,857::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6f234731-1ef8-49e6-bd95-e2839501b2ff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,857::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,857::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,857::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6f234731-1ef8-49e6-bd95-e2839501b2ff`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,858::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,858::task::995::Storage.TaskManager.Task::(_decref) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,858::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,858::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,858::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,882::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:09,883::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,888::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,888::task::868::Storage.TaskManager.Task::(_setError) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::887::Storage.TaskManager.Task::(_run) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::Task._run: 4c44368b-456e-4dd3-8588-2cc0514de833 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd55038d0-8c92-44f3-b303-57ae9c22adde') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::1248::Storage.TaskManager.Task::(stop) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::995::Storage.TaskManager.Task::(_decref) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,889::task::1173::Storage.TaskManager.Task::(prepare) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::1178::Storage.TaskManager.Task::(prepare) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::995::Storage.TaskManager.Task::(_decref) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::task::930::Storage.TaskManager.Task::(_doAbort) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,889::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::task::597::Storage.TaskManager.Task::(_updateState) Task=`4c44368b-456e-4dd3-8588-2cc0514de833`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,890::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:09,891::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:09,891::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:09,891::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,920::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'355617d2-e82c-4dc5-aee7-267770cc5c70'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,920::task::597::Storage.TaskManager.Task::(_updateState) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,921::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'355617d2-e82c-4dc5-aee7-267770cc5c70', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,921::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`336145dc-c9e4-46b6-b8ff-078056a6ea99`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,921::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,921::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,921::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`336145dc-c9e4-46b6-b8ff-078056a6ea99`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,922::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,922::task::995::Storage.TaskManager.Task::(_decref) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,922::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,922::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,922::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,946::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:09,947::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,952::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,952::task::868::Storage.TaskManager.Task::(_setError) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,952::task::887::Storage.TaskManager.Task::(_run) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::Task._run: 7a96dd05-199e-469b-bce6-5268387071cc (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'355617d2-e82c-4dc5-aee7-267770cc5c70') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,952::task::1248::Storage.TaskManager.Task::(stop) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,952::task::995::Storage.TaskManager.Task::(_decref) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,953::task::1173::Storage.TaskManager.Task::(prepare) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::1178::Storage.TaskManager.Task::(prepare) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::995::Storage.TaskManager.Task::(_decref) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::930::Storage.TaskManager.Task::(_doAbort) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::597::Storage.TaskManager.Task::(_updateState) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::task::597::Storage.TaskManager.Task::(_updateState) Task=`7a96dd05-199e-469b-bce6-5268387071cc`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,953::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,954::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,954::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,954::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,954::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:09,954::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:09,954::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:09,955::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,984::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'8d856357-ed7f-4b0d-9fd6-cb33e070d562'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,985::task::597::Storage.TaskManager.Task::(_updateState) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:09,985::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'8d856357-ed7f-4b0d-9fd6-cb33e070d562', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,985::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`a7bd4389-65c4-4845-8a54-ee3be632ce30`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,985::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,985::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,986::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`a7bd4389-65c4-4845-8a54-ee3be632ce30`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,986::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,986::task::995::Storage.TaskManager.Task::(_decref) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,986::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:09,986::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:09,986::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,010::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:10,011::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,016::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,016::task::868::Storage.TaskManager.Task::(_setError) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::887::Storage.TaskManager.Task::(_run) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::Task._run: 643e83b1-7751-4baa-b0d4-8dc1bfbba54e (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'8d856357-ed7f-4b0d-9fd6-cb33e070d562') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::1248::Storage.TaskManager.Task::(stop) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::995::Storage.TaskManager.Task::(_decref) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:10,017::task::1173::Storage.TaskManager.Task::(prepare) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::1178::Storage.TaskManager.Task::(prepare) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::995::Storage.TaskManager.Task::(_decref) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::930::Storage.TaskManager.Task::(_doAbort) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,017::task::597::Storage.TaskManager.Task::(_updateState) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::task::597::Storage.TaskManager.Task::(_updateState) Task=`643e83b1-7751-4baa-b0d4-8dc1bfbba54e`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,018::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,019::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:10,019::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,048::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'b920d22f-592c-4fa7-a189-00f5f43a6be9'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,049::task::597::Storage.TaskManager.Task::(_updateState) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,049::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'b920d22f-592c-4fa7-a189-00f5f43a6be9', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,049::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3d10f48a-3219-4cb4-bdcd-5a442b6ac657`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,049::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,050::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,050::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3d10f48a-3219-4cb4-bdcd-5a442b6ac657`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,050::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,050::task::995::Storage.TaskManager.Task::(_decref) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,050::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,050::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,051::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,075::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:10,076::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,080::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,081::task::868::Storage.TaskManager.Task::(_setError) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,081::task::887::Storage.TaskManager.Task::(_run) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::Task._run: 0fa0a41b-467c-4738-99fd-a384d2f7623a (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'b920d22f-592c-4fa7-a189-00f5f43a6be9') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,081::task::1248::Storage.TaskManager.Task::(stop) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,081::task::995::Storage.TaskManager.Task::(_decref) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,081::task::1173::Storage.TaskManager.Task::(prepare) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,081::task::1178::Storage.TaskManager.Task::(prepare) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::task::995::Storage.TaskManager.Task::(_decref) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::task::930::Storage.TaskManager.Task::(_doAbort) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::task::597::Storage.TaskManager.Task::(_updateState) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::task::597::Storage.TaskManager.Task::(_updateState) Task=`0fa0a41b-467c-4738-99fd-a384d2f7623a`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,082::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,083::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,083::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,083::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,083::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,083::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,112::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'939b85e4-cbfd-48e4-97f4-4a27f1b34ce3'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,113::task::597::Storage.TaskManager.Task::(_updateState) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,113::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'939b85e4-cbfd-48e4-97f4-4a27f1b34ce3', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,113::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8ad73648-80aa-4e82-a28b-428b7d01fd48`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,114::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,114::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,114::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8ad73648-80aa-4e82-a28b-428b7d01fd48`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,114::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,114::task::995::Storage.TaskManager.Task::(_decref) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,114::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,115::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,115::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,139::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:10,140::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,145::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,145::task::868::Storage.TaskManager.Task::(_setError) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,145::task::887::Storage.TaskManager.Task::(_run) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::Task._run: f76b55f6-0998-4276-8ddb-64b5a989fbf7 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'939b85e4-cbfd-48e4-97f4-4a27f1b34ce3') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,145::task::1248::Storage.TaskManager.Task::(stop) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::995::Storage.TaskManager.Task::(_decref) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,146::task::1173::Storage.TaskManager.Task::(prepare) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::1178::Storage.TaskManager.Task::(prepare) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::995::Storage.TaskManager.Task::(_decref) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::930::Storage.TaskManager.Task::(_doAbort) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::597::Storage.TaskManager.Task::(_updateState) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::task::597::Storage.TaskManager.Task::(_updateState) Task=`f76b55f6-0998-4276-8ddb-64b5a989fbf7`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,146::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,147::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,147::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,147::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,147::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,147::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,147::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,148::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,176::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'a5b19c52-4e79-4cbb-ac07-66ed4671310a'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,177::task::597::Storage.TaskManager.Task::(_updateState) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,177::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'a5b19c52-4e79-4cbb-ac07-66ed4671310a', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f7c6e41d-4d57-493e-94b2-a279ac020137`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f7c6e41d-4d57-493e-94b2-a279ac020137`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,178::task::995::Storage.TaskManager.Task::(_decref) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,179::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,179::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,179::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,203::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:10,204::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,209::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,209::task::868::Storage.TaskManager.Task::(_setError) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,209::task::887::Storage.TaskManager.Task::(_run) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::Task._run: 8a5c7c73-5d17-4d1b-9283-37c71b87f578 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'a5b19c52-4e79-4cbb-ac07-66ed4671310a') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,209::task::1248::Storage.TaskManager.Task::(stop) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::995::Storage.TaskManager.Task::(_decref) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,210::task::1173::Storage.TaskManager.Task::(prepare) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::1178::Storage.TaskManager.Task::(prepare) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::995::Storage.TaskManager.Task::(_decref) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::597::Storage.TaskManager.Task::(_updateState) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,210::task::597::Storage.TaskManager.Task::(_updateState) Task=`8a5c7c73-5d17-4d1b-9283-37c71b87f578`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,211::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,212::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,212::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,241::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'02a904c0-6c7f-45c5-8f9a-e4b4a84a3dc6'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,241::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,241::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'02a904c0-6c7f-45c5-8f9a-e4b4a84a3dc6', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,242::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bd385544-6624-4901-9c84-e4de4c6ede0a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,242::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,242::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,242::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bd385544-6624-4901-9c84-e4de4c6ede0a`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,243::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,243::task::995::Storage.TaskManager.Task::(_decref) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,243::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,243::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,243::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,267::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:10,268::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,273::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,273::task::868::Storage.TaskManager.Task::(_setError) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,273::task::887::Storage.TaskManager.Task::(_run) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::Task._run: 3a5da29a-5d94-4358-8235-081954aa1e18 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'02a904c0-6c7f-45c5-8f9a-e4b4a84a3dc6') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,273::task::1248::Storage.TaskManager.Task::(stop) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::995::Storage.TaskManager.Task::(_decref) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,274::task::1173::Storage.TaskManager.Task::(prepare) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::1178::Storage.TaskManager.Task::(prepare) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::995::Storage.TaskManager.Task::(_decref) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::930::Storage.TaskManager.Task::(_doAbort) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,274::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a5da29a-5d94-4358-8235-081954aa1e18`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,275::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,275::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,276::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,305::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'437cbd0f-da60-445a-b312-1842f87c8c44'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,305::task::597::Storage.TaskManager.Task::(_updateState) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,306::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'437cbd0f-da60-445a-b312-1842f87c8c44', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,306::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f9e7a1d8-49cf-4058-87be-b3d831f4052d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,306::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,306::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,306::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f9e7a1d8-49cf-4058-87be-b3d831f4052d`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,307::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,307::task::995::Storage.TaskManager.Task::(_decref) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,307::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,307::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,307::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,331::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:10,332::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,337::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,337::task::868::Storage.TaskManager.Task::(_setError) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,337::task::887::Storage.TaskManager.Task::(_run) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::Task._run: 67658b0a-f09b-4b5e-9d90-1986f428cad5 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'437cbd0f-da60-445a-b312-1842f87c8c44') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::1248::Storage.TaskManager.Task::(stop) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::995::Storage.TaskManager.Task::(_decref) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,338::task::1173::Storage.TaskManager.Task::(prepare) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::1178::Storage.TaskManager.Task::(prepare) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::995::Storage.TaskManager.Task::(_decref) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::930::Storage.TaskManager.Task::(_doAbort) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::597::Storage.TaskManager.Task::(_updateState) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,338::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::task::597::Storage.TaskManager.Task::(_updateState) Task=`67658b0a-f09b-4b5e-9d90-1986f428cad5`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,339::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,340::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,340::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,369::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'be923431-8530-461b-8473-011702740010'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,370::task::597::Storage.TaskManager.Task::(_updateState) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,370::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'be923431-8530-461b-8473-011702740010', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,370::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f4e52874-d3c0-4ce3-a251-f6ade75a31e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,370::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,370::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,371::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f4e52874-d3c0-4ce3-a251-f6ade75a31e1`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,371::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,371::task::995::Storage.TaskManager.Task::(_decref) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,371::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,371::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,371::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,395::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:10,396::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,401::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,402::task::868::Storage.TaskManager.Task::(_setError) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,402::task::887::Storage.TaskManager.Task::(_run) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::Task._run: c5fd6f39-74aa-44ac-9934-a0efbae34df7 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'be923431-8530-461b-8473-011702740010') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,402::task::1248::Storage.TaskManager.Task::(stop) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,402::task::995::Storage.TaskManager.Task::(_decref) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,402::task::1173::Storage.TaskManager.Task::(prepare) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,402::task::1178::Storage.TaskManager.Task::(prepare) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,402::task::995::Storage.TaskManager.Task::(_decref) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::task::597::Storage.TaskManager.Task::(_updateState) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::task::597::Storage.TaskManager.Task::(_updateState) Task=`c5fd6f39-74aa-44ac-9934-a0efbae34df7`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,403::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,404::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,404::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,404::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,404::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,404::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,433::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'6418f797-d4e7-467d-9d29-0c275cfb4721'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,434::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,434::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'6418f797-d4e7-467d-9d29-0c275cfb4721', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,434::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ed84eb4e-6c60-4fde-ba1c-ed85cbbb8a7d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,434::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,435::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,435::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ed84eb4e-6c60-4fde-ba1c-ed85cbbb8a7d`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,435::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,435::task::995::Storage.TaskManager.Task::(_decref) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,435::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,435::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,436::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,459::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:10,460::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,465::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,466::task::868::Storage.TaskManager.Task::(_setError) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,466::task::887::Storage.TaskManager.Task::(_run) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::Task._run: 3a04b27f-1323-4a5f-8c64-e76603d75def (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'6418f797-d4e7-467d-9d29-0c275cfb4721') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,466::task::1248::Storage.TaskManager.Task::(stop) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,466::task::995::Storage.TaskManager.Task::(_decref) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,466::task::1173::Storage.TaskManager.Task::(prepare) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,466::task::1178::Storage.TaskManager.Task::(prepare) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,466::task::995::Storage.TaskManager.Task::(_decref) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::task::930::Storage.TaskManager.Task::(_doAbort) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::task::597::Storage.TaskManager.Task::(_updateState) Task=`3a04b27f-1323-4a5f-8c64-e76603d75def`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,467::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,468::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,468::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,468::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,468::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,468::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,497::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'3828c76e-2f5c-4be5-a3b5-7bee7bb412d5'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,498::task::597::Storage.TaskManager.Task::(_updateState) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:10,498::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'3828c76e-2f5c-4be5-a3b5-7bee7bb412d5', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,498::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8d4fd0fe-e810-4167-a78b-d97e4657b3ee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,498::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,499::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,499::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8d4fd0fe-e810-4167-a78b-d97e4657b3ee`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,499::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,499::task::995::Storage.TaskManager.Task::(_decref) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,499::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,499::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,500::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,523::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:10,524::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,529::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,530::task::868::Storage.TaskManager.Task::(_setError) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,530::task::887::Storage.TaskManager.Task::(_run) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::Task._run: 8047117c-270e-40c2-8750-6790f21a8ee8 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'3828c76e-2f5c-4be5-a3b5-7bee7bb412d5') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,530::task::1248::Storage.TaskManager.Task::(stop) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,530::task::995::Storage.TaskManager.Task::(_decref) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:10,530::task::1173::Storage.TaskManager.Task::(prepare) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,530::task::1178::Storage.TaskManager.Task::(prepare) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,530::task::995::Storage.TaskManager.Task::(_decref) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::task::930::Storage.TaskManager.Task::(_doAbort) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::task::597::Storage.TaskManager.Task::(_updateState) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::task::597::Storage.TaskManager.Task::(_updateState) Task=`8047117c-270e-40c2-8750-6790f21a8ee8`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,531::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,532::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:10,532::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:10,532::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:10,532::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,561::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd66338c3-3ef6-4e8d-ae80-217d84dff814'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,562::task::597::Storage.TaskManager.Task::(_updateState) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,562::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd66338c3-3ef6-4e8d-ae80-217d84dff814', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,562::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`82507e42-1bf6-4ad3-9594-67b155c24767`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,563::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,563::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,563::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`82507e42-1bf6-4ad3-9594-67b155c24767`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,563::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,563::task::995::Storage.TaskManager.Task::(_decref) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,563::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,564::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,564::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,588::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:10,589::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,593::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,594::task::868::Storage.TaskManager.Task::(_setError) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,594::task::887::Storage.TaskManager.Task::(_run) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::Task._run: c6fd7474-ebd2-4313-b908-516014a77426 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd66338c3-3ef6-4e8d-ae80-217d84dff814') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,594::task::1248::Storage.TaskManager.Task::(stop) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,594::task::995::Storage.TaskManager.Task::(_decref) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,594::task::1173::Storage.TaskManager.Task::(prepare) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,594::task::1178::Storage.TaskManager.Task::(prepare) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::task::995::Storage.TaskManager.Task::(_decref) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::task::597::Storage.TaskManager.Task::(_updateState) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::task::597::Storage.TaskManager.Task::(_updateState) Task=`c6fd7474-ebd2-4313-b908-516014a77426`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,595::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,596::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,596::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:10,596::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:10,596::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:10,596::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,625::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'2d690844-00bc-484a-91dc-5874d9a98493'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,626::task::597::Storage.TaskManager.Task::(_updateState) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,626::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'2d690844-00bc-484a-91dc-5874d9a98493', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf781775-a680-41f9-9b04-96e85807d2bf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`bf781775-a680-41f9-9b04-96e85807d2bf`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,627::task::995::Storage.TaskManager.Task::(_decref) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,628::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,628::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,628::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,652::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:10,653::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,658::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,658::task::868::Storage.TaskManager.Task::(_setError) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,658::task::887::Storage.TaskManager.Task::(_run) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::Task._run: ffe74412-a7d2-4a3c-a338-2a17d50259a2 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'2d690844-00bc-484a-91dc-5874d9a98493') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,658::task::1248::Storage.TaskManager.Task::(stop) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,658::task::995::Storage.TaskManager.Task::(_decref) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,659::task::1173::Storage.TaskManager.Task::(prepare) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::1178::Storage.TaskManager.Task::(prepare) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::995::Storage.TaskManager.Task::(_decref) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::930::Storage.TaskManager.Task::(_doAbort) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::597::Storage.TaskManager.Task::(_updateState) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::task::597::Storage.TaskManager.Task::(_updateState) Task=`ffe74412-a7d2-4a3c-a338-2a17d50259a2`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,659::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,660::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,660::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,660::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,660::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:10,660::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:10,660::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:10,661::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,689::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'a4d9dd90-5510-41a6-8880-ea74d521bf15'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,690::task::597::Storage.TaskManager.Task::(_updateState) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,690::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'a4d9dd90-5510-41a6-8880-ea74d521bf15', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,691::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cb33701e-0480-488e-888b-da5bd2f50a92`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,691::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,691::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,691::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cb33701e-0480-488e-888b-da5bd2f50a92`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,691::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,692::task::995::Storage.TaskManager.Task::(_decref) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,692::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,692::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,692::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,715::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:10,716::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,721::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,721::task::868::Storage.TaskManager.Task::(_setError) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::887::Storage.TaskManager.Task::(_run) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::Task._run: 22089512-1ad6-4522-ad79-c3881aa2ffc4 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'a4d9dd90-5510-41a6-8880-ea74d521bf15') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::1248::Storage.TaskManager.Task::(stop) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::995::Storage.TaskManager.Task::(_decref) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,722::task::1173::Storage.TaskManager.Task::(prepare) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::1178::Storage.TaskManager.Task::(prepare) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::995::Storage.TaskManager.Task::(_decref) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,722::task::930::Storage.TaskManager.Task::(_doAbort) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::task::597::Storage.TaskManager.Task::(_updateState) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::task::597::Storage.TaskManager.Task::(_updateState) Task=`22089512-1ad6-4522-ad79-c3881aa2ffc4`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,723::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,724::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,724::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:10,724::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:10,724::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:10,724::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,754::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'78c819af-6a8f-43ce-b5b7-0b3b1333cdc8'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,754::task::597::Storage.TaskManager.Task::(_updateState) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,755::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'78c819af-6a8f-43ce-b5b7-0b3b1333cdc8', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,755::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e08cbbb0-f0d3-40ad-9db5-82e676aa78ff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,755::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,755::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,755::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`e08cbbb0-f0d3-40ad-9db5-82e676aa78ff`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,756::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,756::task::995::Storage.TaskManager.Task::(_decref) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,756::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,756::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,756::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,780::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:10,781::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,786::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,786::task::868::Storage.TaskManager.Task::(_setError) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,786::task::887::Storage.TaskManager.Task::(_run) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::Task._run: bcdbe4ba-6e5e-495f-83cd-a408cf3925cd (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'78c819af-6a8f-43ce-b5b7-0b3b1333cdc8') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,786::task::1248::Storage.TaskManager.Task::(stop) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::995::Storage.TaskManager.Task::(_decref) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,787::task::1173::Storage.TaskManager.Task::(prepare) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::1178::Storage.TaskManager.Task::(prepare) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::995::Storage.TaskManager.Task::(_decref) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::930::Storage.TaskManager.Task::(_doAbort) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::597::Storage.TaskManager.Task::(_updateState) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,787::task::597::Storage.TaskManager.Task::(_updateState) Task=`bcdbe4ba-6e5e-495f-83cd-a408cf3925cd`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:10,788::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:10,788::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:10,789::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,818::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'874dea73-06eb-484c-8725-47c2db53162a'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,818::task::597::Storage.TaskManager.Task::(_updateState) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,819::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'874dea73-06eb-484c-8725-47c2db53162a', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,819::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9ad82069-079e-4d5f-97d4-63769bd8c1d1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,819::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,819::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,819::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`9ad82069-079e-4d5f-97d4-63769bd8c1d1`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,820::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,820::task::995::Storage.TaskManager.Task::(_decref) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,820::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,820::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,820::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,844::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:10,845::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,850::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,850::task::868::Storage.TaskManager.Task::(_setError) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::887::Storage.TaskManager.Task::(_run) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::Task._run: 6fb68bb9-1916-4370-a32d-7d941b90b80b (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'874dea73-06eb-484c-8725-47c2db53162a') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::1248::Storage.TaskManager.Task::(stop) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::995::Storage.TaskManager.Task::(_decref) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,851::task::1173::Storage.TaskManager.Task::(prepare) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::1178::Storage.TaskManager.Task::(prepare) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::995::Storage.TaskManager.Task::(_decref) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::930::Storage.TaskManager.Task::(_doAbort) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,851::task::597::Storage.TaskManager.Task::(_updateState) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::task::597::Storage.TaskManager.Task::(_updateState) Task=`6fb68bb9-1916-4370-a32d-7d941b90b80b`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:10,852::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:10,853::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:10,853::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,882::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'7effa041-0e19-4675-b029-6fe37f3ff3fb'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,883::task::597::Storage.TaskManager.Task::(_updateState) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,883::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'7effa041-0e19-4675-b029-6fe37f3ff3fb', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,883::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cca0277c-4084-411c-bb4a-579941c3fa7e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,883::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,883::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,884::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cca0277c-4084-411c-bb4a-579941c3fa7e`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,884::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,884::task::995::Storage.TaskManager.Task::(_decref) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,884::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,884::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,885::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,908::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:10,909::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,914::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,915::task::868::Storage.TaskManager.Task::(_setError) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,915::task::887::Storage.TaskManager.Task::(_run) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::Task._run: c39e495d-3d8f-4065-bd99-a83bbecd465e (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'7effa041-0e19-4675-b029-6fe37f3ff3fb') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,915::task::1248::Storage.TaskManager.Task::(stop) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,915::task::995::Storage.TaskManager.Task::(_decref) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,915::task::1173::Storage.TaskManager.Task::(prepare) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,915::task::1178::Storage.TaskManager.Task::(prepare) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,915::task::995::Storage.TaskManager.Task::(_decref) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::task::597::Storage.TaskManager.Task::(_updateState) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::task::597::Storage.TaskManager.Task::(_updateState) Task=`c39e495d-3d8f-4065-bd99-a83bbecd465e`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,916::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,917::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,917::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:10,917::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:10,917::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:10,917::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,946::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'bdaf35ab-c4d3-403b-b4be-b426d7d7f08e'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,947::task::597::Storage.TaskManager.Task::(_updateState) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,947::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'bdaf35ab-c4d3-403b-b4be-b426d7d7f08e', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,947::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3c13da7d-398f-4224-a28b-f1fd162491ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,947::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,948::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,948::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3c13da7d-398f-4224-a28b-f1fd162491ec`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,948::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,948::task::995::Storage.TaskManager.Task::(_decref) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,948::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,948::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,949::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,973::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:10,974::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,978::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,979::task::868::Storage.TaskManager.Task::(_setError) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,979::task::887::Storage.TaskManager.Task::(_run) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::Task._run: 049c01df-bdce-4203-a503-cb0ab756126b (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'bdaf35ab-c4d3-403b-b4be-b426d7d7f08e') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,979::task::1248::Storage.TaskManager.Task::(stop) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,979::task::995::Storage.TaskManager.Task::(_decref) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,979::task::1173::Storage.TaskManager.Task::(prepare) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,979::task::1178::Storage.TaskManager.Task::(prepare) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::task::995::Storage.TaskManager.Task::(_decref) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::task::930::Storage.TaskManager.Task::(_doAbort) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::task::597::Storage.TaskManager.Task::(_updateState) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::task::597::Storage.TaskManager.Task::(_updateState) Task=`049c01df-bdce-4203-a503-cb0ab756126b`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,980::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,981::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,981::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,981::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:10,981::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:10,981::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:10,981::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,010::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'028a03c2-b5d5-4a7f-a868-16a728d66396'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,011::task::597::Storage.TaskManager.Task::(_updateState) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,011::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'028a03c2-b5d5-4a7f-a868-16a728d66396', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,011::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`299a4f80-44ed-4672-9530-e307e33e77ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,011::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,012::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,012::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`299a4f80-44ed-4672-9530-e307e33e77ef`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,012::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,012::task::995::Storage.TaskManager.Task::(_decref) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,012::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,012::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,013::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,037::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:11,038::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,042::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,043::task::868::Storage.TaskManager.Task::(_setError) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,043::task::887::Storage.TaskManager.Task::(_run) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::Task._run: 2c5e47de-c754-411a-b45a-29a8e65b8393 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'028a03c2-b5d5-4a7f-a868-16a728d66396') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,043::task::1248::Storage.TaskManager.Task::(stop) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,043::task::995::Storage.TaskManager.Task::(_decref) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,043::task::1173::Storage.TaskManager.Task::(prepare) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,043::task::1178::Storage.TaskManager.Task::(prepare) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::task::995::Storage.TaskManager.Task::(_decref) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::task::930::Storage.TaskManager.Task::(_doAbort) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::task::597::Storage.TaskManager.Task::(_updateState) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::task::597::Storage.TaskManager.Task::(_updateState) Task=`2c5e47de-c754-411a-b45a-29a8e65b8393`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,044::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,045::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,045::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,045::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,045::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,045::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,074::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'91d4607d-0782-47fb-b9a0-808500ed6a88'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,075::task::597::Storage.TaskManager.Task::(_updateState) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,075::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'91d4607d-0782-47fb-b9a0-808500ed6a88', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,075::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f6a68313-f08f-49df-952a-169bad0fd7e3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,076::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,076::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,076::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f6a68313-f08f-49df-952a-169bad0fd7e3`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,076::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,076::task::995::Storage.TaskManager.Task::(_decref) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,076::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,077::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,077::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,101::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:11,102::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,106::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,107::task::868::Storage.TaskManager.Task::(_setError) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,107::task::887::Storage.TaskManager.Task::(_run) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::Task._run: 301d1305-bf65-49a5-ab66-5b2f7c7994a1 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'91d4607d-0782-47fb-b9a0-808500ed6a88') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,107::task::1248::Storage.TaskManager.Task::(stop) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,107::task::995::Storage.TaskManager.Task::(_decref) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,107::task::1173::Storage.TaskManager.Task::(prepare) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,107::task::1178::Storage.TaskManager.Task::(prepare) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,107::task::995::Storage.TaskManager.Task::(_decref) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::task::930::Storage.TaskManager.Task::(_doAbort) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::task::597::Storage.TaskManager.Task::(_updateState) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::task::597::Storage.TaskManager.Task::(_updateState) Task=`301d1305-bf65-49a5-ab66-5b2f7c7994a1`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,108::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,109::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,109::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,109::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,109::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,109::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,138::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'f44635c8-b248-4000-aed2-a7c1cce10144'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,139::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,139::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'f44635c8-b248-4000-aed2-a7c1cce10144', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`039824ef-4111-4c05-89d1-041206493a82`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`039824ef-4111-4c05-89d1-041206493a82`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,140::task::995::Storage.TaskManager.Task::(_decref) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,141::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,141::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,141::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,165::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:11,166::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,171::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,171::task::868::Storage.TaskManager.Task::(_setError) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,171::task::887::Storage.TaskManager.Task::(_run) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::Task._run: b7566ea8-2f0b-49aa-96a0-921f838611ff (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'f44635c8-b248-4000-aed2-a7c1cce10144') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,171::task::1248::Storage.TaskManager.Task::(stop) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,171::task::995::Storage.TaskManager.Task::(_decref) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,172::task::1173::Storage.TaskManager.Task::(prepare) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::1178::Storage.TaskManager.Task::(prepare) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::995::Storage.TaskManager.Task::(_decref) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::930::Storage.TaskManager.Task::(_doAbort) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7566ea8-2f0b-49aa-96a0-921f838611ff`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,172::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,173::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,173::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,173::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,173::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,173::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,173::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,174::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,202::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'453201e4-f801-4e18-876f-3a601b81d42d'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,203::task::597::Storage.TaskManager.Task::(_updateState) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,203::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'453201e4-f801-4e18-876f-3a601b81d42d', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,204::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`136263cb-72c6-49d1-8895-6e3a501c05a6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,204::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,204::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,204::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`136263cb-72c6-49d1-8895-6e3a501c05a6`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,204::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,205::task::995::Storage.TaskManager.Task::(_decref) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,205::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,205::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,205::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,229::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:11,230::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,235::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,235::task::868::Storage.TaskManager.Task::(_setError) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,235::task::887::Storage.TaskManager.Task::(_run) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::Task._run: c0f7ea75-5415-4184-93fe-4f0a537c2293 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'453201e4-f801-4e18-876f-3a601b81d42d') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,235::task::1248::Storage.TaskManager.Task::(stop) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::995::Storage.TaskManager.Task::(_decref) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,236::task::1173::Storage.TaskManager.Task::(prepare) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::1178::Storage.TaskManager.Task::(prepare) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::995::Storage.TaskManager.Task::(_decref) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::597::Storage.TaskManager.Task::(_updateState) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,236::task::597::Storage.TaskManager.Task::(_updateState) Task=`c0f7ea75-5415-4184-93fe-4f0a537c2293`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,237::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,237::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,238::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,267::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'aafb4855-0c78-4e58-be6a-479bcfd065fd'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,267::task::597::Storage.TaskManager.Task::(_updateState) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,268::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'aafb4855-0c78-4e58-be6a-479bcfd065fd', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,268::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`48239235-4e6a-44f4-8df0-3b5e83f539ba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,268::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,268::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,268::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`48239235-4e6a-44f4-8df0-3b5e83f539ba`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,269::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,269::task::995::Storage.TaskManager.Task::(_decref) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,269::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,269::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,269::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,293::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:11,294::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,299::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,299::task::868::Storage.TaskManager.Task::(_setError) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::887::Storage.TaskManager.Task::(_run) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::Task._run: c984bcef-252f-40fb-a9f8-6b9cb52b44d3 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'aafb4855-0c78-4e58-be6a-479bcfd065fd') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::1248::Storage.TaskManager.Task::(stop) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::995::Storage.TaskManager.Task::(_decref) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,300::task::1173::Storage.TaskManager.Task::(prepare) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::1178::Storage.TaskManager.Task::(prepare) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::995::Storage.TaskManager.Task::(_decref) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,300::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::task::597::Storage.TaskManager.Task::(_updateState) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::task::597::Storage.TaskManager.Task::(_updateState) Task=`c984bcef-252f-40fb-a9f8-6b9cb52b44d3`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,301::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,302::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,302::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,302::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,331::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'a3c38056-acd4-4d5b-aa73-68f633034033'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,331::task::597::Storage.TaskManager.Task::(_updateState) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,332::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'a3c38056-acd4-4d5b-aa73-68f633034033', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,332::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ea2399fb-e953-4dfe-a0cb-6965d3afe261`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,332::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,332::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,332::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`ea2399fb-e953-4dfe-a0cb-6965d3afe261`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,333::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,333::task::995::Storage.TaskManager.Task::(_decref) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,333::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,333::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,333::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,357::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:11,358::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,363::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,363::task::868::Storage.TaskManager.Task::(_setError) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,363::task::887::Storage.TaskManager.Task::(_run) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::Task._run: d7b2d8ad-893b-4346-9242-76c1e2c2e1d2 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'a3c38056-acd4-4d5b-aa73-68f633034033') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::1248::Storage.TaskManager.Task::(stop) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::995::Storage.TaskManager.Task::(_decref) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,364::task::1173::Storage.TaskManager.Task::(prepare) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::1178::Storage.TaskManager.Task::(prepare) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::995::Storage.TaskManager.Task::(_decref) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::930::Storage.TaskManager.Task::(_doAbort) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::597::Storage.TaskManager.Task::(_updateState) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,364::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::task::597::Storage.TaskManager.Task::(_updateState) Task=`d7b2d8ad-893b-4346-9242-76c1e2c2e1d2`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,365::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,366::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,366::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,395::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'8291cef3-4af2-472f-98bf-1af38dff1a6d'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,395::task::597::Storage.TaskManager.Task::(_updateState) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,396::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'8291cef3-4af2-472f-98bf-1af38dff1a6d', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,396::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`52aff388-dd59-4160-9375-4a548c23b197`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,396::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,396::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,396::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`52aff388-dd59-4160-9375-4a548c23b197`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,397::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,397::task::995::Storage.TaskManager.Task::(_decref) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,397::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,397::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,397::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,421::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:11,422::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,427::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,427::task::868::Storage.TaskManager.Task::(_setError) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,427::task::887::Storage.TaskManager.Task::(_run) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::Task._run: b33ab7f3-7546-49a5-a2de-f669ac690298 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'8291cef3-4af2-472f-98bf-1af38dff1a6d') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::1248::Storage.TaskManager.Task::(stop) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::995::Storage.TaskManager.Task::(_decref) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,428::task::1173::Storage.TaskManager.Task::(prepare) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::1178::Storage.TaskManager.Task::(prepare) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::995::Storage.TaskManager.Task::(_decref) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::930::Storage.TaskManager.Task::(_doAbort) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::597::Storage.TaskManager.Task::(_updateState) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,428::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::task::597::Storage.TaskManager.Task::(_updateState) Task=`b33ab7f3-7546-49a5-a2de-f669ac690298`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,429::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,430::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,430::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,459::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'dcd0ef41-c38e-4e99-87ae-305f71a0cb6c'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,460::task::597::Storage.TaskManager.Task::(_updateState) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:11,460::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'dcd0ef41-c38e-4e99-87ae-305f71a0cb6c', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,460::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`89bbbd44-7670-464d-95b4-953e41bbb9fb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,460::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,461::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,461::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`89bbbd44-7670-464d-95b4-953e41bbb9fb`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,461::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,461::task::995::Storage.TaskManager.Task::(_decref) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,461::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,461::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,462::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,485::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:11,486::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,491::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,492::task::868::Storage.TaskManager.Task::(_setError) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,492::task::887::Storage.TaskManager.Task::(_run) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::Task._run: 3cd7cdda-cece-4b0a-a27b-519c6fc5c634 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'dcd0ef41-c38e-4e99-87ae-305f71a0cb6c') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,492::task::1248::Storage.TaskManager.Task::(stop) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,492::task::995::Storage.TaskManager.Task::(_decref) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:11,492::task::1173::Storage.TaskManager.Task::(prepare) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,492::task::1178::Storage.TaskManager.Task::(prepare) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,492::task::995::Storage.TaskManager.Task::(_decref) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::task::930::Storage.TaskManager.Task::(_doAbort) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::task::597::Storage.TaskManager.Task::(_updateState) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::task::597::Storage.TaskManager.Task::(_updateState) Task=`3cd7cdda-cece-4b0a-a27b-519c6fc5c634`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,493::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,494::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,494::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,494::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,494::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:11,494::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,523::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'39e9a84b-3aef-4a22-b7fc-4b7be9bd66d5'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,524::task::597::Storage.TaskManager.Task::(_updateState) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,524::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'39e9a84b-3aef-4a22-b7fc-4b7be9bd66d5', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,524::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`17616478-c875-47b4-8f27-08159662a958`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,525::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,525::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,525::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`17616478-c875-47b4-8f27-08159662a958`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,525::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,525::task::995::Storage.TaskManager.Task::(_decref) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,525::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,526::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,526::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,549::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:11,550::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,555::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,556::task::868::Storage.TaskManager.Task::(_setError) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,556::task::887::Storage.TaskManager.Task::(_run) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::Task._run: 076950fa-b35e-42b4-83c2-bfd566ce6b04 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'39e9a84b-3aef-4a22-b7fc-4b7be9bd66d5') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,556::task::1248::Storage.TaskManager.Task::(stop) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,556::task::995::Storage.TaskManager.Task::(_decref) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,556::task::1173::Storage.TaskManager.Task::(prepare) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,556::task::1178::Storage.TaskManager.Task::(prepare) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::task::995::Storage.TaskManager.Task::(_decref) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::task::930::Storage.TaskManager.Task::(_doAbort) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::task::597::Storage.TaskManager.Task::(_updateState) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::task::597::Storage.TaskManager.Task::(_updateState) Task=`076950fa-b35e-42b4-83c2-bfd566ce6b04`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,557::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,558::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,558::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:11,558::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:11,558::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:11,558::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,588::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'55fd0684-f09a-46d7-8e24-3f4d0d74b145'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,588::task::597::Storage.TaskManager.Task::(_updateState) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,588::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'55fd0684-f09a-46d7-8e24-3f4d0d74b145', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,589::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`889fe687-fae6-4b80-9346-0b75fd254e74`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,589::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,589::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,589::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`889fe687-fae6-4b80-9346-0b75fd254e74`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,589::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,590::task::995::Storage.TaskManager.Task::(_decref) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,590::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,590::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,590::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,614::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:11,615::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,620::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,620::task::868::Storage.TaskManager.Task::(_setError) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,620::task::887::Storage.TaskManager.Task::(_run) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::Task._run: de7bb74e-4dce-4353-8ae1-4fd42165cd17 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'55fd0684-f09a-46d7-8e24-3f4d0d74b145') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,620::task::1248::Storage.TaskManager.Task::(stop) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::995::Storage.TaskManager.Task::(_decref) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,621::task::1173::Storage.TaskManager.Task::(prepare) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::1178::Storage.TaskManager.Task::(prepare) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::995::Storage.TaskManager.Task::(_decref) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::930::Storage.TaskManager.Task::(_doAbort) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::597::Storage.TaskManager.Task::(_updateState) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,621::task::597::Storage.TaskManager.Task::(_updateState) Task=`de7bb74e-4dce-4353-8ae1-4fd42165cd17`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:11,622::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:11,622::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:11,623::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,652::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'9251ebc2-512e-4214-ad5b-953a898cef9e'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,652::task::597::Storage.TaskManager.Task::(_updateState) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,652::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'9251ebc2-512e-4214-ad5b-953a898cef9e', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,653::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b8a72369-f2c6-4d43-9566-63f56d041086`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,653::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,653::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,653::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`b8a72369-f2c6-4d43-9566-63f56d041086`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,654::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,654::task::995::Storage.TaskManager.Task::(_decref) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,654::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,654::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,654::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,678::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:11,679::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,684::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,684::task::868::Storage.TaskManager.Task::(_setError) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,684::task::887::Storage.TaskManager.Task::(_run) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::Task._run: 930a9ded-adab-4cc4-9daf-2e5138ee3c51 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'9251ebc2-512e-4214-ad5b-953a898cef9e') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::1248::Storage.TaskManager.Task::(stop) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::995::Storage.TaskManager.Task::(_decref) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,685::task::1173::Storage.TaskManager.Task::(prepare) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::1178::Storage.TaskManager.Task::(prepare) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::995::Storage.TaskManager.Task::(_decref) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::930::Storage.TaskManager.Task::(_doAbort) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::597::Storage.TaskManager.Task::(_updateState) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,685::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::task::597::Storage.TaskManager.Task::(_updateState) Task=`930a9ded-adab-4cc4-9daf-2e5138ee3c51`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:11,686::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:11,687::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:11,687::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,716::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'0443d23d-8d75-41d5-8c16-d6387ff98919'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,716::task::597::Storage.TaskManager.Task::(_updateState) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,717::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'0443d23d-8d75-41d5-8c16-d6387ff98919', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,717::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1f67011c-a47a-490b-9edb-4fc357bfb175`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,717::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,717::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,717::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`1f67011c-a47a-490b-9edb-4fc357bfb175`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,718::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,718::task::995::Storage.TaskManager.Task::(_decref) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,718::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,718::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,718::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,742::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:11,743::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,748::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,748::task::868::Storage.TaskManager.Task::(_setError) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::887::Storage.TaskManager.Task::(_run) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::Task._run: 36216e4a-d120-4c58-b77f-fb55fbc34406 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'0443d23d-8d75-41d5-8c16-d6387ff98919') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::1248::Storage.TaskManager.Task::(stop) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::995::Storage.TaskManager.Task::(_decref) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,749::task::1173::Storage.TaskManager.Task::(prepare) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::1178::Storage.TaskManager.Task::(prepare) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::995::Storage.TaskManager.Task::(_decref) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,749::task::930::Storage.TaskManager.Task::(_doAbort) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::task::597::Storage.TaskManager.Task::(_updateState) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::task::597::Storage.TaskManager.Task::(_updateState) Task=`36216e4a-d120-4c58-b77f-fb55fbc34406`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,750::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:11,751::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:11,751::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:11,751::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,780::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'77e78181-ff69-4036-b662-8d453290ff9b'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,781::task::597::Storage.TaskManager.Task::(_updateState) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,781::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'77e78181-ff69-4036-b662-8d453290ff9b', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,781::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`fb565b93-91c1-4a8d-911d-ba5b59a6dba5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,781::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,781::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,782::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`fb565b93-91c1-4a8d-911d-ba5b59a6dba5`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,782::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,782::task::995::Storage.TaskManager.Task::(_decref) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,782::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,782::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,782::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,806::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:11,807::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,812::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,812::task::868::Storage.TaskManager.Task::(_setError) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::887::Storage.TaskManager.Task::(_run) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::Task._run: 3b5456d7-7202-4f0a-bc61-3fa534cb03b8 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'77e78181-ff69-4036-b662-8d453290ff9b') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::1248::Storage.TaskManager.Task::(stop) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::995::Storage.TaskManager.Task::(_decref) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,813::task::1173::Storage.TaskManager.Task::(prepare) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::1178::Storage.TaskManager.Task::(prepare) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::995::Storage.TaskManager.Task::(_decref) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::task::930::Storage.TaskManager.Task::(_doAbort) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,813::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::task::597::Storage.TaskManager.Task::(_updateState) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::task::597::Storage.TaskManager.Task::(_updateState) Task=`3b5456d7-7202-4f0a-bc61-3fa534cb03b8`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,814::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:11,815::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:11,815::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:11,815::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,844::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'cebeb179-9776-47de-b890-1b71bb1cbd28'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,845::task::597::Storage.TaskManager.Task::(_updateState) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,845::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'cebeb179-9776-47de-b890-1b71bb1cbd28', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,845::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`7b56b059-bb4f-4e7e-993f-14497784db4f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,845::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,846::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,846::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`7b56b059-bb4f-4e7e-993f-14497784db4f`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,846::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,846::task::995::Storage.TaskManager.Task::(_decref) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,846::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,846::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,847::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,870::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:11,871::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,876::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,877::task::868::Storage.TaskManager.Task::(_setError) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,877::task::887::Storage.TaskManager.Task::(_run) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::Task._run: 2ec898c3-f3d1-4a99-9af7-d15b09415ef5 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'cebeb179-9776-47de-b890-1b71bb1cbd28') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,877::task::1248::Storage.TaskManager.Task::(stop) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,877::task::995::Storage.TaskManager.Task::(_decref) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,877::task::1173::Storage.TaskManager.Task::(prepare) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,877::task::1178::Storage.TaskManager.Task::(prepare) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::task::995::Storage.TaskManager.Task::(_decref) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::task::930::Storage.TaskManager.Task::(_doAbort) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::task::597::Storage.TaskManager.Task::(_updateState) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::task::597::Storage.TaskManager.Task::(_updateState) Task=`2ec898c3-f3d1-4a99-9af7-d15b09415ef5`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,878::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,879::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,879::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:11,879::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:11,879::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:11,879::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,908::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'3bcc55d1-2ff5-4814-9d11-1b43f0c615d8'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,909::task::597::Storage.TaskManager.Task::(_updateState) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,909::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'3bcc55d1-2ff5-4814-9d11-1b43f0c615d8', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,909::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`902b79f1-b3e0-42a3-aac0-451e91c7f38b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,909::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,910::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,910::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`902b79f1-b3e0-42a3-aac0-451e91c7f38b`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,910::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,910::task::995::Storage.TaskManager.Task::(_decref) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,910::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,910::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,911::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,934::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:11,935::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,940::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,941::task::868::Storage.TaskManager.Task::(_setError) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,941::task::887::Storage.TaskManager.Task::(_run) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::Task._run: d175e3db-fafe-4f62-b77a-0795737ac860 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'3bcc55d1-2ff5-4814-9d11-1b43f0c615d8') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,941::task::1248::Storage.TaskManager.Task::(stop) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,941::task::995::Storage.TaskManager.Task::(_decref) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,941::task::1173::Storage.TaskManager.Task::(prepare) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,941::task::1178::Storage.TaskManager.Task::(prepare) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::task::995::Storage.TaskManager.Task::(_decref) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::task::930::Storage.TaskManager.Task::(_doAbort) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::task::597::Storage.TaskManager.Task::(_updateState) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::task::597::Storage.TaskManager.Task::(_updateState) Task=`d175e3db-fafe-4f62-b77a-0795737ac860`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,942::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,943::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,943::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:11,943::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:11,943::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:11,943::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,972::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'30ab0f9d-afbe-44f1-a805-4a7732c7966d'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,973::task::597::Storage.TaskManager.Task::(_updateState) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:11,973::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'30ab0f9d-afbe-44f1-a805-4a7732c7966d', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,973::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`716e3c75-a86a-4e2c-bd67-90d25ef60dd0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,974::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,974::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,974::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`716e3c75-a86a-4e2c-bd67-90d25ef60dd0`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,974::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,974::task::995::Storage.TaskManager.Task::(_decref) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,974::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:11,975::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,975::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:11,999::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:12,000::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,005::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,006::task::868::Storage.TaskManager.Task::(_setError) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,006::task::887::Storage.TaskManager.Task::(_run) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::Task._run: 6c7e695a-fb67-46f6-8b04-716f8bfd3797 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'30ab0f9d-afbe-44f1-a805-4a7732c7966d') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,006::task::1248::Storage.TaskManager.Task::(stop) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,006::task::995::Storage.TaskManager.Task::(_decref) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:12,006::task::1173::Storage.TaskManager.Task::(prepare) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,006::task::1178::Storage.TaskManager.Task::(prepare) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::task::995::Storage.TaskManager.Task::(_decref) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::task::930::Storage.TaskManager.Task::(_doAbort) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::task::597::Storage.TaskManager.Task::(_updateState) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::task::597::Storage.TaskManager.Task::(_updateState) Task=`6c7e695a-fb67-46f6-8b04-716f8bfd3797`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,007::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,008::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,008::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,008::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,008::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:12,008::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,036::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'7c0abcc7-b269-4a75-b010-c94c07b048cd'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,037::task::597::Storage.TaskManager.Task::(_updateState) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,037::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'7c0abcc7-b269-4a75-b010-c94c07b048cd', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,038::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0f837602-f799-4e47-9750-7de8b0a57bb8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,038::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,038::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,038::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`0f837602-f799-4e47-9750-7de8b0a57bb8`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,038::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,039::task::995::Storage.TaskManager.Task::(_decref) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,039::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,039::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,039::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,063::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:12,064::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,069::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,069::task::868::Storage.TaskManager.Task::(_setError) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,069::task::887::Storage.TaskManager.Task::(_run) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::Task._run: 7166e8a5-1bfd-496c-aa22-d71eeefca738 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'7c0abcc7-b269-4a75-b010-c94c07b048cd') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::1248::Storage.TaskManager.Task::(stop) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::995::Storage.TaskManager.Task::(_decref) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,070::task::1173::Storage.TaskManager.Task::(prepare) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::1178::Storage.TaskManager.Task::(prepare) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::995::Storage.TaskManager.Task::(_decref) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::930::Storage.TaskManager.Task::(_doAbort) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::597::Storage.TaskManager.Task::(_updateState) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,070::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::task::597::Storage.TaskManager.Task::(_updateState) Task=`7166e8a5-1bfd-496c-aa22-d71eeefca738`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,071::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,071::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,072::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,101::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'63e1c7ad-1c92-4d55-9c23-d0d51cc36139'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,101::task::597::Storage.TaskManager.Task::(_updateState) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,101::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'63e1c7ad-1c92-4d55-9c23-d0d51cc36139', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,102::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`dc0ac292-7b23-4eda-87b7-7e29abf51fca`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,102::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,102::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,102::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`dc0ac292-7b23-4eda-87b7-7e29abf51fca`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,103::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,103::task::995::Storage.TaskManager.Task::(_decref) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,103::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,103::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,103::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,127::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:12,128::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,133::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,133::task::868::Storage.TaskManager.Task::(_setError) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,133::task::887::Storage.TaskManager.Task::(_run) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::Task._run: 699c0f21-2dd5-42e6-aa18-10c91191f5df (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'63e1c7ad-1c92-4d55-9c23-d0d51cc36139') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::1248::Storage.TaskManager.Task::(stop) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::995::Storage.TaskManager.Task::(_decref) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,134::task::1173::Storage.TaskManager.Task::(prepare) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::1178::Storage.TaskManager.Task::(prepare) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::995::Storage.TaskManager.Task::(_decref) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::930::Storage.TaskManager.Task::(_doAbort) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::597::Storage.TaskManager.Task::(_updateState) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,134::task::597::Storage.TaskManager.Task::(_updateState) Task=`699c0f21-2dd5-42e6-aa18-10c91191f5df`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,135::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,135::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,136::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,165::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'4674c630-3549-4ac5-b7da-c6c3e5eef6d3'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,165::task::597::Storage.TaskManager.Task::(_updateState) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,166::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'4674c630-3549-4ac5-b7da-c6c3e5eef6d3', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,166::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`692b09e3-2637-469f-ab77-3ab8831d58ee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,166::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,166::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,166::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`692b09e3-2637-469f-ab77-3ab8831d58ee`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,167::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,167::task::995::Storage.TaskManager.Task::(_decref) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,167::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,167::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,167::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,191::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:12,192::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,197::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,197::task::868::Storage.TaskManager.Task::(_setError) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,197::task::887::Storage.TaskManager.Task::(_run) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::Task._run: 2fbe0c86-cf80-4a64-8289-02fe2106bb2a (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'4674c630-3549-4ac5-b7da-c6c3e5eef6d3') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::1248::Storage.TaskManager.Task::(stop) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::995::Storage.TaskManager.Task::(_decref) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,198::task::1173::Storage.TaskManager.Task::(prepare) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::1178::Storage.TaskManager.Task::(prepare) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::995::Storage.TaskManager.Task::(_decref) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::930::Storage.TaskManager.Task::(_doAbort) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::597::Storage.TaskManager.Task::(_updateState) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,198::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::task::597::Storage.TaskManager.Task::(_updateState) Task=`2fbe0c86-cf80-4a64-8289-02fe2106bb2a`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,199::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,200::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,200::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,229::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'b58304ce-c2b4-4469-a5ec-1485559ce4ab'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,230::task::597::Storage.TaskManager.Task::(_updateState) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,230::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'b58304ce-c2b4-4469-a5ec-1485559ce4ab', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,230::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`4824e37d-f129-49f1-b0ce-6893decae3ca`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,230::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,230::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,230::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`4824e37d-f129-49f1-b0ce-6893decae3ca`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,231::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,231::task::995::Storage.TaskManager.Task::(_decref) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,231::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,231::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,231::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,255::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:12,256::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,261::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,261::task::868::Storage.TaskManager.Task::(_setError) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::887::Storage.TaskManager.Task::(_run) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::Task._run: fd528d83-da8e-48ce-af36-abcda7490f3c (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'b58304ce-c2b4-4469-a5ec-1485559ce4ab') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::1248::Storage.TaskManager.Task::(stop) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::995::Storage.TaskManager.Task::(_decref) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,262::task::1173::Storage.TaskManager.Task::(prepare) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::1178::Storage.TaskManager.Task::(prepare) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::995::Storage.TaskManager.Task::(_decref) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::task::930::Storage.TaskManager.Task::(_doAbort) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,262::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::task::597::Storage.TaskManager.Task::(_updateState) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::task::597::Storage.TaskManager.Task::(_updateState) Task=`fd528d83-da8e-48ce-af36-abcda7490f3c`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,263::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,264::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,264::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,264::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,293::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'7012e0c4-ae52-4e19-acaa-718a5cf7ad3d'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,294::task::597::Storage.TaskManager.Task::(_updateState) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,294::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'7012e0c4-ae52-4e19-acaa-718a5cf7ad3d', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,294::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8398bb6f-0583-48cd-afc6-00b058375bbf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,294::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,294::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,295::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8398bb6f-0583-48cd-afc6-00b058375bbf`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,295::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,295::task::995::Storage.TaskManager.Task::(_decref) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,295::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,295::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,296::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,319::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:12,320::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,325::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,325::task::868::Storage.TaskManager.Task::(_setError) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::887::Storage.TaskManager.Task::(_run) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::Task._run: 569621ef-b860-4b3e-9ebf-94f6ea76c630 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'7012e0c4-ae52-4e19-acaa-718a5cf7ad3d') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::1248::Storage.TaskManager.Task::(stop) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::995::Storage.TaskManager.Task::(_decref) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,326::task::1173::Storage.TaskManager.Task::(prepare) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::1178::Storage.TaskManager.Task::(prepare) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::995::Storage.TaskManager.Task::(_decref) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::task::930::Storage.TaskManager.Task::(_doAbort) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,326::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::task::597::Storage.TaskManager.Task::(_updateState) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::task::597::Storage.TaskManager.Task::(_updateState) Task=`569621ef-b860-4b3e-9ebf-94f6ea76c630`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,327::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,328::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,328::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,328::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,357::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'9546ec82-38e4-43e9-9628-32f5adabec63'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,358::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,358::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'9546ec82-38e4-43e9-9628-32f5adabec63', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,358::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`5ecc13f6-3b66-4d69-a10d-1d722250e662`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,358::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,359::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,359::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`5ecc13f6-3b66-4d69-a10d-1d722250e662`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,359::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,359::task::995::Storage.TaskManager.Task::(_decref) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,359::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,359::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,360::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,384::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:12,385::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,389::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,390::task::868::Storage.TaskManager.Task::(_setError) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,390::task::887::Storage.TaskManager.Task::(_run) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::Task._run: b7dde5b0-47f7-49b4-af93-1bd8e43d4aba (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'9546ec82-38e4-43e9-9628-32f5adabec63') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,390::task::1248::Storage.TaskManager.Task::(stop) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,390::task::995::Storage.TaskManager.Task::(_decref) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,390::task::1173::Storage.TaskManager.Task::(prepare) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,390::task::1178::Storage.TaskManager.Task::(prepare) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,390::task::995::Storage.TaskManager.Task::(_decref) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::task::930::Storage.TaskManager.Task::(_doAbort) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::task::597::Storage.TaskManager.Task::(_updateState) Task=`b7dde5b0-47f7-49b4-af93-1bd8e43d4aba`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,391::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,392::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,392::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,392::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,392::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,392::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,421::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'54519836-932e-42ef-8748-8c979655c6a7'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,422::task::597::Storage.TaskManager.Task::(_updateState) Task=`46ae876d-178d-4669-a234-d6b06d353189`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,422::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'54519836-932e-42ef-8748-8c979655c6a7', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f2e11dea-6749-4d2b-a7a7-6751ff234417`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`f2e11dea-6749-4d2b-a7a7-6751ff234417`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`46ae876d-178d-4669-a234-d6b06d353189`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,423::task::995::Storage.TaskManager.Task::(_decref) Task=`46ae876d-178d-4669-a234-d6b06d353189`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,424::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,424::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,424::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,448::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:12,449::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,454::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,454::task::868::Storage.TaskManager.Task::(_setError) Task=`46ae876d-178d-4669-a234-d6b06d353189`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,454::task::887::Storage.TaskManager.Task::(_run) Task=`46ae876d-178d-4669-a234-d6b06d353189`::Task._run: 46ae876d-178d-4669-a234-d6b06d353189 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'54519836-932e-42ef-8748-8c979655c6a7') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,454::task::1248::Storage.TaskManager.Task::(stop) Task=`46ae876d-178d-4669-a234-d6b06d353189`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::995::Storage.TaskManager.Task::(_decref) Task=`46ae876d-178d-4669-a234-d6b06d353189`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,455::task::1173::Storage.TaskManager.Task::(prepare) Task=`46ae876d-178d-4669-a234-d6b06d353189`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::1178::Storage.TaskManager.Task::(prepare) Task=`46ae876d-178d-4669-a234-d6b06d353189`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::995::Storage.TaskManager.Task::(_decref) Task=`46ae876d-178d-4669-a234-d6b06d353189`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::930::Storage.TaskManager.Task::(_doAbort) Task=`46ae876d-178d-4669-a234-d6b06d353189`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::597::Storage.TaskManager.Task::(_updateState) Task=`46ae876d-178d-4669-a234-d6b06d353189`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`46ae876d-178d-4669-a234-d6b06d353189`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::task::597::Storage.TaskManager.Task::(_updateState) Task=`46ae876d-178d-4669-a234-d6b06d353189`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,455::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,456::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,456::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,456::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,456::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,456::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,456::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,457::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,486::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'cc613941-506c-4438-844d-9c4030f49c5e'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,486::task::597::Storage.TaskManager.Task::(_updateState) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:12,486::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'cc613941-506c-4438-844d-9c4030f49c5e', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,487::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`2605c066-bc89-4563-9ed2-3b43cca4c900`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,487::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,487::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,487::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`2605c066-bc89-4563-9ed2-3b43cca4c900`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,488::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,488::task::995::Storage.TaskManager.Task::(_decref) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,488::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,488::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,488::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,512::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:12,513::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,518::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,518::task::868::Storage.TaskManager.Task::(_setError) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::887::Storage.TaskManager.Task::(_run) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::Task._run: 75c0b021-02df-4791-9ed6-52b3b5d700df (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'cc613941-506c-4438-844d-9c4030f49c5e') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::1248::Storage.TaskManager.Task::(stop) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::995::Storage.TaskManager.Task::(_decref) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:12,519::task::1173::Storage.TaskManager.Task::(prepare) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::1178::Storage.TaskManager.Task::(prepare) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::995::Storage.TaskManager.Task::(_decref) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::task::930::Storage.TaskManager.Task::(_doAbort) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,519::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::task::597::Storage.TaskManager.Task::(_updateState) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::task::597::Storage.TaskManager.Task::(_updateState) Task=`75c0b021-02df-4791-9ed6-52b3b5d700df`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,520::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,521::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:12,521::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:12,521::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,550::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'8bd51541-68f8-4299-a054-ba3bae8fcd5c'} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,550::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::moving from state init -> state preparing jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,551::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'8bd51541-68f8-4299-a054-ba3bae8fcd5c', options=None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,551::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eda73c4f-5f69-4414-b73a-94ca97e58bed`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,551::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,551::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,551::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`eda73c4f-5f69-4414-b73a-94ca97e58bed`::Granted request jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,552::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,552::task::995::Storage.TaskManager.Task::(_decref) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::ref 1 aborting False jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,552::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,552::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,552::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,576::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/6::WARNING::2017-02-19 13:40:12,577::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,582::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,582::task::868::Storage.TaskManager.Task::(_setError) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::887::Storage.TaskManager.Task::(_run) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::Task._run: f83e3aae-f783-4704-b7e6-91e2bbe8219f (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'8bd51541-68f8-4299-a054-ba3bae8fcd5c') {} failed - stopping task jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::1248::Storage.TaskManager.Task::(stop) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::stopping in state preparing (force False) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::995::Storage.TaskManager.Task::(_decref) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::ref 1 aborting True jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,583::task::1173::Storage.TaskManager.Task::(prepare) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::1178::Storage.TaskManager.Task::(prepare) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::995::Storage.TaskManager.Task::(_decref) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::ref 0 aborting True jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::task::930::Storage.TaskManager.Task::(_doAbort) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::Task._doAbort: force False jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,583::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::moving from state preparing -> state aborting jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::_aborting: recover policy none jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::task::597::Storage.TaskManager.Task::(_updateState) Task=`f83e3aae-f783-4704-b7e6-91e2bbe8219f`::moving from state aborting -> state failed jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,584::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:12,585::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/6::ERROR::2017-02-19 13:40:12,585::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/6::INFO::2017-02-19 13:40:12,585::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,614::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'a380d17e-eb9c-4fdd-910c-a53b5de2b55f'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,615::task::597::Storage.TaskManager.Task::(_updateState) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,615::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'a380d17e-eb9c-4fdd-910c-a53b5de2b55f', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,615::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`476b217b-3671-4692-a5fe-7a2c3bc065c8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,615::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,615::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,616::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`476b217b-3671-4692-a5fe-7a2c3bc065c8`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,616::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,616::task::995::Storage.TaskManager.Task::(_decref) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,616::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,616::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,616::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,640::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:12,641::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,646::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,646::task::868::Storage.TaskManager.Task::(_setError) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::887::Storage.TaskManager.Task::(_run) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::Task._run: 24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'a380d17e-eb9c-4fdd-910c-a53b5de2b55f') {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::1248::Storage.TaskManager.Task::(stop) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::995::Storage.TaskManager.Task::(_decref) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,647::task::1173::Storage.TaskManager.Task::(prepare) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::1178::Storage.TaskManager.Task::(prepare) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::995::Storage.TaskManager.Task::(_decref) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::930::Storage.TaskManager.Task::(_doAbort) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,647::task::597::Storage.TaskManager.Task::(_updateState) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::task::597::Storage.TaskManager.Task::(_updateState) Task=`24d4e5a9-90b3-411a-a3cf-9e3f9caf3e78`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,648::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:12,649::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:12,649::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:12,649::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,678::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'05fbcafc-caee-4b19-9465-56fcd695ac5d'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,679::task::597::Storage.TaskManager.Task::(_updateState) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,679::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'05fbcafc-caee-4b19-9465-56fcd695ac5d', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,679::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`730072f5-6dfb-41e1-a83a-fe7507811903`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,679::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,680::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,680::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`730072f5-6dfb-41e1-a83a-fe7507811903`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,680::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,680::task::995::Storage.TaskManager.Task::(_decref) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,680::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,680::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,681::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,705::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:12,706::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,710::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,711::task::868::Storage.TaskManager.Task::(_setError) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,711::task::887::Storage.TaskManager.Task::(_run) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::Task._run: 14465a72-0ee8-4f36-a1dc-6ef00c269fbb (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'05fbcafc-caee-4b19-9465-56fcd695ac5d') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,711::task::1248::Storage.TaskManager.Task::(stop) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,711::task::995::Storage.TaskManager.Task::(_decref) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,711::task::1173::Storage.TaskManager.Task::(prepare) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,711::task::1178::Storage.TaskManager.Task::(prepare) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::task::995::Storage.TaskManager.Task::(_decref) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::task::930::Storage.TaskManager.Task::(_doAbort) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::task::597::Storage.TaskManager.Task::(_updateState) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::task::597::Storage.TaskManager.Task::(_updateState) Task=`14465a72-0ee8-4f36-a1dc-6ef00c269fbb`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,712::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,713::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,713::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:12,713::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:12,713::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:12,713::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,742::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'5792b645-a556-4402-a3ac-7a881bfaed8f'} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,743::task::597::Storage.TaskManager.Task::(_updateState) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::moving from state init -> state preparing jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,743::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'5792b645-a556-4402-a3ac-7a881bfaed8f', options=None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,743::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8b670aa5-cb79-498e-a18c-ca04fadc0102`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,744::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,744::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,744::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`8b670aa5-cb79-498e-a18c-ca04fadc0102`::Granted request jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,744::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,744::task::995::Storage.TaskManager.Task::(_decref) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::ref 1 aborting False jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,744::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,745::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,745::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,769::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/1::WARNING::2017-02-19 13:40:12,770::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,774::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,775::task::868::Storage.TaskManager.Task::(_setError) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,775::task::887::Storage.TaskManager.Task::(_run) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::Task._run: 41821bba-48f8-4db3-a4df-b4fa7744fff3 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'5792b645-a556-4402-a3ac-7a881bfaed8f') {} failed - stopping task jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,775::task::1248::Storage.TaskManager.Task::(stop) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::stopping in state preparing (force False) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,775::task::995::Storage.TaskManager.Task::(_decref) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::ref 1 aborting True jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,775::task::1173::Storage.TaskManager.Task::(prepare) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,775::task::1178::Storage.TaskManager.Task::(prepare) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::task::995::Storage.TaskManager.Task::(_decref) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::ref 0 aborting True jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::task::930::Storage.TaskManager.Task::(_doAbort) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::Task._doAbort: force False jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::task::597::Storage.TaskManager.Task::(_updateState) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::moving from state preparing -> state aborting jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::_aborting: recover policy none jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::task::597::Storage.TaskManager.Task::(_updateState) Task=`41821bba-48f8-4db3-a4df-b4fa7744fff3`::moving from state aborting -> state failed jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,776::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,777::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,777::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:12,777::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/1::ERROR::2017-02-19 13:40:12,777::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/1::INFO::2017-02-19 13:40:12,777::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,806::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'a2b42adb-eed0-4de4-80bd-426b29ac05ec'} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,807::task::597::Storage.TaskManager.Task::(_updateState) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::moving from state init -> state preparing jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,807::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'a2b42adb-eed0-4de4-80bd-426b29ac05ec', options=None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`2a6cd2e9-5014-4968-a3bc-6763b4a60746`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`2a6cd2e9-5014-4968-a3bc-6763b4a60746`::Granted request jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,808::task::995::Storage.TaskManager.Task::(_decref) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::ref 1 aborting False jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,809::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,809::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,809::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,833::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/2::WARNING::2017-02-19 13:40:12,834::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,838::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,839::task::868::Storage.TaskManager.Task::(_setError) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,839::task::887::Storage.TaskManager.Task::(_run) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::Task._run: 3e518359-0af2-416f-8ac6-cfda685eedea (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'a2b42adb-eed0-4de4-80bd-426b29ac05ec') {} failed - stopping task jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,839::task::1248::Storage.TaskManager.Task::(stop) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::stopping in state preparing (force False) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,839::task::995::Storage.TaskManager.Task::(_decref) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::ref 1 aborting True jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,839::task::1173::Storage.TaskManager.Task::(prepare) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,839::task::1178::Storage.TaskManager.Task::(prepare) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::task::995::Storage.TaskManager.Task::(_decref) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::ref 0 aborting True jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::task::930::Storage.TaskManager.Task::(_doAbort) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::Task._doAbort: force False jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::task::597::Storage.TaskManager.Task::(_updateState) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::moving from state preparing -> state aborting jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::_aborting: recover policy none jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::task::597::Storage.TaskManager.Task::(_updateState) Task=`3e518359-0af2-416f-8ac6-cfda685eedea`::moving from state aborting -> state failed jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,840::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,841::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,841::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:12,841::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/2::ERROR::2017-02-19 13:40:12,841::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/2::INFO::2017-02-19 13:40:12,841::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,871::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'f983ab6c-92c6-40a3-b06c-9e775ae3a16e'} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,871::task::597::Storage.TaskManager.Task::(_updateState) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::moving from state init -> state preparing jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,871::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'f983ab6c-92c6-40a3-b06c-9e775ae3a16e', options=None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,872::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`55c801f3-d68f-467a-9009-0a158b59d960`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,872::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,872::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,872::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`55c801f3-d68f-467a-9009-0a158b59d960`::Granted request jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,873::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,873::task::995::Storage.TaskManager.Task::(_decref) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::ref 1 aborting False jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,873::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,873::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,873::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,897::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/3::WARNING::2017-02-19 13:40:12,898::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,903::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,903::task::868::Storage.TaskManager.Task::(_setError) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::887::Storage.TaskManager.Task::(_run) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::Task._run: 91e782bb-0945-4570-a371-89f0b8704efb (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'f983ab6c-92c6-40a3-b06c-9e775ae3a16e') {} failed - stopping task jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::1248::Storage.TaskManager.Task::(stop) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::stopping in state preparing (force False) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::995::Storage.TaskManager.Task::(_decref) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::ref 1 aborting True jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,904::task::1173::Storage.TaskManager.Task::(prepare) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::1178::Storage.TaskManager.Task::(prepare) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::995::Storage.TaskManager.Task::(_decref) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::ref 0 aborting True jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::task::930::Storage.TaskManager.Task::(_doAbort) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::Task._doAbort: force False jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,904::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::task::597::Storage.TaskManager.Task::(_updateState) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::moving from state preparing -> state aborting jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::_aborting: recover policy none jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::task::597::Storage.TaskManager.Task::(_updateState) Task=`91e782bb-0945-4570-a371-89f0b8704efb`::moving from state aborting -> state failed jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,905::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:12,906::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/3::ERROR::2017-02-19 13:40:12,906::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/3::INFO::2017-02-19 13:40:12,906::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,935::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'4daa28fc-6590-436b-a12b-e3515dc28fa8'} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,935::task::597::Storage.TaskManager.Task::(_updateState) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::moving from state init -> state preparing jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,936::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'4daa28fc-6590-436b-a12b-e3515dc28fa8', options=None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,936::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`c4772061-aa44-44d5-a352-3fe44b6e6d24`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,936::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,936::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,936::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`c4772061-aa44-44d5-a352-3fe44b6e6d24`::Granted request jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,937::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,937::task::995::Storage.TaskManager.Task::(_decref) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::ref 1 aborting False jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,937::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,937::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,937::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,961::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/4::WARNING::2017-02-19 13:40:12,962::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,967::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,967::task::868::Storage.TaskManager.Task::(_setError) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,967::task::887::Storage.TaskManager.Task::(_run) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::Task._run: 91a7d98d-c200-4698-9ca0-2e829f3ea905 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'4daa28fc-6590-436b-a12b-e3515dc28fa8') {} failed - stopping task jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::1248::Storage.TaskManager.Task::(stop) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::stopping in state preparing (force False) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::995::Storage.TaskManager.Task::(_decref) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::ref 1 aborting True jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,968::task::1173::Storage.TaskManager.Task::(prepare) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::1178::Storage.TaskManager.Task::(prepare) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::995::Storage.TaskManager.Task::(_decref) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::ref 0 aborting True jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::930::Storage.TaskManager.Task::(_doAbort) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::Task._doAbort: force False jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,968::task::597::Storage.TaskManager.Task::(_updateState) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::moving from state preparing -> state aborting jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::_aborting: recover policy none jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::task::597::Storage.TaskManager.Task::(_updateState) Task=`91a7d98d-c200-4698-9ca0-2e829f3ea905`::moving from state aborting -> state failed jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/4::DEBUG::2017-02-19 13:40:12,969::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/4::ERROR::2017-02-19 13:40:12,970::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/4::INFO::2017-02-19 13:40:12,970::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.03 seconds jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:12,999::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'eaf3e550-3f4e-4b50-8c1b-c309e695a09d'} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,000::task::597::Storage.TaskManager.Task::(_updateState) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::moving from state init -> state preparing jsonrpc.Executor/5::INFO::2017-02-19 13:40:13,000::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'eaf3e550-3f4e-4b50-8c1b-c309e695a09d', options=None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,000::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cf557abc-f733-4a7a-9939-bb167fea5c52`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,000::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,000::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,001::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`cf557abc-f733-4a7a-9939-bb167fea5c52`::Granted request jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,001::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,001::task::995::Storage.TaskManager.Task::(_decref) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::ref 1 aborting False jsonrpc.Executor/5::ERROR::2017-02-19 13:40:13,001::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::ERROR::2017-02-19 13:40:13,001::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,001::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,026::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/5::WARNING::2017-02-19 13:40:13,027::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/5::ERROR::2017-02-19 13:40:13,031::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::ERROR::2017-02-19 13:40:13,032::task::868::Storage.TaskManager.Task::(_setError) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,032::task::887::Storage.TaskManager.Task::(_run) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::Task._run: 132189bf-d50c-436c-97d6-0dea83d0a1f2 (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'eaf3e550-3f4e-4b50-8c1b-c309e695a09d') {} failed - stopping task jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,032::task::1248::Storage.TaskManager.Task::(stop) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::stopping in state preparing (force False) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,032::task::995::Storage.TaskManager.Task::(_decref) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::ref 1 aborting True jsonrpc.Executor/5::INFO::2017-02-19 13:40:13,032::task::1173::Storage.TaskManager.Task::(prepare) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,032::task::1178::Storage.TaskManager.Task::(prepare) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::task::995::Storage.TaskManager.Task::(_decref) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::ref 0 aborting True jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::task::930::Storage.TaskManager.Task::(_doAbort) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::Task._doAbort: force False jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::task::597::Storage.TaskManager.Task::(_updateState) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::moving from state preparing -> state aborting jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::_aborting: recover policy none jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::task::597::Storage.TaskManager.Task::(_updateState) Task=`132189bf-d50c-436c-97d6-0dea83d0a1f2`::moving from state aborting -> state failed jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,033::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,034::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,034::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/5::DEBUG::2017-02-19 13:40:13,034::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/5::ERROR::2017-02-19 13:40:13,034::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/5::INFO::2017-02-19 13:40:13,034::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds JsonRpc (StompReactor)::ERROR::2017-02-19 13:40:13,062::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof Reactor thread::INFO::2017-02-19 13:40:15,067::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52842 Reactor thread::DEBUG::2017-02-19 13:40:15,073::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:15,074::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol stomp from ::1:52842 Reactor thread::INFO::2017-02-19 13:40:15,074::stompreactor::101::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request Reactor thread::DEBUG::2017-02-19 13:40:15,075::stompreactor::492::protocoldetector.StompDetector::(handle_socket) Stomp detected from ('::1', 52842) JsonRpc (StompReactor)::INFO::2017-02-19 13:40:15,075::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe) Subscribe command received jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:15,292::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getHardwareInfo' in bridge with {} jsonrpc.Executor/6::DEBUG::2017-02-19 13:40:15,293::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'X9DBU', 'systemSerialNumber': '0123456789', 'systemFamily': 'To be filled by O.E.M.', 'systemVersion': '0123456789', 'systemUUID': '00000000-0000-0000-0000-0CC47A3C0510', 'systemManufacturer': 'Supermicro'} jsonrpc.Executor/6::INFO::2017-02-19 13:40:15,293::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getHardwareInfo succeeded in 0.01 seconds jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,295::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'StorageDomain.getImages' in bridge with {u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5'} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,296::task::597::Storage.TaskManager.Task::(_updateState) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::moving from state init -> state preparing jsonrpc.Executor/7::INFO::2017-02-19 13:40:15,296::logUtils::49::dispatcher::(wrapper) Run and protect: getImagesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', options=None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,296::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`22458998-a907-4c53-affb-84d0fc56285f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3216' at 'getImagesList' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,296::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,297::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,297::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`22458998-a907-4c53-affb-84d0fc56285f`::Granted request jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,297::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,297::task::995::Storage.TaskManager.Task::(_decref) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::ref 1 aborting False jsonrpc.Executor/7::ERROR::2017-02-19 13:40:15,297::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::ERROR::2017-02-19 13:40:15,297::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,298::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,323::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/7::WARNING::2017-02-19 13:40:15,324::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/7::ERROR::2017-02-19 13:40:15,328::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::ERROR::2017-02-19 13:40:15,329::task::868::Storage.TaskManager.Task::(_setError) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3217, in getImagesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,329::task::887::Storage.TaskManager.Task::(_run) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::Task._run: c2ff98cf-aac6-440f-824d-971dc57c4858 (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) {} failed - stopping task jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,329::task::1248::Storage.TaskManager.Task::(stop) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::stopping in state preparing (force False) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,329::task::995::Storage.TaskManager.Task::(_decref) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::ref 1 aborting True jsonrpc.Executor/7::INFO::2017-02-19 13:40:15,329::task::1173::Storage.TaskManager.Task::(prepare) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::1178::Storage.TaskManager.Task::(prepare) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::995::Storage.TaskManager.Task::(_decref) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::ref 0 aborting True jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::Task._doAbort: force False jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::597::Storage.TaskManager.Task::(_updateState) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::moving from state preparing -> state aborting jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::_aborting: recover policy none jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::task::597::Storage.TaskManager.Task::(_updateState) Task=`c2ff98cf-aac6-440f-824d-971dc57c4858`::moving from state aborting -> state failed jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,330::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,331::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,331::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,331::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/7::DEBUG::2017-02-19 13:40:15,331::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/7::ERROR::2017-02-19 13:40:15,331::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/7::INFO::2017-02-19 13:40:15,331::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StorageDomain.getImages failed (error 358) in 0.03 seconds jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,360::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Image.getVolumes' in bridge with {u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'storagedomainID': u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'imageID': u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356'} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,361::task::597::Storage.TaskManager.Task::(_updateState) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::moving from state init -> state preparing jsonrpc.Executor/0::INFO::2017-02-19 13:40:15,361::logUtils::49::dispatcher::(wrapper) Run and protect: getVolumesList(sdUUID=u'c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID=u'00000000-0000-0000-0000-000000000000', imgUUID=u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356', options=None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`436ef237-07c1-40ad-9509-a9cb8914c513`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3194' at 'getVolumesList' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`436ef237-07c1-40ad-9509-a9cb8914c513`::Granted request jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,362::task::995::Storage.TaskManager.Task::(_decref) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::ref 1 aborting False jsonrpc.Executor/0::ERROR::2017-02-19 13:40:15,362::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::ERROR::2017-02-19 13:40:15,363::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,363::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,387::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 jsonrpc.Executor/0::WARNING::2017-02-19 13:40:15,388::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] jsonrpc.Executor/0::ERROR::2017-02-19 13:40:15,392::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::ERROR::2017-02-19 13:40:15,393::task::868::Storage.TaskManager.Task::(_setError) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3195, in getVolumesList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,393::task::887::Storage.TaskManager.Task::(_run) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::Task._run: 1d574fe1-13c7-4f59-acb0-5ea98af7be5d (u'c7c466c5-983d-40b2-8f8c-d569755281f5', u'00000000-0000-0000-0000-000000000000', u'd8f49ab3-f3af-4cc5-9d57-76ccb8335356') {} failed - stopping task jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,393::task::1248::Storage.TaskManager.Task::(stop) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::stopping in state preparing (force False) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,393::task::995::Storage.TaskManager.Task::(_decref) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::ref 1 aborting True jsonrpc.Executor/0::INFO::2017-02-19 13:40:15,393::task::1173::Storage.TaskManager.Task::(prepare) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,393::task::1178::Storage.TaskManager.Task::(prepare) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::Prepare: aborted: Storage domain does not exist jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::task::995::Storage.TaskManager.Task::(_decref) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::ref 0 aborting True jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::task::930::Storage.TaskManager.Task::(_doAbort) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::Task._doAbort: force False jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::task::597::Storage.TaskManager.Task::(_updateState) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::moving from state preparing -> state aborting jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::_aborting: recover policy none jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::task::597::Storage.TaskManager.Task::(_updateState) Task=`1d574fe1-13c7-4f59-acb0-5ea98af7be5d`::moving from state aborting -> state failed jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,394::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,395::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,395::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,395::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. jsonrpc.Executor/0::DEBUG::2017-02-19 13:40:15,395::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. jsonrpc.Executor/0::ERROR::2017-02-19 13:40:15,395::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: (u'c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} jsonrpc.Executor/0::INFO::2017-02-19 13:40:15,395::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Image.getVolumes failed (error 358) in 0.04 seconds JsonRpc (StompReactor)::ERROR::2017-02-19 13:40:15,437::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof Reactor thread::INFO::2017-02-19 13:40:15,456::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52844 Reactor thread::INFO::2017-02-19 13:40:15,458::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52846 Reactor thread::DEBUG::2017-02-19 13:40:15,465::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::DEBUG::2017-02-19 13:40:15,470::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:15,471::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52848 Reactor thread::DEBUG::2017-02-19 13:40:15,476::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:15,476::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52844 Reactor thread::DEBUG::2017-02-19 13:40:15,477::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52844) Reactor thread::INFO::2017-02-19 13:40:15,477::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52846 Reactor thread::DEBUG::2017-02-19 13:40:15,477::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52846) Reactor thread::INFO::2017-02-19 13:40:15,477::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52850 Reactor thread::INFO::2017-02-19 13:40:15,477::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52848 Reactor thread::DEBUG::2017-02-19 13:40:15,477::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52848) BindingXMLRPC::INFO::2017-02-19 13:40:15,478::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52844 BindingXMLRPC::INFO::2017-02-19 13:40:15,478::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52846 Thread-601::INFO::2017-02-19 13:40:15,478::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52844 started Thread-601::DEBUG::2017-02-19 13:40:15,479::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getCapabilities with () {} Reactor thread::DEBUG::2017-02-19 13:40:15,483::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:15,483::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52852 Reactor thread::INFO::2017-02-19 13:40:15,484::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52850 Reactor thread::DEBUG::2017-02-19 13:40:15,484::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52850) Reactor thread::DEBUG::2017-02-19 13:40:15,489::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:15,490::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52852 Reactor thread::DEBUG::2017-02-19 13:40:15,490::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52852) BindingXMLRPC::INFO::2017-02-19 13:40:15,511::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52848 Thread-602::INFO::2017-02-19 13:40:15,512::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52846 started Thread-602::DEBUG::2017-02-19 13:40:15,512::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getStats with () {} Thread-602::DEBUG::2017-02-19 13:40:15,513::task::597::Storage.TaskManager.Task::(_updateState) Task=`bae1d9d5-4e85-492e-9770-90933711a94b`::moving from state init -> state preparing Thread-602::INFO::2017-02-19 13:40:15,513::logUtils::49::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-602::INFO::2017-02-19 13:40:15,513::logUtils::52::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-602::DEBUG::2017-02-19 13:40:15,513::task::1193::Storage.TaskManager.Task::(prepare) Task=`bae1d9d5-4e85-492e-9770-90933711a94b`::finished: {} Thread-602::DEBUG::2017-02-19 13:40:15,513::task::597::Storage.TaskManager.Task::(_updateState) Task=`bae1d9d5-4e85-492e-9770-90933711a94b`::moving from state preparing -> state finished Thread-602::DEBUG::2017-02-19 13:40:15,513::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-602::DEBUG::2017-02-19 13:40:15,513::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-602::DEBUG::2017-02-19 13:40:15,513::task::995::Storage.TaskManager.Task::(_decref) Task=`bae1d9d5-4e85-492e-9770-90933711a94b`::ref 0 aborting False BindingXMLRPC::INFO::2017-02-19 13:40:15,515::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52850 Thread-603::INFO::2017-02-19 13:40:15,515::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52848 started Thread-603::DEBUG::2017-02-19 13:40:15,516::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call vmGetStats with ('ea5cb711-45b5-4767-aa05-47fe36710d8a',) {} Thread-603::DEBUG::2017-02-19 13:40:15,516::bindingxmlrpc::1260::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-603::INFO::2017-02-19 13:40:15,516::bindingxmlrpc::1287::vds::(wrapper) RPC call vmGetStats finished (code=1) in 0.00 seconds Thread-603::INFO::2017-02-19 13:40:15,517::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52848 stopped BindingXMLRPC::INFO::2017-02-19 13:40:15,518::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52852 Thread-604::INFO::2017-02-19 13:40:15,521::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52850 started Thread-604::DEBUG::2017-02-19 13:40:15,521::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call vmGetStats with ('ea5cb711-45b5-4767-aa05-47fe36710d8a',) {} Thread-604::DEBUG::2017-02-19 13:40:15,521::bindingxmlrpc::1260::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-604::INFO::2017-02-19 13:40:15,521::bindingxmlrpc::1287::vds::(wrapper) RPC call vmGetStats finished (code=1) in 0.00 seconds Thread-605::INFO::2017-02-19 13:40:15,522::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52852 started Thread-604::INFO::2017-02-19 13:40:15,522::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52850 stopped Thread-605::DEBUG::2017-02-19 13:40:15,523::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-605::DEBUG::2017-02-19 13:40:15,524::task::597::Storage.TaskManager.Task::(_updateState) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::moving from state init -> state preparing Thread-605::INFO::2017-02-19 13:40:15,524::logUtils::49::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='4fc108de-838f-4acc-9688-c8d45739d19b', leafUUID='f25d5ae6-eaea-4132-a5b4-5765d8ac73f6', allowIllegal=False) Thread-605::DEBUG::2017-02-19 13:40:15,524::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`12df6400-00c1-4113-a6e1-3a8eb52a3055`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3113' at 'prepareImage' Thread-605::DEBUG::2017-02-19 13:40:15,525::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' Thread-605::DEBUG::2017-02-19 13:40:15,525::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) Thread-605::DEBUG::2017-02-19 13:40:15,525::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`12df6400-00c1-4113-a6e1-3a8eb52a3055`::Granted request Thread-605::DEBUG::2017-02-19 13:40:15,525::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) Thread-605::DEBUG::2017-02-19 13:40:15,526::task::995::Storage.TaskManager.Task::(_decref) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::ref 1 aborting False Thread-605::ERROR::2017-02-19 13:40:15,526::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-605::ERROR::2017-02-19 13:40:15,526::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-605::DEBUG::2017-02-19 13:40:15,526::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) Thread-605::DEBUG::2017-02-19 13:40:15,555::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 Thread-605::WARNING::2017-02-19 13:40:15,556::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-605::ERROR::2017-02-19 13:40:15,561::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-605::ERROR::2017-02-19 13:40:15,562::task::868::Storage.TaskManager.Task::(_setError) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3116, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-605::DEBUG::2017-02-19 13:40:15,562::task::887::Storage.TaskManager.Task::(_run) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::Task._run: 18619998-26fc-4daf-8ce0-54eaefb002ae ('c7c466c5-983d-40b2-8f8c-d569755281f5', '00000000-0000-0000-0000-000000000000', '4fc108de-838f-4acc-9688-c8d45739d19b', 'f25d5ae6-eaea-4132-a5b4-5765d8ac73f6') {'allowIllegal': False} failed - stopping task Thread-605::DEBUG::2017-02-19 13:40:15,562::task::1248::Storage.TaskManager.Task::(stop) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::stopping in state preparing (force False) Thread-605::DEBUG::2017-02-19 13:40:15,562::task::995::Storage.TaskManager.Task::(_decref) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::ref 1 aborting True Thread-605::INFO::2017-02-19 13:40:15,562::task::1173::Storage.TaskManager.Task::(prepare) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 Thread-605::DEBUG::2017-02-19 13:40:15,563::task::1178::Storage.TaskManager.Task::(prepare) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::Prepare: aborted: Storage domain does not exist Thread-605::DEBUG::2017-02-19 13:40:15,563::task::995::Storage.TaskManager.Task::(_decref) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::ref 0 aborting True Thread-605::DEBUG::2017-02-19 13:40:15,563::task::930::Storage.TaskManager.Task::(_doAbort) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::Task._doAbort: force False Thread-605::DEBUG::2017-02-19 13:40:15,563::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-605::DEBUG::2017-02-19 13:40:15,563::task::597::Storage.TaskManager.Task::(_updateState) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::moving from state preparing -> state aborting Thread-605::DEBUG::2017-02-19 13:40:15,563::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::_aborting: recover policy none Thread-605::DEBUG::2017-02-19 13:40:15,563::task::597::Storage.TaskManager.Task::(_updateState) Task=`18619998-26fc-4daf-8ce0-54eaefb002ae`::moving from state aborting -> state failed Thread-605::DEBUG::2017-02-19 13:40:15,563::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} Thread-605::DEBUG::2017-02-19 13:40:15,563::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-605::DEBUG::2017-02-19 13:40:15,563::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' Thread-605::DEBUG::2017-02-19 13:40:15,564::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) Thread-605::DEBUG::2017-02-19 13:40:15,564::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. Thread-605::DEBUG::2017-02-19 13:40:15,564::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. Thread-605::ERROR::2017-02-19 13:40:15,564::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} Thread-605::INFO::2017-02-19 13:40:15,564::bindingxmlrpc::331::vds::(wrapper) RPC call imagePrepare finished (code=358) in 0.05 seconds Thread-605::INFO::2017-02-19 13:40:15,565::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52852 stopped Thread-602::ERROR::2017-02-19 13:40:15,567::api::255::root::(_getHaInfo) failed to retrieve Hosted Engine HA info Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 234, in _getHaInfo stats = instance.get_all_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats self._configure_broker_conn(broker) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn dom_type=dom_type) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain .format(sd_type, options, e)) RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'iscsi', 'sd_uuid': 'c7c466c5-983d-40b2-8f8c-d569755281f5'}: Request failed: Thread-602::DEBUG::2017-02-19 13:40:15,568::bindingxmlrpc::1260::vds::(wrapper) return getStats with {'status': {'message': 'Done', 'code': 0}, 'info': {'memShared': 0, 'haScore': 0, 'ksmMergeAcrossNodes': True, 'rxRate': '0.00', 'vmCount': 0, 'memUsed': '0.00', 'storageDomains': {}, 'incomingVmMigrations': 0, 'anonHugePages': '0.00', 'ksmPages': 100, 'elapsedTime': '14.42', 'cpuLoad': '0.00', 'cpuSys': '0.00', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 63045, 'txRate': '0.00', 'cpuUserVdsmd': '0.00', 'haStats': {'active': False, 'configured': True, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'outgoingVmMigrations': 0, 'swapTotal': 19023, 'swapFree': 19023, 'dateTime': '2017-02-19T10:40:15 GMT', 'cpuUser': '0.00', 'memFree': 63301, 'cpuIdle': '100.00', 'vmActive': 0, 'cpuSysVdsmd': '0.00'}} Thread-602::INFO::2017-02-19 13:40:15,568::bindingxmlrpc::1287::vds::(wrapper) RPC call getStats finished (code=0) in 0.06 seconds Thread-602::INFO::2017-02-19 13:40:15,571::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52846 stopped Thread-601::DEBUG::2017-02-19 13:40:15,604::commands::68::root::(execCmd) /usr/bin/taskset --cpu-list 0-23 /sbin/sysctl -n -e kernel.numa_balancing (cwd None) Thread-601::DEBUG::2017-02-19 13:40:15,612::commands::86::root::(execCmd) SUCCESS: = ''; = 0 Thread-601::DEBUG::2017-02-19 13:40:15,623::bindingxmlrpc::1260::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2'}], 'FC': []}, 'packages2': {'kernel': {'release': '514.6.1.el7.x86_64', 'buildtime': 1484733996.0, 'version': '3.10.0'}, 'spice-server': {'release': '19.el7', 'buildtime': 1478932140L, 'version': '0.12.4'}, 'librbd1': {'release': '1.el7', 'buildtime': 1478375483L, 'version': '0.94.5'}, 'vdsm': {'release': '1.el7.centos', 'buildtime': 1481708591L, 'version': '4.18.21'}, 'qemu-kvm': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'libvirt': {'release': '10.el7_3.4', 'buildtime': 1484696570L, 'version': '2.0.0'}, 'qemu-img': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'mom': {'release': '1.el7.centos', 'buildtime': 1477300773L, 'version': '0.5.8'}, 'glusterfs-cli': {'release': '1.el7', 'buildtime': 1483963815L, 'version': '3.7.19'}}, 'numaNodeDistance': {'1': [21, 10], '0': [10, 21]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz', 'liveMerge': 'true', 'hooks': {'before_nic_hotplug': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}, 'before_vm_start': {'50_hostedengine': {'md5': '2a6d96c26a3599812be6cf1a13d9f485'}, '50_nestedvt': {'md5': '70009fa966ddf89b7f9ad60c286077ae'}, '50_macspoof': {'md5': '68cf48c446387550a9a82283079b80f8'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}}, 'before_device_create': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}}, 'supportsIPv6': True, 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'kdumpStatus': 1, 'networks': {'ovirtmgmt': {'ipv4addrs': ['192.168.3.12/24'], 'iface': 'ovirtmgmt', 'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'netmask': '255.255.255.0', 'mtu': '1500', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'stp': 'off', 'bridged': True, 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'ports': ['bond1']}, 'storage': {'ipv4addrs': ['172.16.0.3/28'], 'iface': 'bond0', 'ipv6autoconf': True, 'addr': '172.16.0.3', 'ipv6addrs': [], 'netmask': '255.255.255.240', 'mtu': '9000', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'bridged': False, 'interface': 'bond0', 'ipv6gateway': '::', 'gateway': ''}}, 'kernelArgs': 'BOOT_IMAGE=/vmlinuz-3.10.0-514.6.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=ru_RU.UTF-8', 'bridges': {'ovirtmgmt': {'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': ['192.168.3.12/24'], 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '143', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.0cc47a3c0510', 'bridge_id': '8000.0cc47a3c0510', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '4968', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['bond1']}}, 'uuid': '00000000-0000-0000-0000-0CC47A3C0510', 'onlineCpus': '0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', 'nameservers': ['192.168.3.16'], 'nics': {'eno1': {'permhwaddr': '0c:c4:7a:3c:05:10', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'eno2': {'permhwaddr': '0c:c4:7a:3c:05:11', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'enp5s0f0': {'permhwaddr': '0c:c4:7a:bb:9d:40', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f0', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}, 'enp5s0f1': {'permhwaddr': '0c:c4:7a:bb:9d:41', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}}, 'software_revision': '1', 'hostdevPassthrough': 'false', 'clusterLevels': ['3.5', '3.6', '4.0'], 'cpuFlags': 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,pln,pts,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,xsaveopt,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_IvyBridge,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.5', '3.6', '4.0'], 'autoNumaBalancing': 1, 'additionalFeatures': [], 'reservedMem': '321', 'bondings': {'bond0': {'ipv6autoconf': True, 'ipv4addrs': ['172.16.0.3/28'], 'cfg': {'DEVICE': 'bond0', 'IPV6INIT': 'yes', 'DEFROUTE': 'no', 'IPADDR': '172.16.0.3', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.240', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'IPV6_AUTOCONF': 'yes', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '255.255.255.240', 'dhcpv6': False, 'ad_partner_mac': 'e4:6f:13:46:7b:30', 'hwaddr': '0c:c4:7a:bb:9d:40', 'slaves': ['enp5s0f0', 'enp5s0f1'], 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': '172.16.0.3'}, 'bond1': {'ipv6autoconf': False, 'ipv4addrs': [], 'cfg': {'BRIDGE': 'ovirtmgmt', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'DEVICE': 'bond1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ad_partner_mac': 'e8:cc:18:31:3b:60', 'hwaddr': '0c:c4:7a:3c:05:10', 'slaves': ['eno1', 'eno2'], 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': ''}}, 'software_version': '4.18', 'memSize': '64231', 'cpuSpeed': '2502.242', 'numaNodes': {'1': {'totalMemory': '32768', 'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, '0': {'totalMemory': '32733', 'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, 'cpuSockets': '2', 'vlans': {}, 'lastClientIface': 'lo', 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'version_name': 'Snow Man', 'cpuThreads': '24', 'emulatedMachines': ['pc-i440fx-rhel7.1.0', 'pc-q35-rhel7.3.0', 'rhel6.3.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-i440fx-rhel7.3.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '3.1611.el7.centos', 'pretty_name': 'CentOS Linux 7 (Core)', 'version': '7', 'name': 'RHEL'}, 'lastClient': '::1'}} Thread-601::INFO::2017-02-19 13:40:15,623::bindingxmlrpc::1287::vds::(wrapper) RPC call getCapabilities finished (code=0) in 0.15 seconds Thread-601::INFO::2017-02-19 13:40:15,640::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52844 stopped jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:18,346::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getAllVmStats' in bridge with {} jsonrpc.Executor/1::DEBUG::2017-02-19 13:40:18,347::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getAllVmStats' in bridge with (suppressed) jsonrpc.Executor/1::INFO::2017-02-19 13:40:18,347::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getAllVmStats succeeded in 0.00 seconds jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:18,549::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.setKsmTune' in bridge with {u'tuningParams': {u'merge_across_nodes': 1, u'run': 0}} jsonrpc.Executor/2::DEBUG::2017-02-19 13:40:18,551::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.setKsmTune' in bridge with {'message': 'Done', 'code': 0} jsonrpc.Executor/2::INFO::2017-02-19 13:40:18,552::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.setKsmTune succeeded in 0.00 seconds Reactor thread::INFO::2017-02-19 13:40:20,575::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52854 Reactor thread::DEBUG::2017-02-19 13:40:20,581::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:20,581::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52854 Reactor thread::DEBUG::2017-02-19 13:40:20,582::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52854) BindingXMLRPC::INFO::2017-02-19 13:40:20,582::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52854 Thread-612::INFO::2017-02-19 13:40:20,582::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52854 started Thread-612::DEBUG::2017-02-19 13:40:20,583::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-612::DEBUG::2017-02-19 13:40:20,583::task::597::Storage.TaskManager.Task::(_updateState) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::moving from state init -> state preparing Thread-612::INFO::2017-02-19 13:40:20,583::logUtils::49::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='4fc108de-838f-4acc-9688-c8d45739d19b', leafUUID='f25d5ae6-eaea-4132-a5b4-5765d8ac73f6', allowIllegal=False) Thread-612::DEBUG::2017-02-19 13:40:20,584::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`58853369-1a8f-4a23-a5c6-cf285dfb8825`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3113' at 'prepareImage' Thread-612::DEBUG::2017-02-19 13:40:20,584::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' Thread-612::DEBUG::2017-02-19 13:40:20,584::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) Thread-612::DEBUG::2017-02-19 13:40:20,584::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`58853369-1a8f-4a23-a5c6-cf285dfb8825`::Granted request Thread-612::DEBUG::2017-02-19 13:40:20,584::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) Thread-612::DEBUG::2017-02-19 13:40:20,584::task::995::Storage.TaskManager.Task::(_decref) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::ref 1 aborting False Thread-612::ERROR::2017-02-19 13:40:20,585::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-612::ERROR::2017-02-19 13:40:20,585::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-612::DEBUG::2017-02-19 13:40:20,585::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) Thread-612::DEBUG::2017-02-19 13:40:20,612::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 Thread-612::WARNING::2017-02-19 13:40:20,613::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-612::ERROR::2017-02-19 13:40:20,618::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-612::ERROR::2017-02-19 13:40:20,618::task::868::Storage.TaskManager.Task::(_setError) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3116, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-612::DEBUG::2017-02-19 13:40:20,618::task::887::Storage.TaskManager.Task::(_run) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::Task._run: 979b7a01-09fa-443a-80d8-fb8ec9e80545 ('c7c466c5-983d-40b2-8f8c-d569755281f5', '00000000-0000-0000-0000-000000000000', '4fc108de-838f-4acc-9688-c8d45739d19b', 'f25d5ae6-eaea-4132-a5b4-5765d8ac73f6') {'allowIllegal': False} failed - stopping task Thread-612::DEBUG::2017-02-19 13:40:20,619::task::1248::Storage.TaskManager.Task::(stop) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::stopping in state preparing (force False) Thread-612::DEBUG::2017-02-19 13:40:20,619::task::995::Storage.TaskManager.Task::(_decref) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::ref 1 aborting True Thread-612::INFO::2017-02-19 13:40:20,619::task::1173::Storage.TaskManager.Task::(prepare) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 Thread-612::DEBUG::2017-02-19 13:40:20,619::task::1178::Storage.TaskManager.Task::(prepare) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::Prepare: aborted: Storage domain does not exist Thread-612::DEBUG::2017-02-19 13:40:20,619::task::995::Storage.TaskManager.Task::(_decref) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::ref 0 aborting True Thread-612::DEBUG::2017-02-19 13:40:20,619::task::930::Storage.TaskManager.Task::(_doAbort) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::Task._doAbort: force False Thread-612::DEBUG::2017-02-19 13:40:20,619::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-612::DEBUG::2017-02-19 13:40:20,619::task::597::Storage.TaskManager.Task::(_updateState) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::moving from state preparing -> state aborting Thread-612::DEBUG::2017-02-19 13:40:20,619::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::_aborting: recover policy none Thread-612::DEBUG::2017-02-19 13:40:20,620::task::597::Storage.TaskManager.Task::(_updateState) Task=`979b7a01-09fa-443a-80d8-fb8ec9e80545`::moving from state aborting -> state failed Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. Thread-612::DEBUG::2017-02-19 13:40:20,620::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. Thread-612::ERROR::2017-02-19 13:40:20,620::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} Thread-612::INFO::2017-02-19 13:40:20,621::bindingxmlrpc::331::vds::(wrapper) RPC call imagePrepare finished (code=358) in 0.04 seconds Thread-612::INFO::2017-02-19 13:40:20,622::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52854 stopped Reactor thread::INFO::2017-02-19 13:40:25,531::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52856 Reactor thread::DEBUG::2017-02-19 13:40:25,537::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:25,537::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52856 Reactor thread::DEBUG::2017-02-19 13:40:25,537::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52856) Reactor thread::INFO::2017-02-19 13:40:25,538::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52858 BindingXMLRPC::INFO::2017-02-19 13:40:25,538::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52856 Thread-619::INFO::2017-02-19 13:40:25,538::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52856 started Thread-619::DEBUG::2017-02-19 13:40:25,539::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call vmGetStats with ('ea5cb711-45b5-4767-aa05-47fe36710d8a',) {} Thread-619::DEBUG::2017-02-19 13:40:25,539::bindingxmlrpc::1260::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-619::INFO::2017-02-19 13:40:25,539::bindingxmlrpc::1287::vds::(wrapper) RPC call vmGetStats finished (code=1) in 0.00 seconds Thread-619::INFO::2017-02-19 13:40:25,540::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52856 stopped Reactor thread::DEBUG::2017-02-19 13:40:25,543::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:25,543::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52858 Reactor thread::DEBUG::2017-02-19 13:40:25,544::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52858) BindingXMLRPC::INFO::2017-02-19 13:40:25,544::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52858 Thread-620::INFO::2017-02-19 13:40:25,544::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52858 started Thread-620::DEBUG::2017-02-19 13:40:25,544::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call vmGetStats with ('ea5cb711-45b5-4767-aa05-47fe36710d8a',) {} Thread-620::DEBUG::2017-02-19 13:40:25,545::bindingxmlrpc::1260::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-620::INFO::2017-02-19 13:40:25,545::bindingxmlrpc::1287::vds::(wrapper) RPC call vmGetStats finished (code=1) in 0.00 seconds Thread-620::INFO::2017-02-19 13:40:25,545::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52858 stopped Reactor thread::INFO::2017-02-19 13:40:25,583::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52860 Reactor thread::DEBUG::2017-02-19 13:40:25,589::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:25,589::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52860 Reactor thread::DEBUG::2017-02-19 13:40:25,589::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52860) BindingXMLRPC::INFO::2017-02-19 13:40:25,589::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52860 Thread-621::INFO::2017-02-19 13:40:25,589::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52860 started Thread-621::DEBUG::2017-02-19 13:40:25,590::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getStats with () {} Thread-621::DEBUG::2017-02-19 13:40:25,591::task::597::Storage.TaskManager.Task::(_updateState) Task=`10c73f43-d3e9-4853-9cfe-3e0f7a8edaca`::moving from state init -> state preparing Thread-621::INFO::2017-02-19 13:40:25,591::logUtils::49::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-621::INFO::2017-02-19 13:40:25,591::logUtils::52::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-621::DEBUG::2017-02-19 13:40:25,591::task::1193::Storage.TaskManager.Task::(prepare) Task=`10c73f43-d3e9-4853-9cfe-3e0f7a8edaca`::finished: {} Thread-621::DEBUG::2017-02-19 13:40:25,591::task::597::Storage.TaskManager.Task::(_updateState) Task=`10c73f43-d3e9-4853-9cfe-3e0f7a8edaca`::moving from state preparing -> state finished Thread-621::DEBUG::2017-02-19 13:40:25,591::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-621::DEBUG::2017-02-19 13:40:25,591::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-621::DEBUG::2017-02-19 13:40:25,592::task::995::Storage.TaskManager.Task::(_decref) Task=`10c73f43-d3e9-4853-9cfe-3e0f7a8edaca`::ref 0 aborting False Thread-621::ERROR::2017-02-19 13:40:25,598::api::255::root::(_getHaInfo) failed to retrieve Hosted Engine HA info Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 234, in _getHaInfo stats = instance.get_all_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats self._configure_broker_conn(broker) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn dom_type=dom_type) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain .format(sd_type, options, e)) RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'iscsi', 'sd_uuid': 'c7c466c5-983d-40b2-8f8c-d569755281f5'}: Request failed: Thread-621::DEBUG::2017-02-19 13:40:25,598::bindingxmlrpc::1260::vds::(wrapper) return getStats with {'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '21': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.67'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '8.80', 'nodeIndex': 0, 'cpuSys': '6.07', 'cpuIdle': '85.13'}, '0': {'cpuUser': '0.13', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.80'}, '3': {'cpuUser': '0.47', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.46'}, '2': {'cpuUser': '5.13', 'nodeIndex': 0, 'cpuSys': '0.67', 'cpuIdle': '94.20'}, '5': {'cpuUser': '1.60', 'nodeIndex': 0, 'cpuSys': '0.20', 'cpuIdle': '98.20'}, '4': {'cpuUser': '0.40', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.53'}, '7': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.47'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '8': {'cpuUser': '0.13', 'nodeIndex': 1, 'cpuSys': '0.13', 'cpuIdle': '99.74'}, '11': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.73'}, '10': {'cpuUser': '0.67', 'nodeIndex': 1, 'cpuSys': '0.40', 'cpuIdle': '98.93'}, '13': {'cpuUser': '3.00', 'nodeIndex': 0, 'cpuSys': '0.60', 'cpuIdle': '96.40'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '4.20', 'nodeIndex': 1, 'cpuSys': '5.20', 'cpuIdle': '90.60'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 4, 'memFree': '31735'}, '0': {'memPercent': 5, 'memFree': '31234'}}, 'memShared': 0, 'haScore': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'rxRate': '0.00', 'vmCount': 0, 'memUsed': '2', 'storageDomains': {}, 'incomingVmMigrations': 0, 'network': {'bond0': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '20000', 'rxDropped': '1062', 'name': 'bond0', 'tx': '1614178', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '8200323', 'state': 'up'}, 'bond1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '0', 'name': 'bond1', 'tx': '103710465', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '13861165', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '103072726', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '8584664', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1041', 'name': 'ovirtmgmt', 'tx': '99103527', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '10432505', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '19867826', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '19867826', 'state': 'up'}, 'eno2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno2', 'tx': '683117', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '5597356', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '0', 'state': 'down'}, 'enp5s0f1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '10000', 'rxDropped': '0', 'name': 'enp5s0f1', 'tx': '643164', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '3364803', 'state': 'up'}, 'enp5s0f0': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '10000', 'rxDropped': '0', 'name': 'enp5s0f0', 'tx': '971014', 'txDropped': '0', 'sampleTime': 1487500816.099152, 'rx': '4835640', 'state': 'up'}}, 'txDropped': '0', 'cpuUser': '1.06', 'ksmPages': 100, 'elapsedTime': '24.50', 'cpuLoad': '0.06', 'cpuSys': '0.59', 'diskStats': {'/var/log': {'free': '48148'}, '/var/log/core': {'free': '48148'}, '/var/run/vdsm/': {'free': '32098'}, '/tmp': {'free': '48148'}}, 'cpuUserVdsmd': '8.20', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 63043, 'txRate': '0.00', 'bootTime': '1487495620', 'haStats': {'active': False, 'configured': True, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'rxDropped': '2103', 'outgoingVmMigrations': 0, 'swapTotal': 19023, 'swapFree': 19023, 'dateTime': '2017-02-19T10:40:25 GMT', 'anonHugePages': '32', 'memFree': 63299, 'cpuIdle': '98.35', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '2.80'}} Thread-621::INFO::2017-02-19 13:40:25,599::bindingxmlrpc::1287::vds::(wrapper) RPC call getStats finished (code=0) in 0.01 seconds Thread-621::INFO::2017-02-19 13:40:25,611::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52860 stopped Reactor thread::INFO::2017-02-19 13:40:25,631::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52862 Reactor thread::DEBUG::2017-02-19 13:40:25,637::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:25,637::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52862 Reactor thread::DEBUG::2017-02-19 13:40:25,637::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52862) BindingXMLRPC::INFO::2017-02-19 13:40:25,637::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52862 Thread-622::INFO::2017-02-19 13:40:25,638::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52862 started Thread-622::DEBUG::2017-02-19 13:40:25,638::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-622::DEBUG::2017-02-19 13:40:25,638::task::597::Storage.TaskManager.Task::(_updateState) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::moving from state init -> state preparing Thread-622::INFO::2017-02-19 13:40:25,639::logUtils::49::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='4fc108de-838f-4acc-9688-c8d45739d19b', leafUUID='f25d5ae6-eaea-4132-a5b4-5765d8ac73f6', allowIllegal=False) Thread-622::DEBUG::2017-02-19 13:40:25,639::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`932eea90-8d20-4bf3-af68-621f9b6cf670`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3113' at 'prepareImage' Thread-622::DEBUG::2017-02-19 13:40:25,639::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' Thread-622::DEBUG::2017-02-19 13:40:25,639::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) Thread-622::DEBUG::2017-02-19 13:40:25,639::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`932eea90-8d20-4bf3-af68-621f9b6cf670`::Granted request Thread-622::DEBUG::2017-02-19 13:40:25,640::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) Thread-622::DEBUG::2017-02-19 13:40:25,640::task::995::Storage.TaskManager.Task::(_decref) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::ref 1 aborting False Thread-622::ERROR::2017-02-19 13:40:25,640::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-622::ERROR::2017-02-19 13:40:25,640::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-622::DEBUG::2017-02-19 13:40:25,640::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) Reactor thread::INFO::2017-02-19 13:40:25,653::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52864 Reactor thread::DEBUG::2017-02-19 13:40:25,659::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:25,659::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52864 Reactor thread::DEBUG::2017-02-19 13:40:25,660::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52864) BindingXMLRPC::INFO::2017-02-19 13:40:25,660::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52864 Thread-623::INFO::2017-02-19 13:40:25,660::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52864 started Thread-623::DEBUG::2017-02-19 13:40:25,661::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getCapabilities with () {} Thread-622::DEBUG::2017-02-19 13:40:25,666::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 Thread-622::WARNING::2017-02-19 13:40:25,666::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-622::ERROR::2017-02-19 13:40:25,671::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-622::ERROR::2017-02-19 13:40:25,671::task::868::Storage.TaskManager.Task::(_setError) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3116, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-622::DEBUG::2017-02-19 13:40:25,671::task::887::Storage.TaskManager.Task::(_run) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::Task._run: 5406574b-15bd-48cd-84b1-07dff4f369d7 ('c7c466c5-983d-40b2-8f8c-d569755281f5', '00000000-0000-0000-0000-000000000000', '4fc108de-838f-4acc-9688-c8d45739d19b', 'f25d5ae6-eaea-4132-a5b4-5765d8ac73f6') {'allowIllegal': False} failed - stopping task Thread-622::DEBUG::2017-02-19 13:40:25,672::task::1248::Storage.TaskManager.Task::(stop) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::stopping in state preparing (force False) Thread-622::DEBUG::2017-02-19 13:40:25,672::task::995::Storage.TaskManager.Task::(_decref) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::ref 1 aborting True Thread-622::INFO::2017-02-19 13:40:25,672::task::1173::Storage.TaskManager.Task::(prepare) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 Thread-622::DEBUG::2017-02-19 13:40:25,672::task::1178::Storage.TaskManager.Task::(prepare) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::Prepare: aborted: Storage domain does not exist Thread-622::DEBUG::2017-02-19 13:40:25,672::task::995::Storage.TaskManager.Task::(_decref) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::ref 0 aborting True Thread-622::DEBUG::2017-02-19 13:40:25,672::task::930::Storage.TaskManager.Task::(_doAbort) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::Task._doAbort: force False Thread-622::DEBUG::2017-02-19 13:40:25,672::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-622::DEBUG::2017-02-19 13:40:25,672::task::597::Storage.TaskManager.Task::(_updateState) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::moving from state preparing -> state aborting Thread-622::DEBUG::2017-02-19 13:40:25,672::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::_aborting: recover policy none Thread-622::DEBUG::2017-02-19 13:40:25,673::task::597::Storage.TaskManager.Task::(_updateState) Task=`5406574b-15bd-48cd-84b1-07dff4f369d7`::moving from state aborting -> state failed Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. Thread-622::DEBUG::2017-02-19 13:40:25,673::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. Thread-622::ERROR::2017-02-19 13:40:25,673::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} Thread-622::INFO::2017-02-19 13:40:25,674::bindingxmlrpc::331::vds::(wrapper) RPC call imagePrepare finished (code=358) in 0.04 seconds Thread-622::INFO::2017-02-19 13:40:25,675::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52862 stopped Thread-623::DEBUG::2017-02-19 13:40:25,709::bindingxmlrpc::1260::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2'}], 'FC': []}, 'packages2': {'kernel': {'release': '514.6.1.el7.x86_64', 'buildtime': 1484733996.0, 'version': '3.10.0'}, 'spice-server': {'release': '19.el7', 'buildtime': 1478932140L, 'version': '0.12.4'}, 'librbd1': {'release': '1.el7', 'buildtime': 1478375483L, 'version': '0.94.5'}, 'vdsm': {'release': '1.el7.centos', 'buildtime': 1481708591L, 'version': '4.18.21'}, 'qemu-kvm': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'libvirt': {'release': '10.el7_3.4', 'buildtime': 1484696570L, 'version': '2.0.0'}, 'qemu-img': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'mom': {'release': '1.el7.centos', 'buildtime': 1477300773L, 'version': '0.5.8'}, 'glusterfs-cli': {'release': '1.el7', 'buildtime': 1483963815L, 'version': '3.7.19'}}, 'numaNodeDistance': {'1': [21, 10], '0': [10, 21]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz', 'liveMerge': 'true', 'hooks': {'before_nic_hotplug': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}, 'before_vm_start': {'50_hostedengine': {'md5': '2a6d96c26a3599812be6cf1a13d9f485'}, '50_nestedvt': {'md5': '70009fa966ddf89b7f9ad60c286077ae'}, '50_macspoof': {'md5': '68cf48c446387550a9a82283079b80f8'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}}, 'before_device_create': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}}, 'supportsIPv6': True, 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'kdumpStatus': 1, 'networks': {'ovirtmgmt': {'ipv4addrs': ['192.168.3.12/24'], 'iface': 'ovirtmgmt', 'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'netmask': '255.255.255.0', 'mtu': '1500', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'stp': 'off', 'bridged': True, 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'ports': ['bond1']}, 'storage': {'ipv4addrs': ['172.16.0.3/28'], 'iface': 'bond0', 'ipv6autoconf': True, 'addr': '172.16.0.3', 'ipv6addrs': [], 'netmask': '255.255.255.240', 'mtu': '9000', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'bridged': False, 'interface': 'bond0', 'ipv6gateway': '::', 'gateway': ''}}, 'kernelArgs': 'BOOT_IMAGE=/vmlinuz-3.10.0-514.6.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=ru_RU.UTF-8', 'bridges': {'ovirtmgmt': {'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': ['192.168.3.12/24'], 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '130', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.0cc47a3c0510', 'bridge_id': '8000.0cc47a3c0510', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '3955', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['bond1']}}, 'uuid': '00000000-0000-0000-0000-0CC47A3C0510', 'onlineCpus': '0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', 'nameservers': ['192.168.3.16'], 'nics': {'eno1': {'permhwaddr': '0c:c4:7a:3c:05:10', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'eno2': {'permhwaddr': '0c:c4:7a:3c:05:11', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'enp5s0f0': {'permhwaddr': '0c:c4:7a:bb:9d:40', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f0', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}, 'enp5s0f1': {'permhwaddr': '0c:c4:7a:bb:9d:41', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}}, 'software_revision': '1', 'hostdevPassthrough': 'false', 'clusterLevels': ['3.5', '3.6', '4.0'], 'cpuFlags': 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,pln,pts,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,xsaveopt,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_IvyBridge,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.5', '3.6', '4.0'], 'autoNumaBalancing': 1, 'additionalFeatures': [], 'reservedMem': '321', 'bondings': {'bond0': {'ipv6autoconf': True, 'ipv4addrs': ['172.16.0.3/28'], 'cfg': {'DEVICE': 'bond0', 'IPV6INIT': 'yes', 'DEFROUTE': 'no', 'IPADDR': '172.16.0.3', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.240', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'IPV6_AUTOCONF': 'yes', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '255.255.255.240', 'dhcpv6': False, 'ad_partner_mac': 'e4:6f:13:46:7b:30', 'hwaddr': '0c:c4:7a:bb:9d:40', 'slaves': ['enp5s0f0', 'enp5s0f1'], 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': '172.16.0.3'}, 'bond1': {'ipv6autoconf': False, 'ipv4addrs': [], 'cfg': {'BRIDGE': 'ovirtmgmt', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'DEVICE': 'bond1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ad_partner_mac': 'e8:cc:18:31:3b:60', 'hwaddr': '0c:c4:7a:3c:05:10', 'slaves': ['eno1', 'eno2'], 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': ''}}, 'software_version': '4.18', 'memSize': '64231', 'cpuSpeed': '2502.242', 'numaNodes': {'1': {'totalMemory': '32768', 'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, '0': {'totalMemory': '32733', 'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, 'cpuSockets': '2', 'vlans': {}, 'lastClientIface': 'lo', 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'version_name': 'Snow Man', 'cpuThreads': '24', 'emulatedMachines': ['pc-i440fx-rhel7.1.0', 'pc-q35-rhel7.3.0', 'rhel6.3.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-i440fx-rhel7.3.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '3.1611.el7.centos', 'pretty_name': 'CentOS Linux 7 (Core)', 'version': '7', 'name': 'RHEL'}, 'lastClient': '::1'}} Thread-623::INFO::2017-02-19 13:40:25,710::bindingxmlrpc::1287::vds::(wrapper) RPC call getCapabilities finished (code=0) in 0.05 seconds Thread-623::INFO::2017-02-19 13:40:25,727::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52864 stopped Reactor thread::INFO::2017-02-19 13:40:30,684::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52866 Reactor thread::DEBUG::2017-02-19 13:40:30,690::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:30,690::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52866 Reactor thread::DEBUG::2017-02-19 13:40:30,690::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52866) BindingXMLRPC::INFO::2017-02-19 13:40:30,690::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52866 Thread-630::INFO::2017-02-19 13:40:30,690::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52866 started Thread-630::DEBUG::2017-02-19 13:40:30,691::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-630::DEBUG::2017-02-19 13:40:30,691::task::597::Storage.TaskManager.Task::(_updateState) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::moving from state init -> state preparing Thread-630::INFO::2017-02-19 13:40:30,692::logUtils::49::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='4fc108de-838f-4acc-9688-c8d45739d19b', leafUUID='f25d5ae6-eaea-4132-a5b4-5765d8ac73f6', allowIllegal=False) Thread-630::DEBUG::2017-02-19 13:40:30,692::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6d50c92b-42fc-4ce4-b828-d0269edce6f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3113' at 'prepareImage' Thread-630::DEBUG::2017-02-19 13:40:30,692::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' Thread-630::DEBUG::2017-02-19 13:40:30,692::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) Thread-630::DEBUG::2017-02-19 13:40:30,692::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`6d50c92b-42fc-4ce4-b828-d0269edce6f4`::Granted request Thread-630::DEBUG::2017-02-19 13:40:30,692::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) Thread-630::DEBUG::2017-02-19 13:40:30,693::task::995::Storage.TaskManager.Task::(_decref) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::ref 1 aborting False Thread-630::ERROR::2017-02-19 13:40:30,693::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-630::ERROR::2017-02-19 13:40:30,693::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-630::DEBUG::2017-02-19 13:40:30,693::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) Thread-630::DEBUG::2017-02-19 13:40:30,720::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 Thread-630::WARNING::2017-02-19 13:40:30,720::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-630::ERROR::2017-02-19 13:40:30,725::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-630::ERROR::2017-02-19 13:40:30,726::task::868::Storage.TaskManager.Task::(_setError) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3116, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-630::DEBUG::2017-02-19 13:40:30,726::task::887::Storage.TaskManager.Task::(_run) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::Task._run: c607233f-97cd-4313-8848-ef164100e5bb ('c7c466c5-983d-40b2-8f8c-d569755281f5', '00000000-0000-0000-0000-000000000000', '4fc108de-838f-4acc-9688-c8d45739d19b', 'f25d5ae6-eaea-4132-a5b4-5765d8ac73f6') {'allowIllegal': False} failed - stopping task Thread-630::DEBUG::2017-02-19 13:40:30,726::task::1248::Storage.TaskManager.Task::(stop) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::stopping in state preparing (force False) Thread-630::DEBUG::2017-02-19 13:40:30,726::task::995::Storage.TaskManager.Task::(_decref) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::ref 1 aborting True Thread-630::INFO::2017-02-19 13:40:30,726::task::1173::Storage.TaskManager.Task::(prepare) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 Thread-630::DEBUG::2017-02-19 13:40:30,726::task::1178::Storage.TaskManager.Task::(prepare) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::Prepare: aborted: Storage domain does not exist Thread-630::DEBUG::2017-02-19 13:40:30,726::task::995::Storage.TaskManager.Task::(_decref) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::ref 0 aborting True Thread-630::DEBUG::2017-02-19 13:40:30,726::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::Task._doAbort: force False Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-630::DEBUG::2017-02-19 13:40:30,727::task::597::Storage.TaskManager.Task::(_updateState) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::moving from state preparing -> state aborting Thread-630::DEBUG::2017-02-19 13:40:30,727::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::_aborting: recover policy none Thread-630::DEBUG::2017-02-19 13:40:30,727::task::597::Storage.TaskManager.Task::(_updateState) Task=`c607233f-97cd-4313-8848-ef164100e5bb`::moving from state aborting -> state failed Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) Thread-630::DEBUG::2017-02-19 13:40:30,727::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. Thread-630::DEBUG::2017-02-19 13:40:30,728::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. Thread-630::ERROR::2017-02-19 13:40:30,728::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} Thread-630::INFO::2017-02-19 13:40:30,728::bindingxmlrpc::331::vds::(wrapper) RPC call imagePrepare finished (code=358) in 0.04 seconds Thread-630::INFO::2017-02-19 13:40:30,729::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52866 stopped jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:33,365::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getAllVmStats' in bridge with {} jsonrpc.Executor/3::DEBUG::2017-02-19 13:40:33,366::__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'Host.getAllVmStats' in bridge with (suppressed) jsonrpc.Executor/3::INFO::2017-02-19 13:40:33,366::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getAllVmStats succeeded in 0.00 seconds Reactor thread::INFO::2017-02-19 13:40:35,558::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52868 Reactor thread::DEBUG::2017-02-19 13:40:35,564::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:35,565::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52868 Reactor thread::DEBUG::2017-02-19 13:40:35,565::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52868) BindingXMLRPC::INFO::2017-02-19 13:40:35,565::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52868 Thread-637::INFO::2017-02-19 13:40:35,565::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52868 started Thread-637::DEBUG::2017-02-19 13:40:35,566::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call vmGetStats with ('ea5cb711-45b5-4767-aa05-47fe36710d8a',) {} Thread-637::DEBUG::2017-02-19 13:40:35,566::bindingxmlrpc::1260::vds::(wrapper) return vmGetStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-637::INFO::2017-02-19 13:40:35,566::bindingxmlrpc::1287::vds::(wrapper) RPC call vmGetStats finished (code=1) in 0.00 seconds Thread-637::INFO::2017-02-19 13:40:35,567::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52868 stopped Reactor thread::INFO::2017-02-19 13:40:35,622::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52870 Reactor thread::DEBUG::2017-02-19 13:40:35,628::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:35,628::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52870 Reactor thread::DEBUG::2017-02-19 13:40:35,629::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52870) BindingXMLRPC::INFO::2017-02-19 13:40:35,629::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52870 Thread-638::INFO::2017-02-19 13:40:35,629::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52870 started Thread-638::DEBUG::2017-02-19 13:40:35,629::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getStats with () {} Thread-638::DEBUG::2017-02-19 13:40:35,630::task::597::Storage.TaskManager.Task::(_updateState) Task=`73df5860-2d9c-4167-a8da-3c3d6ac7ee0a`::moving from state init -> state preparing Thread-638::INFO::2017-02-19 13:40:35,630::logUtils::49::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-638::INFO::2017-02-19 13:40:35,630::logUtils::52::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-638::DEBUG::2017-02-19 13:40:35,631::task::1193::Storage.TaskManager.Task::(prepare) Task=`73df5860-2d9c-4167-a8da-3c3d6ac7ee0a`::finished: {} Thread-638::DEBUG::2017-02-19 13:40:35,631::task::597::Storage.TaskManager.Task::(_updateState) Task=`73df5860-2d9c-4167-a8da-3c3d6ac7ee0a`::moving from state preparing -> state finished Thread-638::DEBUG::2017-02-19 13:40:35,631::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-638::DEBUG::2017-02-19 13:40:35,631::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-638::DEBUG::2017-02-19 13:40:35,631::task::995::Storage.TaskManager.Task::(_decref) Task=`73df5860-2d9c-4167-a8da-3c3d6ac7ee0a`::ref 0 aborting False Thread-638::ERROR::2017-02-19 13:40:35,637::api::255::root::(_getHaInfo) failed to retrieve Hosted Engine HA info Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 234, in _getHaInfo stats = instance.get_all_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats self._configure_broker_conn(broker) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn dom_type=dom_type) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain .format(sd_type, options, e)) RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'iscsi', 'sd_uuid': 'c7c466c5-983d-40b2-8f8c-d569755281f5'}: Request failed: Thread-638::DEBUG::2017-02-19 13:40:35,638::bindingxmlrpc::1260::vds::(wrapper) return getStats with {'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '22': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.60', 'nodeIndex': 0, 'cpuSys': '0.27', 'cpuIdle': '99.13'}, '0': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '3': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '2': {'cpuUser': '0.33', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.60'}, '5': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.66'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '10': {'cpuUser': '0.07', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '13': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '17': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '19': {'cpuUser': '0.27', 'nodeIndex': 1, 'cpuSys': '0.20', 'cpuIdle': '99.53'}, '18': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}}, 'numaNodeMemFree': {'1': {'memPercent': 4, 'memFree': '31736'}, '0': {'memPercent': 5, 'memFree': '31234'}}, 'memShared': 0, 'haScore': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'rxRate': '0.00', 'vmCount': 0, 'memUsed': '2', 'storageDomains': {}, 'incomingVmMigrations': 0, 'network': {'bond0': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '20000', 'rxDropped': '1064', 'name': 'bond0', 'tx': '1618618', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '8221806', 'state': 'up'}, 'bond1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '2000', 'rxDropped': '0', 'name': 'bond1', 'tx': '103765785', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '13899818', 'state': 'up'}, 'eno1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno1', 'tx': '103126088', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '8606878', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '1043', 'name': 'ovirtmgmt', 'tx': '99153615', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '10461964', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '19936809', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '19936809', 'state': 'up'}, 'eno2': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'eno2', 'tx': '685075', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '5613795', 'state': 'up'}, ';vdsmdummy;': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '0', 'state': 'down'}, 'enp5s0f1': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '10000', 'rxDropped': '0', 'name': 'enp5s0f1', 'tx': '645024', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '3375032', 'state': 'up'}, 'enp5s0f0': {'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'speed': '10000', 'rxDropped': '0', 'name': 'enp5s0f0', 'tx': '973594', 'txDropped': '0', 'sampleTime': 1487500831.102692, 'rx': '4846894', 'state': 'up'}}, 'txDropped': '0', 'cpuUser': '0.07', 'ksmPages': 100, 'elapsedTime': '34.54', 'cpuLoad': '0.06', 'cpuSys': '0.04', 'diskStats': {'/var/log': {'free': '48148'}, '/var/log/core': {'free': '48148'}, '/var/run/vdsm/': {'free': '32098'}, '/tmp': {'free': '48148'}}, 'cpuUserVdsmd': '0.60', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 63044, 'txRate': '0.00', 'bootTime': '1487495620', 'haStats': {'active': False, 'configured': True, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'rxDropped': '2107', 'outgoingVmMigrations': 0, 'swapTotal': 19023, 'swapFree': 19023, 'dateTime': '2017-02-19T10:40:35 GMT', 'anonHugePages': '36', 'memFree': 63300, 'cpuIdle': '99.89', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.20'}} Thread-638::INFO::2017-02-19 13:40:35,638::bindingxmlrpc::1287::vds::(wrapper) RPC call getStats finished (code=0) in 0.01 seconds Thread-638::INFO::2017-02-19 13:40:35,650::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52870 stopped Reactor thread::INFO::2017-02-19 13:40:35,739::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52872 Reactor thread::INFO::2017-02-19 13:40:35,739::protocoldetector::76::ProtocolDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:52874 Reactor thread::DEBUG::2017-02-19 13:40:35,745::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::DEBUG::2017-02-19 13:40:35,751::protocoldetector::92::ProtocolDetector.Detector::(__init__) Using required_size=11 Reactor thread::INFO::2017-02-19 13:40:35,752::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52872 Reactor thread::DEBUG::2017-02-19 13:40:35,752::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52872) Reactor thread::INFO::2017-02-19 13:40:35,752::protocoldetector::128::ProtocolDetector.Detector::(handle_read) Detected protocol xml from ::1:52874 Reactor thread::DEBUG::2017-02-19 13:40:35,752::bindingxmlrpc::1307::XmlDetector::(handle_socket) xml over http detected from ('::1', 52874) BindingXMLRPC::INFO::2017-02-19 13:40:35,752::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52872 BindingXMLRPC::INFO::2017-02-19 13:40:35,753::xmlrpc::73::vds.XMLRPCServer::(handle_request) Starting request handler for ::1:52874 Thread-639::INFO::2017-02-19 13:40:35,753::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52872 started Thread-639::DEBUG::2017-02-19 13:40:35,753::bindingxmlrpc::1252::vds::(wrapper) client [::1]::call getCapabilities with () {} Thread-640::INFO::2017-02-19 13:40:35,754::xmlrpc::83::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52874 started Thread-640::DEBUG::2017-02-19 13:40:35,754::bindingxmlrpc::319::vds::(wrapper) client [::1] Thread-640::DEBUG::2017-02-19 13:40:35,755::task::597::Storage.TaskManager.Task::(_updateState) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::moving from state init -> state preparing Thread-640::INFO::2017-02-19 13:40:35,755::logUtils::49::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID='c7c466c5-983d-40b2-8f8c-d569755281f5', spUUID='00000000-0000-0000-0000-000000000000', imgUUID='4fc108de-838f-4acc-9688-c8d45739d19b', leafUUID='f25d5ae6-eaea-4132-a5b4-5765d8ac73f6', allowIllegal=False) Thread-640::DEBUG::2017-02-19 13:40:35,755::resourceManager::201::Storage.ResourceManager.Request::(__init__) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3d2f3d02-7b14-4852-85e1-a27d1241f36d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3113' at 'prepareImage' Thread-640::DEBUG::2017-02-19 13:40:35,755::resourceManager::554::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' for lock type 'shared' Thread-640::DEBUG::2017-02-19 13:40:35,755::resourceManager::613::Storage.ResourceManager::(registerResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free. Now locking as 'shared' (1 active user) Thread-640::DEBUG::2017-02-19 13:40:35,755::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.c7c466c5-983d-40b2-8f8c-d569755281f5`ReqID=`3d2f3d02-7b14-4852-85e1-a27d1241f36d`::Granted request Thread-640::DEBUG::2017-02-19 13:40:35,756::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::_resourcesAcquired: Storage.c7c466c5-983d-40b2-8f8c-d569755281f5 (shared) Thread-640::DEBUG::2017-02-19 13:40:35,756::task::995::Storage.TaskManager.Task::(_decref) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::ref 1 aborting False Thread-640::ERROR::2017-02-19 13:40:35,756::sdc::140::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-640::ERROR::2017-02-19 13:40:35,756::sdc::157::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain c7c466c5-983d-40b2-8f8c-d569755281f5 Thread-640::DEBUG::2017-02-19 13:40:35,756::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/3600605b00a20bf201e29c6bf236a6651|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name c7c466c5-983d-40b2-8f8c-d569755281f5 (cwd None) Thread-640::DEBUG::2017-02-19 13:40:35,784::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: = ' WARNING: Not using lvmetad because config setting use_lvmetad=0.\n WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\n Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found\n Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5\n'; = 5 Thread-640::WARNING::2017-02-19 13:40:35,785::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: Not using lvmetad because config setting use_lvmetad=0.', ' WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).', ' Volume group "c7c466c5-983d-40b2-8f8c-d569755281f5" not found', ' Cannot process volume group c7c466c5-983d-40b2-8f8c-d569755281f5'] Thread-640::ERROR::2017-02-19 13:40:35,789::sdc::146::Storage.StorageDomainCache::(_findDomain) domain c7c466c5-983d-40b2-8f8c-d569755281f5 not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-640::ERROR::2017-02-19 13:40:35,790::task::868::Storage.TaskManager.Task::(_setError) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3116, in prepareImage dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',) Thread-640::DEBUG::2017-02-19 13:40:35,790::task::887::Storage.TaskManager.Task::(_run) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::Task._run: c79b8c1a-fb38-462c-b447-27877ebdc648 ('c7c466c5-983d-40b2-8f8c-d569755281f5', '00000000-0000-0000-0000-000000000000', '4fc108de-838f-4acc-9688-c8d45739d19b', 'f25d5ae6-eaea-4132-a5b4-5765d8ac73f6') {'allowIllegal': False} failed - stopping task Thread-640::DEBUG::2017-02-19 13:40:35,790::task::1248::Storage.TaskManager.Task::(stop) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::stopping in state preparing (force False) Thread-640::DEBUG::2017-02-19 13:40:35,790::task::995::Storage.TaskManager.Task::(_decref) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::ref 1 aborting True Thread-640::INFO::2017-02-19 13:40:35,790::task::1173::Storage.TaskManager.Task::(prepare) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 Thread-640::DEBUG::2017-02-19 13:40:35,790::task::1178::Storage.TaskManager.Task::(prepare) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::Prepare: aborted: Storage domain does not exist Thread-640::DEBUG::2017-02-19 13:40:35,790::task::995::Storage.TaskManager.Task::(_decref) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::ref 0 aborting True Thread-640::DEBUG::2017-02-19 13:40:35,791::task::930::Storage.TaskManager.Task::(_doAbort) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::Task._doAbort: force False Thread-640::DEBUG::2017-02-19 13:40:35,791::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-640::DEBUG::2017-02-19 13:40:35,791::task::597::Storage.TaskManager.Task::(_updateState) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::moving from state preparing -> state aborting Thread-640::DEBUG::2017-02-19 13:40:35,791::task::552::Storage.TaskManager.Task::(__state_aborting) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::_aborting: recover policy none Thread-640::DEBUG::2017-02-19 13:40:35,791::task::597::Storage.TaskManager.Task::(_updateState) Task=`c79b8c1a-fb38-462c-b447-27877ebdc648`::moving from state aborting -> state failed Thread-640::DEBUG::2017-02-19 13:40:35,791::resourceManager::952::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5': < ResourceRef 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', isValid: 'True' obj: 'None'>} Thread-640::DEBUG::2017-02-19 13:40:35,791::resourceManager::989::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-640::DEBUG::2017-02-19 13:40:35,791::resourceManager::628::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' Thread-640::DEBUG::2017-02-19 13:40:35,791::resourceManager::647::Storage.ResourceManager::(releaseResource) Released resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' (0 active users) Thread-640::DEBUG::2017-02-19 13:40:35,792::resourceManager::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5' is free, finding out if anyone is waiting for it. Thread-640::DEBUG::2017-02-19 13:40:35,792::resourceManager::661::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c7c466c5-983d-40b2-8f8c-d569755281f5', Clearing records. Thread-640::ERROR::2017-02-19 13:40:35,792::dispatcher::77::Storage.Dispatcher::(wrapper) {'status': {'message': "Storage domain does not exist: ('c7c466c5-983d-40b2-8f8c-d569755281f5',)", 'code': 358}} Thread-640::INFO::2017-02-19 13:40:35,792::bindingxmlrpc::331::vds::(wrapper) RPC call imagePrepare finished (code=358) in 0.03 seconds Thread-640::INFO::2017-02-19 13:40:35,793::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52874 stopped Thread-639::DEBUG::2017-02-19 13:40:35,808::bindingxmlrpc::1260::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2'}], 'FC': []}, 'packages2': {'kernel': {'release': '514.6.1.el7.x86_64', 'buildtime': 1484733996.0, 'version': '3.10.0'}, 'spice-server': {'release': '19.el7', 'buildtime': 1478932140L, 'version': '0.12.4'}, 'librbd1': {'release': '1.el7', 'buildtime': 1478375483L, 'version': '0.94.5'}, 'vdsm': {'release': '1.el7.centos', 'buildtime': 1481708591L, 'version': '4.18.21'}, 'qemu-kvm': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'libvirt': {'release': '10.el7_3.4', 'buildtime': 1484696570L, 'version': '2.0.0'}, 'qemu-img': {'release': '28.el7_3.3.1', 'buildtime': 1484923072L, 'version': '2.6.0'}, 'mom': {'release': '1.el7.centos', 'buildtime': 1477300773L, 'version': '0.5.8'}, 'glusterfs-cli': {'release': '1.el7', 'buildtime': 1483963815L, 'version': '3.7.19'}}, 'numaNodeDistance': {'1': [21, 10], '0': [10, 21]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz', 'liveMerge': 'true', 'hooks': {'before_nic_hotplug': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}, 'before_vm_start': {'50_hostedengine': {'md5': '2a6d96c26a3599812be6cf1a13d9f485'}, '50_nestedvt': {'md5': '70009fa966ddf89b7f9ad60c286077ae'}, '50_macspoof': {'md5': '68cf48c446387550a9a82283079b80f8'}}, 'before_device_migrate_destination': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}}, 'before_device_create': {'50_vmfex': {'md5': 'e05994261acaea7dcf4b88ea0e81f1f5'}, '50_macspoof': {'md5': '6c5cee1d729261e99dde5c9412c7ee22'}}}, 'supportsIPv6': True, 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'kdumpStatus': 1, 'networks': {'ovirtmgmt': {'ipv4addrs': ['192.168.3.12/24'], 'iface': 'ovirtmgmt', 'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'netmask': '255.255.255.0', 'mtu': '1500', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'stp': 'off', 'bridged': True, 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'ports': ['bond1']}, 'storage': {'ipv4addrs': ['172.16.0.3/28'], 'iface': 'bond0', 'ipv6autoconf': True, 'addr': '172.16.0.3', 'ipv6addrs': [], 'netmask': '255.255.255.240', 'mtu': '9000', 'dhcpv4': False, 'switch': 'legacy', 'dhcpv6': False, 'bridged': False, 'interface': 'bond0', 'ipv6gateway': '::', 'gateway': ''}}, 'kernelArgs': 'BOOT_IMAGE=/vmlinuz-3.10.0-514.6.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=ru_RU.UTF-8', 'bridges': {'ovirtmgmt': {'ipv6autoconf': False, 'addr': '192.168.3.12', 'cfg': {'IPV6INIT': 'no', 'DEFROUTE': 'yes', 'DNS1': '192.168.3.16', 'IPADDR': '192.168.3.12', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.3.9'}, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': 'off', 'ipv4addrs': ['192.168.3.12/24'], 'ipv6gateway': '::', 'gateway': '192.168.3.9', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '120', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.0cc47a3c0510', 'bridge_id': '8000.0cc47a3c0510', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '2945', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'default_pvid': '1', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['bond1']}}, 'uuid': '00000000-0000-0000-0000-0CC47A3C0510', 'onlineCpus': '0,1,2,3,4,5,12,13,14,15,16,17,6,7,8,9,10,11,18,19,20,21,22,23', 'nameservers': ['192.168.3.16'], 'nics': {'eno1': {'permhwaddr': '0c:c4:7a:3c:05:10', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'eno2': {'permhwaddr': '0c:c4:7a:3c:05:11', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond1', 'DEVICE': 'eno2', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:3c:05:10', 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'speed': 1000}, 'enp5s0f0': {'permhwaddr': '0c:c4:7a:bb:9d:40', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f0', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}, 'enp5s0f1': {'permhwaddr': '0c:c4:7a:bb:9d:41', 'ipv6autoconf': False, 'addr': '', 'cfg': {'SLAVE': 'yes', 'IPV6INIT': 'no', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'MASTER': 'bond0', 'DEVICE': 'enp5s0f1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ipv4addrs': [], 'hwaddr': '0c:c4:7a:bb:9d:40', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'speed': 10000}}, 'software_revision': '1', 'hostdevPassthrough': 'false', 'clusterLevels': ['3.5', '3.6', '4.0'], 'cpuFlags': 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,eagerfpu,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,pln,pts,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,xsaveopt,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_IvyBridge,model_Westmere,model_n270,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f4ca2e1c3ca2', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.5', '3.6', '4.0'], 'autoNumaBalancing': 1, 'additionalFeatures': [], 'reservedMem': '321', 'bondings': {'bond0': {'ipv6autoconf': True, 'ipv4addrs': ['172.16.0.3/28'], 'cfg': {'DEVICE': 'bond0', 'IPV6INIT': 'yes', 'DEFROUTE': 'no', 'IPADDR': '172.16.0.3', 'MTU': '9000', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.240', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'IPV6_AUTOCONF': 'yes', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '255.255.255.240', 'dhcpv6': False, 'ad_partner_mac': 'e4:6f:13:46:7b:30', 'hwaddr': '0c:c4:7a:bb:9d:40', 'slaves': ['enp5s0f0', 'enp5s0f1'], 'mtu': '9000', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': '172.16.0.3'}, 'bond1': {'ipv6autoconf': False, 'ipv4addrs': [], 'cfg': {'BRIDGE': 'ovirtmgmt', 'IPV6INIT': 'no', 'MTU': '1500', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'mode=802.3ad lacp_rate=1 miimon=100 xmit_hash_policy=layer2+3', 'DEVICE': 'bond1', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'switch': 'legacy', 'active_slave': '', 'ad_aggregator_id': '3', 'dhcpv4': False, 'netmask': '', 'dhcpv6': False, 'ad_partner_mac': 'e8:cc:18:31:3b:60', 'hwaddr': '0c:c4:7a:3c:05:10', 'slaves': ['eno1', 'eno2'], 'mtu': '1500', 'ipv6gateway': '::', 'gateway': '', 'opts': {'lacp_rate': '1', 'miimon': '100', 'mode': '4', 'xmit_hash_policy': '2'}, 'addr': ''}}, 'software_version': '4.18', 'memSize': '64231', 'cpuSpeed': '2502.242', 'numaNodes': {'1': {'totalMemory': '32768', 'cpus': [6, 7, 8, 9, 10, 11, 18, 19, 20, 21, 22, 23]}, '0': {'totalMemory': '32733', 'cpus': [0, 1, 2, 3, 4, 5, 12, 13, 14, 15, 16, 17]}}, 'cpuSockets': '2', 'vlans': {}, 'lastClientIface': 'lo', 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'version_name': 'Snow Man', 'cpuThreads': '24', 'emulatedMachines': ['pc-i440fx-rhel7.1.0', 'pc-q35-rhel7.3.0', 'rhel6.3.0', 'pc-i440fx-rhel7.0.0', 'rhel6.1.0', 'rhel6.6.0', 'rhel6.2.0', 'pc', 'pc-i440fx-rhel7.3.0', 'q35', 'pc-i440fx-rhel7.2.0', 'rhel6.4.0', 'rhel6.0.0', 'rhel6.5.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '3.1611.el7.centos', 'pretty_name': 'CentOS Linux 7 (Core)', 'version': '7', 'name': 'RHEL'}, 'lastClient': '::1'}} Thread-639::INFO::2017-02-19 13:40:35,809::bindingxmlrpc::1287::vds::(wrapper) RPC call getCapabilities finished (code=0) in 0.06 seconds Thread-639::INFO::2017-02-19 13:40:35,825::xmlrpc::91::vds.XMLRPCServer::(_process_requests) Request handler for ::1:52872 stopped