From: ml(a)ohnewald.net To: users(a)ovirt.org Sent: Monday, May 18, 2015 2:16:23 PM Subject: [ovirt-users] vdsm storage problem - maybe cache problem? Hello List, i did a simple update in my CentOS 6.6 Engine (3.5) and then i rebootet it. After that i ran into trouble. My NFS mounts from my clients to my engine got stuck. So i unmounted them manually with: umount -f -l <path to my mount point> Which worked. I hoped vdsm would remount it again. With no luck. So i deleted the nfs export and iso domain from the cluster. Now i am getting this error in my vdsm log: 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,682::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,683::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,717::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,717::sp::329::Storage.StoragePool::(startSpm) Unexpected error 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,717::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18 13:13:25,928::task::866::TaskManager.Task::(_setError) Task=`2c6b4422-7faa-4847-ab30-fc713d7012af`::Unexpected error Thread-37::ERROR::2015-05-18 13:13:35,683::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:35,683::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:35,720::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-37::ERROR::2015-05-18 13:13:35,720::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Thread-482::ERROR::2015-05-18 13:13:40,062::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-482::ERROR::2015-05-18 13:13:40,063::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-482::ERROR::2015-05-18 13:13:40,147::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,149::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,152::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,191::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,191::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,193::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,193::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,228::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,228::sp::329::Storage.StoragePool::(startSpm) Unexpected error 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,228::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18 13:13:40,932::task::866::TaskManager.Task::(_setError) Task=`3140b81f-a434-4877-9a34-3923505e4a1f`::Unexpected error Thread-37::ERROR::2015-05-18 13:13:45,721::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:45,721::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:45,757::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-37::ERROR::2015-05-18 13:13:45,758::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Thread-501::ERROR::2015-05-18 13:13:54,714::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-501::ERROR::2015-05-18 13:13:54,715::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-501::ERROR::2015-05-18 13:13:54,751::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,751::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,752::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,787::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,788::sp::288::Storage.StoragePool::(startSpm) Backup domain validation failed 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,790::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,790::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,824::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,824::sp::329::Storage.StoragePool::(startSpm) Unexpected error 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,824::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18 13:13:54,937::task::866::TaskManager.Task::(_setError) Task=`5893d839-b69b-4d25-816a-67f2c9f24e12`::Unexpected error Thread-37::ERROR::2015-05-18 13:13:55,758::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:55,759::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-37::ERROR::2015-05-18 13:13:55,794::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-37::ERROR::2015-05-18 13:13:55,795::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information In older engine logs i was able to find out what id 036b5575-51fa-4f14-8b05-890d7807894c is: ---------------------------- 2015-05-18 10:58:55,397 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-43) domain 036b5575-51fa-4f14-8b05-890d7807894c:EXPORT2 in problem. vds: ovirt-node01.stuttgart.imos.net Now my Question: Why does the vdsm node not know that i deleted the storage? Has the vdsm cached this mount informations? Why does it still try to access 036b5575-51fa-4f14-8b05-890d7807894c?

I can NOT reboot my node because i have active running vms on it. Thanks, Mario _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

From: ml(a)ohnewald.net To: "Maor Lipchuk" <mlipchuk(a)redhat.com&gt; Cc: users(a)ovirt.org Sent: Monday, May 18, 2015 2:36:38 PM Subject: Re: [ovirt-users] vdsm storage problem - maybe cache problem? Hi Maor, thanks for the quick reply. Am 18.05.15 um 13:25 schrieb Maor Lipchuk: >> Now my Question: Why does the vdsm node not know that i deleted the >> storage? Has the vdsm cached this mount informations? Why does it still >> try to access 036b5575-51fa-4f14-8b05-890d7807894c? > > > Yes, the vdsm use a cache for Storage Domains, you can try to restart the > vdsmd service instead of rebooting the host. > I am still getting the same error. [root@ovirt-node01 ~]# /etc/init.d/vdsmd stop Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm: Running run_final_hooks [ OK ] vdsm stop [ OK ] [root@ovirt-node01 ~]# [root@ovirt-node01 ~]# [root@ovirt-node01 ~]# [root@ovirt-node01 ~]# ps aux | grep vdsmd root 3198 0.0 0.0 11304 740 ? S< May07 0:00 /bin/bash -e /usr/share/vdsm/respawn --minlifetime 10 --daemon --masterpid /var/run/vdsm/supervdsm_respawn.pid /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock --pidfile /var/run/vdsm/supervdsmd.pid root 3205 0.0 0.0 922368 26724 ? S<l May07 12:10 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock --pidfile /var/run/vdsm/supervdsmd.pid root 15842 0.0 0.0 103248 900 pts/0 S+ 13:35 0:00 grep vdsmd [root@ovirt-node01 ~]# /etc/init.d/vdsmd start initctl: Job is already running: libvirtd vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running gencerts vdsm: Running check_is_configured libvirt is already configured for vdsm sanlock service is already configured vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running restore_nets vdsm: Running unified_network_persistence_upgrade vdsm: Running upgrade_300_nets Starting up vdsm daemon: vdsm start [ OK ] [root@ovirt-node01 ~]# [root@ovirt-node01 ~]# grep ERROR /var/log/vdsm/vdsm.log | tail -n 20 Thread-13::ERROR::2015-05-18 13:35:03,631::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain abc51e26-7175-4b38-b3a8-95c6928fbc2b Thread-13::ERROR::2015-05-18 13:35:03,632::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b Thread-36::ERROR::2015-05-18 13:35:11,607::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:11,621::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:11,960::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-36::ERROR::2015-05-18 13:35:11,960::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Thread-36::ERROR::2015-05-18 13:35:21,962::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:21,965::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:22,068::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-36::ERROR::2015-05-18 13:35:22,072::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Thread-15::ERROR::2015-05-18 13:35:33,821::task::866::TaskManager.Task::(_setError) Task=`54bdfc77-f63a-493b-b24e-e5a3bc4977bb`::Unexpected error Thread-15::ERROR::2015-05-18 13:35:33,864::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('b384b3da-02a6-44f3-a3f6-56751ce8c26d',)", 'code': 309}} Thread-13::ERROR::2015-05-18 13:35:33,930::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain abc51e26-7175-4b38-b3a8-95c6928fbc2b Thread-15::ERROR::2015-05-18 13:35:33,928::task::866::TaskManager.Task::(_setError) Task=`fe9bb0fa-cf1e-4b21-af00-0698c6d1718f`::Unexpected error Thread-13::ERROR::2015-05-18 13:35:33,932::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b Thread-15::ERROR::2015-05-18 13:35:33,978::dispatcher::65::Storage.Dispatcher.Protect::(run) {'status': {'message': 'Not SPM: ()', 'code': 654}} Thread-36::ERROR::2015-05-18 13:35:41,117::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:41,131::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 13:35:41,452::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Thread-36::ERROR::2015-05-18 13:35:41,453::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Thanks, Mario

Manual mounting works: [root@ovirt-node01 ~]# mount ovirt.hq.example.net:/export2 /tmp/mnt/ [root@ovirt-node01 ~]# umount /tmp/mnt/ (but i removed the export2 domain from the webgui engine. I still wonder why the node want to access it....) vdsm log: ================ StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,099::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,099::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,100::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,100::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 036b5575-51fa-4f14-8b05-890d7807894c' (cwd None) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,121::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "036b5575-51fa-4f14-8b05-890d7807894c" not found\n Skipping volume group 036b5575-51fa-4f14-8b05-890d7807894c\n'; <rc> = 5 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::WARNING::2015-05-18 14:35:35,122::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "036b5575-51fa-4f14-8b05-890d7807894c" not found', ' Skipping volume group 036b5575-51fa-4f14-8b05-890d7807894c'] 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,122::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,133::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,133::sp::329::Storage.StoragePool::(startSpm) Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm self._updateDomainsRole() File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole domain = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,134::sp::330::Storage.StoragePool::(startSpm) failed: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,134::sp::336::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,134::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d`ReqID=`24d881e8-eba3-4b51-85e7-32301218b2e9`::Request was made in '/usr/share/vdsm/storage/sp.py' line '338' at '_shutDownUpgrade' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,135::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d' for lock type 'exclusive' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,135::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d' is free. Now locking as 'exclusive' (1 active user) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,135::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d`ReqID=`24d881e8-eba3-4b51-85e7-32301218b2e9`::Granted request 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,136::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c`ReqID=`140bc21d-520f-4796-aab0-595f7aba6932`::Request was made in '/usr/share/vdsm/storage/sp.py' line '358' at '_shutDownUpgrade' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,136::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c' for lock type 'exclusive' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,136::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c' is free. Now locking as 'exclusive' (1 active user) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,136::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c`ReqID=`140bc21d-520f-4796-aab0-595f7aba6932`::Granted request 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,136::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,137::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c' (0 active users) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,137::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c' is free, finding out if anyone is waiting for it. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,137::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_036b5575-51fa-4f14-8b05-890d7807894c', Clearing records. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,137::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,137::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d' (0 active users) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,138::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d' is free, finding out if anyone is waiting for it. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,138::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_b384b3da-02a6-44f3-a3f6-56751ce8c26d', Clearing records. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,138::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,139::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,139::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=RaidVolBGluster', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=HP_Proliant_DL180G6', 'POOL_DOMAINS=6d882c77-cdbc-48ef-ae21-1a6d45e7f8a1:Active,036b5575-51fa-4f14-8b05-890d7807894c:Active,abc51e26-7175-4b38-b3a8-95c6928fbc2b:Active,23602741-2967-4fad-a749-d58e1459d5c8:Attached', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=b384b3da-02a6-44f3-a3f6-56751ce8c26d', 'REMOTE_PATH=127.0.0.1:/RaidVolB', 'ROLE=Master', 'SDUUID=abc51e26-7175-4b38-b3a8-95c6928fbc2b', 'TYPE=GLUSTERFS', 'VERSION=3', '_SHA_CKSUM=401970538872dfd4e17afcb84361851559b18e4c'] 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,146::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::INFO::2015-05-18 14:35:35,146::clusterlock::279::SANLock::(release) Releasing cluster lock for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,151::clusterlock::289::SANLock::(release) Cluster lock for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b successfully released 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::ERROR::2015-05-18 14:35:35,151::task::866::TaskManager.Task::(_setError) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/storage/task.py", line 334, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/share/vdsm/storage/sp.py", line 296, in startSpm self._updateDomainsRole() File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 205, in _updateDomainsRole domain = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,151::task::885::TaskManager.Task::(_run) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::Task._run: 47d37321-3f0f-4a1b-a8d5-629065fc1c4c () {} failed - stopping task 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,152::task::1211::TaskManager.Task::(stop) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::stopping in state running (force False) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,152::task::990::TaskManager.Task::(_decref) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::ref 1 aborting True 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,152::task::916::TaskManager.Task::(_runJobs) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,152::task::990::TaskManager.Task::(_decref) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::ref 0 aborting True 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,152::task::925::TaskManager.Task::(_doAbort) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::Task._doAbort: force False 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::task::595::TaskManager.Task::(_updateState) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::moving from state running -> state aborting 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::task::550::TaskManager.Task::(__state_aborting) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::_aborting: recover policy auto 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::task::595::TaskManager.Task::(_updateState) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::moving from state aborting -> state racquiring 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::task::595::TaskManager.Task::(_updateState) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::moving from state racquiring -> state recovering 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,153::task::798::TaskManager.Task::(_recover) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::_recover 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,154::task::805::TaskManager.Task::(_recover) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::running recovery None 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,154::task::786::TaskManager.Task::(_recoverDone) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::Recover Done: state recovering 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,154::task::595::TaskManager.Task::(_updateState) Task=`47d37321-3f0f-4a1b-a8d5-629065fc1c4c`::moving from state recovering -> state recovered 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,154::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d': < ResourceRef 'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d', isValid: 'True' obj: 'None'>} 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,154::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,155::resourceManager::616::ResourceManager::(releaseResource) Trying to release resource 'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d' 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,155::resourceManager::635::ResourceManager::(releaseResource) Released resource 'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d' (0 active users) 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,155::resourceManager::641::ResourceManager::(releaseResource) Resource 'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d' is free, finding out if anyone is waiting for it. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,155::resourceManager::649::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.b384b3da-02a6-44f3-a3f6-56751ce8c26d', Clearing records. 47d37321-3f0f-4a1b-a8d5-629065fc1c4c::DEBUG::2015-05-18 14:35:35,155::threadPool::57::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 Thread-40::DEBUG::2015-05-18 14:35:35,284::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/glusterSD/127.0.0.1:_RaidVolB/abc51e26-7175-4b38-b3a8-95c6928fbc2b/dom_md/metadata bs=4096 count=1' (cwd None) Thread-40::DEBUG::2015-05-18 14:35:35,295::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n613 bytes (613 B) copied, 0.000701751 s, 874 kB/s\n'; <rc> = 0 Thread-5036::DEBUG::2015-05-18 14:35:35,501::task::595::TaskManager.Task::(_updateState) Task=`4ff82d7b-b4a7-48f0-ba5f-7381d2dcee67`::moving from state init -> state preparing Thread-5036::INFO::2015-05-18 14:35:35,501::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-5036::INFO::2015-05-18 14:35:35,502::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'6d882c77-cdbc-48ef-ae21-1a6d45e7f8a1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000757621', 'lastCheck': '1.9', 'valid': True}, '036b5575-51fa-4f14-8b05-890d7807894c': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.1', 'valid': False}, 'abc51e26-7175-4b38-b3a8-95c6928fbc2b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000701751', 'lastCheck': '0.2', 'valid': True}} Thread-5036::DEBUG::2015-05-18 14:35:35,502::task::1185::TaskManager.Task::(prepare) Task=`4ff82d7b-b4a7-48f0-ba5f-7381d2dcee67`::finished: {'6d882c77-cdbc-48ef-ae21-1a6d45e7f8a1': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000757621', 'lastCheck': '1.9', 'valid': True}, '036b5575-51fa-4f14-8b05-890d7807894c': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.1', 'valid': False}, 'abc51e26-7175-4b38-b3a8-95c6928fbc2b': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000701751', 'lastCheck': '0.2', 'valid': True}} Thread-5036::DEBUG::2015-05-18 14:35:35,502::task::595::TaskManager.Task::(_updateState) Task=`4ff82d7b-b4a7-48f0-ba5f-7381d2dcee67`::moving from state preparing -> state finished Thread-5036::DEBUG::2015-05-18 14:35:35,502::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-5036::DEBUG::2015-05-18 14:35:35,503::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-5036::DEBUG::2015-05-18 14:35:35,503::task::990::TaskManager.Task::(_decref) Task=`4ff82d7b-b4a7-48f0-ba5f-7381d2dcee67`::ref 0 aborting False VM Channels Listener::DEBUG::2015-05-18 14:35:35,562::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:35,562::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:35,562::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:35,563::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:35,563::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:35,563::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-5036::DEBUG::2015-05-18 14:35:35,938::BindingXMLRPC::251::vds::(wrapper) client [192.168.30.103] flowID [447665f7] Thread-5036::DEBUG::2015-05-18 14:35:35,939::task::595::TaskManager.Task::(_updateState) Task=`c7cd482f-ee37-4b8e-a879-16bd4f317e97`::moving from state init -> state preparing Thread-5036::INFO::2015-05-18 14:35:35,939::logUtils::44::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='47d37321-3f0f-4a1b-a8d5-629065fc1c4c', spUUID=None, options=None) Thread-5036::DEBUG::2015-05-18 14:35:35,939::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 47d37321-3f0f-4a1b-a8d5-629065fc1c4c Thread-5036::DEBUG::2015-05-18 14:35:35,939::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '47d37321-3f0f-4a1b-a8d5-629065fc1c4c'} Thread-5036::INFO::2015-05-18 14:35:35,939::logUtils::47::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '47d37321-3f0f-4a1b-a8d5-629065fc1c4c'}} Thread-5036::DEBUG::2015-05-18 14:35:35,940::task::1185::TaskManager.Task::(prepare) Task=`c7cd482f-ee37-4b8e-a879-16bd4f317e97`::finished: {'taskStatus': {'code': 358, 'message': 'Storage domain does not exist', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '47d37321-3f0f-4a1b-a8d5-629065fc1c4c'}} Thread-5036::DEBUG::2015-05-18 14:35:35,940::task::595::TaskManager.Task::(_updateState) Task=`c7cd482f-ee37-4b8e-a879-16bd4f317e97`::moving from state preparing -> state finished Thread-5036::DEBUG::2015-05-18 14:35:35,940::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-5036::DEBUG::2015-05-18 14:35:35,940::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-5036::DEBUG::2015-05-18 14:35:35,940::task::990::TaskManager.Task::(_decref) Task=`c7cd482f-ee37-4b8e-a879-16bd4f317e97`::ref 0 aborting False Thread-5036::DEBUG::2015-05-18 14:35:35,952::BindingXMLRPC::251::vds::(wrapper) client [192.168.30.103] flowID [447665f7] Thread-5036::DEBUG::2015-05-18 14:35:35,953::task::595::TaskManager.Task::(_updateState) Task=`e30dd05c-4520-4074-94ea-632be5ffa06d`::moving from state init -> state preparing Thread-5036::INFO::2015-05-18 14:35:35,953::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', options=None) Thread-5036::INFO::2015-05-18 14:35:35,957::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} Thread-5036::DEBUG::2015-05-18 14:35:35,957::task::1185::TaskManager.Task::(prepare) Task=`e30dd05c-4520-4074-94ea-632be5ffa06d`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} Thread-5036::DEBUG::2015-05-18 14:35:35,958::task::595::TaskManager.Task::(_updateState) Task=`e30dd05c-4520-4074-94ea-632be5ffa06d`::moving from state preparing -> state finished Thread-5036::DEBUG::2015-05-18 14:35:35,958::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-5036::DEBUG::2015-05-18 14:35:35,958::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-5036::DEBUG::2015-05-18 14:35:35,958::task::990::TaskManager.Task::(_decref) Task=`e30dd05c-4520-4074-94ea-632be5ffa06d`::ref 0 aborting False Thread-5036::DEBUG::2015-05-18 14:35:36,062::BindingXMLRPC::251::vds::(wrapper) client [192.168.30.103] flowID [447665f7] Thread-5036::DEBUG::2015-05-18 14:35:36,063::task::595::TaskManager.Task::(_updateState) Task=`a8cae188-5991-4513-a077-2f6d2bc7b3e6`::moving from state init -> state preparing Thread-5036::INFO::2015-05-18 14:35:36,063::logUtils::44::dispatcher::(wrapper) Run and protect: clearTask(taskID='47d37321-3f0f-4a1b-a8d5-629065fc1c4c', spUUID=None, options=None) Thread-5036::DEBUG::2015-05-18 14:35:36,063::taskManager::161::TaskManager::(clearTask) Entry. taskID: 47d37321-3f0f-4a1b-a8d5-629065fc1c4c Thread-5036::DEBUG::2015-05-18 14:35:36,063::taskManager::166::TaskManager::(clearTask) Return. Thread-5036::INFO::2015-05-18 14:35:36,063::logUtils::47::dispatcher::(wrapper) Run and protect: clearTask, Return response: None Thread-5036::DEBUG::2015-05-18 14:35:36,063::task::1185::TaskManager.Task::(prepare) Task=`a8cae188-5991-4513-a077-2f6d2bc7b3e6`::finished: None Thread-5036::DEBUG::2015-05-18 14:35:36,064::task::595::TaskManager.Task::(_updateState) Task=`a8cae188-5991-4513-a077-2f6d2bc7b3e6`::moving from state preparing -> state finished Thread-5036::DEBUG::2015-05-18 14:35:36,064::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-5036::DEBUG::2015-05-18 14:35:36,064::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-5036::DEBUG::2015-05-18 14:35:36,064::task::990::TaskManager.Task::(_decref) Task=`a8cae188-5991-4513-a077-2f6d2bc7b3e6`::ref 0 aborting False Thread-5036::DEBUG::2015-05-18 14:35:36,300::BindingXMLRPC::251::vds::(wrapper) client [192.168.30.103] flowID [4f00b66b] Thread-5036::DEBUG::2015-05-18 14:35:36,301::task::595::TaskManager.Task::(_updateState) Task=`0b2423bc-dbf2-483e-85b9-fbf6f446b305`::moving from state init -> state preparing Thread-5036::INFO::2015-05-18 14:35:36,301::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', options=None) Thread-5036::INFO::2015-05-18 14:35:36,306::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} Thread-5036::DEBUG::2015-05-18 14:35:36,306::task::1185::TaskManager.Task::(prepare) Task=`0b2423bc-dbf2-483e-85b9-fbf6f446b305`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} Thread-5036::DEBUG::2015-05-18 14:35:36,307::task::595::TaskManager.Task::(_updateState) Task=`0b2423bc-dbf2-483e-85b9-fbf6f446b305`::moving from state preparing -> state finished Thread-5036::DEBUG::2015-05-18 14:35:36,307::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-5036::DEBUG::2015-05-18 14:35:36,307::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-5036::DEBUG::2015-05-18 14:35:36,307::task::990::TaskManager.Task::(_decref) Task=`0b2423bc-dbf2-483e-85b9-fbf6f446b305`::ref 0 aborting False Thread-36::ERROR::2015-05-18 14:35:36,419::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::ERROR::2015-05-18 14:35:36,419::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 036b5575-51fa-4f14-8b05-890d7807894c Thread-36::DEBUG::2015-05-18 14:35:36,419::lvm::372::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-36::DEBUG::2015-05-18 14:35:36,420::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 036b5575-51fa-4f14-8b05-890d7807894c' (cwd None) Thread-36::DEBUG::2015-05-18 14:35:36,443::lvm::295::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "036b5575-51fa-4f14-8b05-890d7807894c" not found\n Skipping volume group 036b5575-51fa-4f14-8b05-890d7807894c\n'; <rc> = 5 Thread-36::WARNING::2015-05-18 14:35:36,444::lvm::377::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "036b5575-51fa-4f14-8b05-890d7807894c" not found', ' Skipping volume group 036b5575-51fa-4f14-8b05-890d7807894c'] Thread-36::DEBUG::2015-05-18 14:35:36,444::lvm::414::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-36::ERROR::2015-05-18 14:35:36,460::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 036b5575-51fa-4f14-8b05-890d7807894c not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) Thread-36::ERROR::2015-05-18 14:35:36,463::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',) VM Channels Listener::DEBUG::2015-05-18 14:35:36,564::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:36,564::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:36,565::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:36,565::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:36,565::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:36,565::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:37,566::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-27::DEBUG::2015-05-18 14:35:38,253::task::595::TaskManager.Task::(_updateState) Task=`7674d743-bdef-497b-8dd3-98716e8c8f94`::moving from state init -> state preparing Thread-27::INFO::2015-05-18 14:35:38,253::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='9faa9678-d0c3-4886-bb7d-201032c60901', volUUID='3fd534be-fdd3-4f3a-96f1-e8d73f2660a7', options=None) Thread-27::INFO::2015-05-18 14:35:38,257::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '42949672960', 'apparentsize': '42949672960'} Thread-27::DEBUG::2015-05-18 14:35:38,258::task::1185::TaskManager.Task::(prepare) Task=`7674d743-bdef-497b-8dd3-98716e8c8f94`::finished: {'truesize': '42949672960', 'apparentsize': '42949672960'} Thread-27::DEBUG::2015-05-18 14:35:38,258::task::595::TaskManager.Task::(_updateState) Task=`7674d743-bdef-497b-8dd3-98716e8c8f94`::moving from state preparing -> state finished Thread-27::DEBUG::2015-05-18 14:35:38,258::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-27::DEBUG::2015-05-18 14:35:38,258::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-27::DEBUG::2015-05-18 14:35:38,258::task::990::TaskManager.Task::(_decref) Task=`7674d743-bdef-497b-8dd3-98716e8c8f94`::ref 0 aborting False Thread-5036::DEBUG::2015-05-18 14:35:38,480::BindingXMLRPC::1067::vds::(wrapper) client [192.168.30.103]::call volumesList with () {} flowID [e98fff5] VM Channels Listener::DEBUG::2015-05-18 14:35:38,567::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:38,568::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:38,568::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:38,568::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:38,568::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:38,569::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-5036::DEBUG::2015-05-18 14:35:38,592::BindingXMLRPC::1074::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'RaidVolB': {'transportType': ['TCP'], 'uuid': 'd46619e9-9368-4e82-bf3a-a2377b6e85e4', 'bricks': ['ovirt-node01.stuttgart.example.net:/raidvol/volb', 'ovirt-node02.stuttgart.example.net:/raidvol/volb'], 'volumeName': 'RaidVolB', 'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'bricksInfo': [{'name': 'ovirt-node01.stuttgart.example.net:/raidvol/volb', 'hostUuid': 'de2a515f-c728-449d-b91c-d80cabe0539f'}, {'name': 'ovirt-node02.stuttgart.example.net:/raidvol/volb', 'hostUuid': '7540f5c0-c4ba-4520-bdf1-3115c10d0eea'}], 'options': {'user.cifs': 'disable', 'storage.owner-gid': '36', 'storage.owner-uid': '36', 'nfs.disable': 'on', 'auth.allow': '*'}}}} VM Channels Listener::DEBUG::2015-05-18 14:35:39,570::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:39,570::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:39,570::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:39,570::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:39,571::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:39,571::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. VM Channels Listener::DEBUG::2015-05-18 14:35:40,572::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:40,572::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:40,572::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:40,573::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:40,573::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:40,573::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-29::DEBUG::2015-05-18 14:35:41,311::task::595::TaskManager.Task::(_updateState) Task=`23f5dbfe-bfbb-47ef-b456-827361104976`::moving from state init -> state preparing Thread-29::INFO::2015-05-18 14:35:41,312::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='09e9fba1-19e4-4b51-855f-c7d80c437690', volUUID='cc0219bc-f9ae-487f-bca2-a4aa79bea718', options=None) Thread-28::DEBUG::2015-05-18 14:35:41,315::task::595::TaskManager.Task::(_updateState) Task=`e9fac93e-e64a-487a-bea7-754e164b534f`::moving from state init -> state preparing Thread-28::INFO::2015-05-18 14:35:41,315::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='12f8ef8a-8bb8-4458-b1a0-eaf9137bfec6', volUUID='29aa0417-953e-4df7-aee1-46ac96966a62', options=None) Thread-29::INFO::2015-05-18 14:35:41,317::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '38558031872', 'apparentsize': '64424509440'} Thread-29::DEBUG::2015-05-18 14:35:41,317::task::1185::TaskManager.Task::(prepare) Task=`23f5dbfe-bfbb-47ef-b456-827361104976`::finished: {'truesize': '38558031872', 'apparentsize': '64424509440'} Thread-29::DEBUG::2015-05-18 14:35:41,318::task::595::TaskManager.Task::(_updateState) Task=`23f5dbfe-bfbb-47ef-b456-827361104976`::moving from state preparing -> state finished Thread-29::DEBUG::2015-05-18 14:35:41,318::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-29::DEBUG::2015-05-18 14:35:41,319::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-29::DEBUG::2015-05-18 14:35:41,320::task::990::TaskManager.Task::(_decref) Task=`23f5dbfe-bfbb-47ef-b456-827361104976`::ref 0 aborting False Thread-28::INFO::2015-05-18 14:35:41,321::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '56300916736', 'apparentsize': '85899345920'} Thread-28::DEBUG::2015-05-18 14:35:41,321::task::1185::TaskManager.Task::(prepare) Task=`e9fac93e-e64a-487a-bea7-754e164b534f`::finished: {'truesize': '56300916736', 'apparentsize': '85899345920'} Thread-28::DEBUG::2015-05-18 14:35:41,321::task::595::TaskManager.Task::(_updateState) Task=`e9fac93e-e64a-487a-bea7-754e164b534f`::moving from state preparing -> state finished Thread-28::DEBUG::2015-05-18 14:35:41,321::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-28::DEBUG::2015-05-18 14:35:41,321::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-28::DEBUG::2015-05-18 14:35:41,322::task::990::TaskManager.Task::(_decref) Task=`e9fac93e-e64a-487a-bea7-754e164b534f`::ref 0 aborting False Thread-32::DEBUG::2015-05-18 14:35:41,375::task::595::TaskManager.Task::(_updateState) Task=`51bc3b9a-a892-404d-9b4a-2bf55b5a0d15`::moving from state init -> state preparing Thread-32::INFO::2015-05-18 14:35:41,375::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='0963f2b7-d352-4d92-91bf-4bac9ec6b660', volUUID='6c7c2109-353a-4324-8cb0-8b719876b2dc', options=None) Thread-32::INFO::2015-05-18 14:35:41,378::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '5368709120', 'apparentsize': '5368709120'} Thread-32::DEBUG::2015-05-18 14:35:41,378::task::1185::TaskManager.Task::(prepare) Task=`51bc3b9a-a892-404d-9b4a-2bf55b5a0d15`::finished: {'truesize': '5368709120', 'apparentsize': '5368709120'} Thread-32::DEBUG::2015-05-18 14:35:41,378::task::595::TaskManager.Task::(_updateState) Task=`51bc3b9a-a892-404d-9b4a-2bf55b5a0d15`::moving from state preparing -> state finished Thread-32::DEBUG::2015-05-18 14:35:41,378::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-32::DEBUG::2015-05-18 14:35:41,379::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-32::DEBUG::2015-05-18 14:35:41,379::task::990::TaskManager.Task::(_decref) Task=`51bc3b9a-a892-404d-9b4a-2bf55b5a0d15`::ref 0 aborting False Thread-31::DEBUG::2015-05-18 14:35:41,386::task::595::TaskManager.Task::(_updateState) Task=`66eaac56-77c7-4329-87a6-57a6d9763b6f`::moving from state init -> state preparing Thread-31::INFO::2015-05-18 14:35:41,387::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='c7b196de-b8a4-46c9-be72-193d765e3a1b', volUUID='286fb5c6-095c-4872-936b-343dbe4b8106', options=None) Thread-31::INFO::2015-05-18 14:35:41,390::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '3243724800', 'apparentsize': '21474836480'} Thread-31::DEBUG::2015-05-18 14:35:41,390::task::1185::TaskManager.Task::(prepare) Task=`66eaac56-77c7-4329-87a6-57a6d9763b6f`::finished: {'truesize': '3243724800', 'apparentsize': '21474836480'} Thread-31::DEBUG::2015-05-18 14:35:41,390::task::595::TaskManager.Task::(_updateState) Task=`66eaac56-77c7-4329-87a6-57a6d9763b6f`::moving from state preparing -> state finished Thread-31::DEBUG::2015-05-18 14:35:41,390::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-31::DEBUG::2015-05-18 14:35:41,391::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-31::DEBUG::2015-05-18 14:35:41,391::task::990::TaskManager.Task::(_decref) Task=`66eaac56-77c7-4329-87a6-57a6d9763b6f`::ref 0 aborting False VM Channels Listener::DEBUG::2015-05-18 14:35:41,573::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:41,574::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:41,574::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:41,574::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:41,574::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:41,575::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. VM Channels Listener::DEBUG::2015-05-18 14:35:42,576::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:42,576::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:42,576::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:42,576::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:42,577::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:42,577::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-35::DEBUG::2015-05-18 14:35:43,570::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/ovirt.hq.example.net:_var_lib_exports_iso-20140618114713/6d882c77-cdbc-48ef-ae21-1a6d45e7f8a1/dom_md/metadata bs=4096 count=1' (cwd None) VM Channels Listener::DEBUG::2015-05-18 14:35:43,579::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 38. VM Channels Listener::DEBUG::2015-05-18 14:35:43,580::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 72. VM Channels Listener::DEBUG::2015-05-18 14:35:43,581::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 49. VM Channels Listener::DEBUG::2015-05-18 14:35:43,583::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 50. VM Channels Listener::DEBUG::2015-05-18 14:35:43,584::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 54. VM Channels Listener::DEBUG::2015-05-18 14:35:43,585::vmChannels::128::vds::(_handle_unconnected) Trying to connect fileno 62. Thread-35::DEBUG::2015-05-18 14:35:43,587::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n560 bytes (560 B) copied, 0.000340492 s, 1.6 MB/s\n'; <rc> = 0 Thread-30::DEBUG::2015-05-18 14:35:43,695::task::595::TaskManager.Task::(_updateState) Task=`df8a42f5-b6d4-4059-8586-ad3d2f60f040`::moving from state init -> state preparing Thread-30::INFO::2015-05-18 14:35:43,696::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='b68d3a83-d422-4882-a3bf-14809d7ef3a1', volUUID='0a17d461-4fd6-483c-91a9-86796d4aa6bf', options=None) Thread-30::INFO::2015-05-18 14:35:43,701::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '20731297792', 'apparentsize': '21474836480'} Thread-30::DEBUG::2015-05-18 14:35:43,701::task::1185::TaskManager.Task::(prepare) Task=`df8a42f5-b6d4-4059-8586-ad3d2f60f040`::finished: {'truesize': '20731297792', 'apparentsize': '21474836480'} Thread-30::DEBUG::2015-05-18 14:35:43,701::task::595::TaskManager.Task::(_updateState) Task=`df8a42f5-b6d4-4059-8586-ad3d2f60f040`::moving from state preparing -> state finished Thread-30::DEBUG::2015-05-18 14:35:43,701::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-30::DEBUG::2015-05-18 14:35:43,701::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-30::DEBUG::2015-05-18 14:35:43,702::task::990::TaskManager.Task::(_decref) Task=`df8a42f5-b6d4-4059-8586-ad3d2f60f040`::ref 0 aborting False Thread-30::DEBUG::2015-05-18 14:35:43,702::task::595::TaskManager.Task::(_updateState) Task=`7f25c7ef-49c2-4dbf-b785-902efecf0cc3`::moving from state init -> state preparing Thread-30::INFO::2015-05-18 14:35:43,702::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='174aa201-cfea-4ad4-8b54-f04205b0830c', volUUID='9a39b0ff-e3bc-42ae-bedb-0c809e60665d', options=None) Thread-30::INFO::2015-05-18 14:35:43,705::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '18952265728', 'apparentsize': '21474836480'} Thread-30::DEBUG::2015-05-18 14:35:43,705::task::1185::TaskManager.Task::(prepare) Task=`7f25c7ef-49c2-4dbf-b785-902efecf0cc3`::finished: {'truesize': '18952265728', 'apparentsize': '21474836480'} Thread-30::DEBUG::2015-05-18 14:35:43,705::task::595::TaskManager.Task::(_updateState) Task=`7f25c7ef-49c2-4dbf-b785-902efecf0cc3`::moving from state preparing -> state finished Thread-30::DEBUG::2015-05-18 14:35:43,706::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-30::DEBUG::2015-05-18 14:35:43,706::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-30::DEBUG::2015-05-18 14:35:43,706::task::990::TaskManager.Task::(_decref) Task=`7f25c7ef-49c2-4dbf-b785-902efecf0cc3`::ref 0 aborting False Thread-30::DEBUG::2015-05-18 14:35:43,706::task::595::TaskManager.Task::(_updateState) Task=`87a5d5d2-ad55-4dd4-9a53-584831e91fe1`::moving from state init -> state preparing Thread-30::INFO::2015-05-18 14:35:43,706::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID='abc51e26-7175-4b38-b3a8-95c6928fbc2b', spUUID='b384b3da-02a6-44f3-a3f6-56751ce8c26d', imgUUID='423535cb-3a9c-41b7-a1a0-19da94b72e21', volUUID='747dfb7c-f2e6-4cdc-a76c-2c37c243ab4d', options=None) Thread-30::INFO::2015-05-18 14:35:43,709::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '25580142592', 'apparentsize': '26843545600'} Thread-30::DEBUG::2015-05-18 14:35:43,709::task::1185::TaskManager.Task::(prepare) Task=`87a5d5d2-ad55-4dd4-9a53-584831e91fe1`::finished: {'truesize': '25580142592', 'apparentsize': '26843545600'} Thread-30::DEBUG::2015-05-18 14:35:43,709::task::595::TaskManager.Task::(_updateState) Task=`87a5d5d2-ad55-4dd4-9a53-584831e91fe1`::moving from state preparing -> state finished Thread-30::DEBUG::2015-05-18 14:35:43,710::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-30::DEBUG::2015-05-18 14:35:43,710::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-30::DEBUG::2015-05-18 14:35:43,710::task::990::TaskManager.Task::(_decref) Task=`87a5d5d2-ad55-4dd4-9a53-584831e91fe1`::ref 0 aborting False (END) engine.log: ================ 2015-05-18 14:35:48,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:35:48,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:35:48,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:35:48,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:35:48,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e0b2b6e 2015-05-18 14:35:48,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling started: taskId = 5a51fe50-7803-4518-a4d8-f2a369c2490a 2015-05-18 14:35:49,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) Failed in HSMGetTaskStatusVDS method 2015-05-18 14:35:49,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling ended: taskId = 5a51fe50-7803-4518-a4d8-f2a369c2490a task status = finished 2015-05-18 14:35:49,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:35:49,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) spmStart polling ended, spm status: Free 2015-05-18 14:35:49,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=5a51fe50-7803-4518-a4d8-f2a369c2490a), log id: 621a07b5 2015-05-18 14:35:49,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, HSMClearTaskVDSCommand, log id: 621a07b5 2015-05-18 14:35:49,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3addfc90, log id: e0b2b6e 2015-05-18 14:35:49,343 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:35:49,395 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:35:49,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) [43f07021] Irs placed on server 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c failed. Proceed Failover 2015-05-18 14:35:49,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) [43f07021] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:35:49,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-46) [43f07021] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:35:49,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63b55478 2015-05-18 14:35:49,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] spmStart polling started: taskId = 85b96d7e-429c-45a5-acc3-2359d1ad4c53 2015-05-18 14:35:50,155 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 5203a336 2015-05-18 14:35:50,345 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a584232}, log id: 5203a336 2015-05-18 14:35:50,513 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 6316c75 2015-05-18 14:35:50,598 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@383d74c4}, log id: 6316c75 2015-05-18 14:35:50,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:35:50,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] spmStart polling ended: taskId = 85b96d7e-429c-45a5-acc3-2359d1ad4c53 task status = finished 2015-05-18 14:35:50,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:35:50,617 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 2d300f8a 2015-05-18 14:35:50,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] spmStart polling ended, spm status: Free 2015-05-18 14:35:50,658 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@bf0d67a0}, log id: 2d300f8a 2015-05-18 14:35:50,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=85b96d7e-429c-45a5-acc3-2359d1ad4c53), log id: 541fc1 2015-05-18 14:35:50,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] FINISH, HSMClearTaskVDSCommand, log id: 541fc1 2015-05-18 14:35:50,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-46) [43f07021] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@605bfeda, log id: 63b55478 2015-05-18 14:35:50,736 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-46) [37b8cba2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:35:50,775 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) [37b8cba2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:35:55,931 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 758f88b3 2015-05-18 14:35:56,130 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ef0011fe}, log id: 758f88b3 2015-05-18 14:35:56,297 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 2d1069c1 2015-05-18 14:35:56,381 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@971a4c5a}, log id: 2d1069c1 2015-05-18 14:35:56,397 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 2adb5e6b 2015-05-18 14:35:56,437 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@319748ff}, log id: 2adb5e6b 2015-05-18 14:36:01,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:36:01,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:36:01,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:36:01,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:01,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:01,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73b21fc3 2015-05-18 14:36:01,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling started: taskId = cbf7b7bd-2beb-4479-a1f1-a65bcff6ca81 2015-05-18 14:36:01,714 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 6973c795 2015-05-18 14:36:01,917 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@93671002}, log id: 6973c795 2015-05-18 14:36:02,160 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 673a67af 2015-05-18 14:36:02,246 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7cc34691}, log id: 673a67af 2015-05-18 14:36:02,260 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 4839bf55 2015-05-18 14:36:02,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:02,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended: taskId = cbf7b7bd-2beb-4479-a1f1-a65bcff6ca81 task status = finished 2015-05-18 14:36:02,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:02,303 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d92bb04f}, log id: 4839bf55 2015-05-18 14:36:02,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) spmStart polling ended, spm status: Free 2015-05-18 14:36:02,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=cbf7b7bd-2beb-4479-a1f1-a65bcff6ca81), log id: c94020 2015-05-18 14:36:02,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: c94020 2015-05-18 14:36:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@270252f6, log id: 73b21fc3 2015-05-18 14:36:02,436 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:02,506 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:02,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) [994a87c] Irs placed on server 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c failed. Proceed Failover 2015-05-18 14:36:02,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) [994a87c] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:02,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-3) [994a87c] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 51eb1e47 2015-05-18 14:36:02,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] spmStart polling started: taskId = a23e67b5-28e4-4572-970e-2cb1c8c589be 2015-05-18 14:36:03,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:03,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] spmStart polling ended: taskId = a23e67b5-28e4-4572-970e-2cb1c8c589be task status = finished 2015-05-18 14:36:03,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:03,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] spmStart polling ended, spm status: Free 2015-05-18 14:36:04,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=a23e67b5-28e4-4572-970e-2cb1c8c589be), log id: 79674a9b 2015-05-18 14:36:04,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] FINISH, HSMClearTaskVDSCommand, log id: 79674a9b 2015-05-18 14:36:04,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-3) [994a87c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57b706d5, log id: 51eb1e47 2015-05-18 14:36:04,030 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-3) [3121b6f0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:04,078 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) [3121b6f0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:04,777 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GlusterTasksListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 68a5fa4c 2015-05-18 14:36:04,899 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GlusterTasksListVDSCommand, return: [], log id: 68a5fa4c 2015-05-18 14:36:05,046 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GlusterTasksListVDSCommand(HostName = ovirt-node03.stuttgart.example.net, HostId = 58c3a03d-d289-45ad-a5e5-f3d37d66cca7), log id: 3dde7be5 2015-05-18 14:36:05,074 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GlusterTasksListVDSCommand, return: [], log id: 3dde7be5 2015-05-18 14:36:05,078 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GlusterTasksListVDSCommand(HostName = ovirt-node06-stgt.stuttgart.example.net, HostId = e15b95d4-96a5-40f7-97d0-9ec2fccd39f0), log id: 68d3acc2 2015-05-18 14:36:05,157 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GlusterTasksListVDSCommand, return: [], log id: 68d3acc2 2015-05-18 14:36:07,647 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 7192b532 2015-05-18 14:36:07,830 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d1d19f6}, log id: 7192b532 2015-05-18 14:36:08,002 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 40023b20 2015-05-18 14:36:08,090 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@81f414aa}, log id: 40023b20 2015-05-18 14:36:08,106 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 3901fba 2015-05-18 14:36:08,144 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8043d5d4}, log id: 3901fba 2015-05-18 14:36:13,378 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 1c08397d 2015-05-18 14:36:13,610 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@3f192277}, log id: 1c08397d 2015-05-18 14:36:13,892 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: e80f14a 2015-05-18 14:36:13,975 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b4518810}, log id: e80f14a 2015-05-18 14:36:13,995 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 3f6f3ea0 2015-05-18 14:36:14,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:36:14,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:36:14,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:36:14,429 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7da28ce6}, log id: 3f6f3ea0 2015-05-18 14:36:14,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:14,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:14,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26ddbe34 2015-05-18 14:36:14,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) spmStart polling started: taskId = f23ede06-d1ff-4398-aa8d-fec60a9b169a 2015-05-18 14:36:15,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:15,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) spmStart polling ended: taskId = f23ede06-d1ff-4398-aa8d-fec60a9b169a task status = finished 2015-05-18 14:36:15,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:15,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) spmStart polling ended, spm status: Free 2015-05-18 14:36:15,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-86) START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=f23ede06-d1ff-4398-aa8d-fec60a9b169a), log id: 1120709c 2015-05-18 14:36:15,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, HSMClearTaskVDSCommand, log id: 1120709c 2015-05-18 14:36:15,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52c0b61e, log id: 26ddbe34 2015-05-18 14:36:15,818 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:15,858 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:15,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [169f4d5c] Irs placed on server 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c failed. Proceed Failover 2015-05-18 14:36:16,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [169f4d5c] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:16,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-86) [169f4d5c] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:16,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a33371d 2015-05-18 14:36:16,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] spmStart polling started: taskId = c730e46a-57b3-4996-9916-a428c9a9ac3b 2015-05-18 14:36:17,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:17,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] spmStart polling ended: taskId = c730e46a-57b3-4996-9916-a428c9a9ac3b task status = finished 2015-05-18 14:36:17,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:17,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] spmStart polling ended, spm status: Free 2015-05-18 14:36:17,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=c730e46a-57b3-4996-9916-a428c9a9ac3b), log id: 77422494 2015-05-18 14:36:17,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] FINISH, HSMClearTaskVDSCommand, log id: 77422494 2015-05-18 14:36:17,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-86) [169f4d5c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6db1c38e, log id: 3a33371d 2015-05-18 14:36:17,260 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-86) [319dd7e5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:17,366 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) [319dd7e5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:19,702 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 39b26495 2015-05-18 14:36:19,941 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9c2a93b7}, log id: 39b26495 2015-05-18 14:36:20,111 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 67d54c4a 2015-05-18 14:36:20,194 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8a6b9152}, log id: 67d54c4a 2015-05-18 14:36:20,217 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 2e5e3f07 2015-05-18 14:36:20,255 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@409317cd}, log id: 2e5e3f07 2015-05-18 14:36:25,485 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 16543222 2015-05-18 14:36:25,670 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fd879447}, log id: 16543222 2015-05-18 14:36:25,836 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 7ac850fa 2015-05-18 14:36:25,919 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f3e7b162}, log id: 7ac850fa 2015-05-18 14:36:25,933 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 1857ade7 2015-05-18 14:36:25,970 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@11bc654c}, log id: 1857ade7 2015-05-18 14:36:27,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:36:27,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:36:27,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:36:27,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:27,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:27,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 40dee17b 2015-05-18 14:36:27,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] spmStart polling started: taskId = a80f5b5f-6507-44f7-ae03-798fb2bd7d28 2015-05-18 14:36:28,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:28,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] spmStart polling ended: taskId = a80f5b5f-6507-44f7-ae03-798fb2bd7d28 task status = finished 2015-05-18 14:36:28,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:28,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] spmStart polling ended, spm status: Free 2015-05-18 14:36:29,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=a80f5b5f-6507-44f7-ae03-798fb2bd7d28), log id: 65247d54 2015-05-18 14:36:29,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] FINISH, HSMClearTaskVDSCommand, log id: 65247d54 2015-05-18 14:36:29,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [3f3a55ac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20a417c8, log id: 40dee17b 2015-05-18 14:36:29,022 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:29,108 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:29,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [75155c35] Irs placed on server 6948da12-0b8a-4b6d-a9af-162e6c25dad3 failed. Proceed Failover 2015-05-18 14:36:29,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [75155c35] hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:29,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-24) [75155c35] starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:29,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ddc1476 2015-05-18 14:36:29,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] spmStart polling started: taskId = e30a8353-fc32-4300-adba-f51a876246b4 2015-05-18 14:36:30,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:30,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] spmStart polling ended: taskId = e30a8353-fc32-4300-adba-f51a876246b4 task status = finished 2015-05-18 14:36:30,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:30,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] spmStart polling ended, spm status: Free 2015-05-18 14:36:30,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=e30a8353-fc32-4300-adba-f51a876246b4), log id: 7dd12ea9 2015-05-18 14:36:30,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] FINISH, HSMClearTaskVDSCommand, log id: 7dd12ea9 2015-05-18 14:36:30,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-24) [75155c35] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d54458d, log id: 1ddc1476 2015-05-18 14:36:30,547 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-24) [37a2640e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:30,617 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) [37a2640e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:31,176 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 7845333d 2015-05-18 14:36:31,413 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@621cee87}, log id: 7845333d 2015-05-18 14:36:31,667 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 54f77549 2015-05-18 14:36:31,751 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c710ae5}, log id: 54f77549 2015-05-18 14:36:31,766 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 4c70d07 2015-05-18 14:36:31,799 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2136a7e7}, log id: 4c70d07 2015-05-18 14:36:37,034 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 613a7dbb 2015-05-18 14:36:37,261 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@65925efa}, log id: 613a7dbb 2015-05-18 14:36:37,512 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: b285af5 2015-05-18 14:36:37,594 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4ed42f1a}, log id: b285af5 2015-05-18 14:36:37,609 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 40264885 2015-05-18 14:36:37,643 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@baf0c0d5}, log id: 40264885 2015-05-18 14:36:40,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:36:40,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] HostName = ovirt-node01.stuttgart.example.net 2015-05-18 14:36:40,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:36:41,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [59e67616] hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:41,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [59e67616] starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:41,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 225ad910 2015-05-18 14:36:41,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] spmStart polling started: taskId = 64e3d616-9a75-411f-a4d5-f7bd770a4cb8 2015-05-18 14:36:42,148 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:42,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] spmStart polling ended: taskId = 64e3d616-9a75-411f-a4d5-f7bd770a4cb8 task status = finished 2015-05-18 14:36:42,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:42,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] spmStart polling ended, spm status: Free 2015-05-18 14:36:42,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=64e3d616-9a75-411f-a4d5-f7bd770a4cb8), log id: 6918f975 2015-05-18 14:36:42,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] FINISH, HSMClearTaskVDSCommand, log id: 6918f975 2015-05-18 14:36:42,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [59e67616] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@29ac1cac, log id: 225ad910 2015-05-18 14:36:42,286 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:42,383 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [76c93871] Correlation ID: 76c93871, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center HP_Proliant_DL180G6. Setting status to Non Responsive. 2015-05-18 14:36:42,390 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:42,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [76c93871] Irs placed on server 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c failed. Proceed Failover 2015-05-18 14:36:42,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [76c93871] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:42,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-9) [76c93871] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:42,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39ed2e0a 2015-05-18 14:36:42,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] spmStart polling started: taskId = caa54bef-a0c9-40e7-9a1c-ead9b7b10dd3 2015-05-18 14:36:42,839 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 5a5cc3b2 2015-05-18 14:36:43,075 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b94fa000}, log id: 5a5cc3b2 2015-05-18 14:36:43,341 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 74a38188 2015-05-18 14:36:43,429 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b7eeb15b}, log id: 74a38188 2015-05-18 14:36:43,444 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 71a18ff2 2015-05-18 14:36:43,486 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@694b34c3}, log id: 71a18ff2 2015-05-18 14:36:43,685 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:43,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] spmStart polling ended: taskId = caa54bef-a0c9-40e7-9a1c-ead9b7b10dd3 task status = finished 2015-05-18 14:36:43,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:43,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] spmStart polling ended, spm status: Free 2015-05-18 14:36:43,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=caa54bef-a0c9-40e7-9a1c-ead9b7b10dd3), log id: 137e2e59 2015-05-18 14:36:43,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] FINISH, HSMClearTaskVDSCommand, log id: 137e2e59 2015-05-18 14:36:43,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-9) [76c93871] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14bd85e2, log id: 39ed2e0a 2015-05-18 14:36:43,819 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-9) [26a0c311] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:43,889 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [26a0c311] Correlation ID: 26a0c311, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center HP_Proliant_DL180G6. Setting status to Non Responsive. 2015-05-18 14:36:43,895 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [26a0c311] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:48,689 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 5cf2af7b 2015-05-18 14:36:48,958 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7b61fce}, log id: 5cf2af7b 2015-05-18 14:36:49,136 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 58b3a1f1 2015-05-18 14:36:49,241 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@a2b43ca1}, log id: 58b3a1f1 2015-05-18 14:36:49,255 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 231ed250 2015-05-18 14:36:49,293 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@878a1db8}, log id: 231ed250 2015-05-18 14:36:54,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:36:54,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:36:54,249 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:36:54,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) hostFromVds::selectedVds - ovirt-node01.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:54,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) starting spm on vds ovirt-node01.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:54,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) START, SpmStartVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b7e937c 2015-05-18 14:36:54,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) spmStart polling started: taskId = 1df26d40-a863-4be2-8e4d-cc63e144b0e1 2015-05-18 14:36:54,500 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 21a76860 2015-05-18 14:36:54,732 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@236c3904}, log id: 21a76860 2015-05-18 14:36:54,898 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 697b317e 2015-05-18 14:36:54,981 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@17d1446c}, log id: 697b317e 2015-05-18 14:36:54,995 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 7d8aa4d 2015-05-18 14:36:55,030 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7c985d32}, log id: 7d8aa4d 2015-05-18 14:36:55,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:55,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) spmStart polling ended: taskId = 1df26d40-a863-4be2-8e4d-cc63e144b0e1 task status = finished 2015-05-18 14:36:55,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:55,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) spmStart polling ended, spm status: Free 2015-05-18 14:36:55,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) START, HSMClearTaskVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c, taskId=1df26d40-a863-4be2-8e4d-cc63e144b0e1), log id: 1e686f88 2015-05-18 14:36:55,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, HSMClearTaskVDSCommand, log id: 1e686f88 2015-05-18 14:36:55,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60d70516, log id: 6b7e937c 2015-05-18 14:36:55,605 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:55,686 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:36:55,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [448f1905] Irs placed on server 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c failed. Proceed Failover 2015-05-18 14:36:55,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [448f1905] hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:36:55,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-84) [448f1905] starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:36:55,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bcd36d4 2015-05-18 14:36:55,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] spmStart polling started: taskId = 8ebae0ff-b0cd-4e78-ac98-1ffa2de94c2c 2015-05-18 14:36:56,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] Failed in HSMGetTaskStatusVDS method 2015-05-18 14:36:56,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] spmStart polling ended: taskId = 8ebae0ff-b0cd-4e78-ac98-1ffa2de94c2c task status = finished 2015-05-18 14:36:56,982 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Storage domain does not exist, code = 358 2015-05-18 14:36:57,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] spmStart polling ended, spm status: Free 2015-05-18 14:36:57,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] START, HSMClearTaskVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, taskId=8ebae0ff-b0cd-4e78-ac98-1ffa2de94c2c), log id: aa83196 2015-05-18 14:36:57,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] FINISH, HSMClearTaskVDSCommand, log id: aa83196 2015-05-18 14:36:57,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-84) [448f1905] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31dc73ed, log id: 3bcd36d4 2015-05-18 14:36:57,111 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-84) [3ee62e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: b384b3da-02a6-44f3-a3f6-56751ce8c26d Type: StoragePool 2015-05-18 14:36:57,180 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-84) [3ee62e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed 2015-05-18 14:37:00,260 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 5f3899c0 2015-05-18 14:37:00,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) START, GetFileStatsVDSCommand( storagePoolId = cf165524-0971-434c-8a5e-f2d5cb877506, ignoreFailoverLimit = false), log id: 672df19 2015-05-18 14:37:00,478 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c4308c6f}, log id: 5f3899c0 2015-05-18 14:37:00,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-46) FINISH, GetFileStatsVDSCommand, return: {KNOPPIX_V5.0.1CD-2006-06-01-DE.iso={status=0, ctime=1424080000.0, size=730036224}, 5.4-RELEASE-i386-disc1.iso={status=0, ctime=1431100000.0, size=563701760}, debian-31r8-i386-netinst.iso={status=0, ctime=1424080000.0, size=110628864}, grml64-small_2013.09.iso={status=0, ctime=1424080000.0, size=188743680}, debian-7.3.0-amd64-netinst.iso={status=0, ctime=1424080000.0, size=231735296}, debian-7.8.0-amd64-netinst.iso={status=0, ctime=1424080000.0, size=232783872}, debian-7.7.0-amd64-netinst.iso={status=0, ctime=1424080000.0, size=232783872}, grml32-small_2014.03.iso={status=0, ctime=1424080000.0, size=207618048}}, log id: 672df19 2015-05-18 14:37:00,593 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-46) Finished automatic refresh process for ISO file type with success, for storage domain id 6d882c77-cdbc-48ef-ae21-1a6d45e7f8a1. 2015-05-18 14:37:00,756 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 72f57b74 2015-05-18 14:37:00,841 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@999d3275}, log id: 72f57b74 2015-05-18 14:37:00,860 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 23b84657 2015-05-18 14:37:00,900 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@106e5ec1}, log id: 23b84657 2015-05-18 14:37:02,992 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3), log id: 1bad432c 2015-05-18 14:37:03,005 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-31) HA reservation status for cluster HP_Proliant_DL180G6_Cluster_II is OK 2015-05-18 14:37:03,322 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@68b72574, log id: 1bad432c 2015-05-18 14:37:03,485 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = ovirt-node03.stuttgart.example.net, HostId = 58c3a03d-d289-45ad-a5e5-f3d37d66cca7), log id: 2b6cf052 2015-05-18 14:37:03,611 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@cf009849, log id: 2b6cf052 2015-05-18 14:37:03,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 18cc227c 2015-05-18 14:37:04,024 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-8) [1f6e11f1] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@e7361042, log id: 18cc227c 2015-05-18 14:37:05,479 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterTasksListVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3), log id: f8356a4 2015-05-18 14:37:05,593 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterTasksListVDSCommand, return: [], log id: f8356a4 2015-05-18 14:37:05,789 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterTasksListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 1707c285 2015-05-18 14:37:05,816 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterTasksListVDSCommand, return: [], log id: 1707c285 2015-05-18 14:37:05,821 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterTasksListVDSCommand(HostName = ovirt-node05-stgt.stuttgart.example.net, HostId = 4ca44033-4f16-413d-828a-cbfe4dacf891), log id: 29e4ebc3 2015-05-18 14:37:05,885 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterTasksListVDSCommand, return: [], log id: 29e4ebc3 2015-05-18 14:37:06,071 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = ovirt-node01.stuttgart.example.net, HostId = 2e8cec66-23d7-4a5c-b6f3-9758d1d87f5c), log id: 377c4636 2015-05-18 14:37:06,250 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {d46619e9-9368-4e82-bf3a-a2377b6e85e4=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d9e03a61}, log id: 377c4636 2015-05-18 14:37:06,429 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = ovirt-node04.stuttgart.example.net, HostId = 073c24e1-003f-412a-be56-0c41a435829a), log id: 2c323dc0 2015-05-18 14:37:06,577 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {e952fd41-45bf-42d9-b494-8e0195cb9756=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@bd6050c7}, log id: 2c323dc0 2015-05-18 14:37:06,603 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) START, GlusterVolumesListVDSCommand(HostName = ovirt-node07-stgt.stuttgart.example.net, HostId = 32dfc1f1-c474-4410-94c0-38776c121289), log id: 3159e468 2015-05-18 14:37:06,637 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, GlusterVolumesListVDSCommand, return: {8786357b-9d11-4c01-a34b-aee949c116e9=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9d150a0a}, log id: 3159e468 2015-05-18 14:37:07,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] 2015-05-18 14:37:07,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = ovirt-node02.stuttgart.example.net 2015-05-18 14:37:07,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command HSMGetAllTasksStatusesVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2015-05-18 14:37:07,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) hostFromVds::selectedVds - ovirt-node02.stuttgart.example.net, spmStatus Free, storage pool HP_Proliant_DL180G6 2015-05-18 14:37:07,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) starting spm on vds ovirt-node02.stuttgart.example.net, storage pool HP_Proliant_DL180G6, prevId -1, LVER -1 2015-05-18 14:37:07,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) START, SpmStartVDSCommand(HostName = ovirt-node02.stuttgart.example.net, HostId = 6948da12-0b8a-4b6d-a9af-162e6c25dad3, storagePoolId = b384b3da-02a6-44f3-a3f6-56751ce8c26d, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2cf7641 2015-05-18 14:37:07,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-70) spmStart polling started: taskId = 9370fc1f-e128-469f-afd1-67dfa4815089 (END) Thanks, Mario Am 18.05.15 um 14:21 schrieb Maor Lipchuk: > Hi Mario, > > Can u try to mount this directly from the Host? > Can u please attach the VDSM and engine logs > > Thanks, > Maor > > > ----- Original Message ----- >> From: ml(a)ohnewald.net >> To: "Maor Lipchuk" <mlipchuk(a)redhat.com&gt; >> Cc: users(a)ovirt.org >> Sent: Monday, May 18, 2015 2:36:38 PM >> Subject: Re: [ovirt-users] vdsm storage problem - maybe cache problem? >> >> Hi Maor, >> >> thanks for the quick reply. >> >> Am 18.05.15 um 13:25 schrieb Maor Lipchuk: >> >>>> Now my Question: Why does the vdsm node not know that i deleted the >>>> storage? Has the vdsm cached this mount informations? Why does it >>>> still >>>> try to access 036b5575-51fa-4f14-8b05-890d7807894c? >>> >>> >>> Yes, the vdsm use a cache for Storage Domains, you can try to >>> restart the >>> vdsmd service instead of rebooting the host. >>> >> >> I am still getting the same error. >> >> >> [root@ovirt-node01 ~]# /etc/init.d/vdsmd stop >> Shutting down vdsm daemon: >> vdsm watchdog stop [ OK ] >> vdsm: Running run_final_hooks [ OK ] >> vdsm stop [ OK ] >> [root@ovirt-node01 ~]# >> [root@ovirt-node01 ~]# >> [root@ovirt-node01 ~]# >> [root@ovirt-node01 ~]# ps aux | grep vdsmd >> root 3198 0.0 0.0 11304 740 ? S< May07 0:00 >> /bin/bash -e /usr/share/vdsm/respawn --minlifetime 10 --daemon >> --masterpid /var/run/vdsm/supervdsm_respawn.pid >> /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock >> --pidfile /var/run/vdsm/supervdsmd.pid >> root 3205 0.0 0.0 922368 26724 ? S<l May07 12:10 >> /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile >> /var/run/vdsm/svdsm.sock --pidfile /var/run/vdsm/supervdsmd.pid >> root 15842 0.0 0.0 103248 900 pts/0 S+ 13:35 0:00 grep >> vdsmd >> >> >> [root@ovirt-node01 ~]# /etc/init.d/vdsmd start >> initctl: Job is already running: libvirtd >> vdsm: Running mkdirs >> vdsm: Running configure_coredump >> vdsm: Running configure_vdsm_logs >> vdsm: Running run_init_hooks >> vdsm: Running gencerts >> vdsm: Running check_is_configured >> libvirt is already configured for vdsm >> sanlock service is already configured >> vdsm: Running validate_configuration >> SUCCESS: ssl configured to true. No conflicts >> vdsm: Running prepare_transient_repository >> vdsm: Running syslog_available >> vdsm: Running nwfilter >> vdsm: Running dummybr >> vdsm: Running load_needed_modules >> vdsm: Running tune_system >> vdsm: Running test_space >> vdsm: Running test_lo >> vdsm: Running restore_nets >> vdsm: Running unified_network_persistence_upgrade >> vdsm: Running upgrade_300_nets >> Starting up vdsm daemon: >> vdsm start [ OK ] >> [root@ovirt-node01 ~]# >> >> [root@ovirt-node01 ~]# grep ERROR /var/log/vdsm/vdsm.log | tail -n 20 >> Thread-13::ERROR::2015-05-18 >> 13:35:03,631::sdc::137::Storage.StorageDomainCache::(_findDomain) >> looking for unfetched domain abc51e26-7175-4b38-b3a8-95c6928fbc2b >> Thread-13::ERROR::2015-05-18 >> 13:35:03,632::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >> >> looking for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b >> Thread-36::ERROR::2015-05-18 >> 13:35:11,607::sdc::137::Storage.StorageDomainCache::(_findDomain) >> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:11,621::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >> >> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:11,960::sdc::143::Storage.StorageDomainCache::(_findDomain) domain >> 036b5575-51fa-4f14-8b05-890d7807894c not found >> Thread-36::ERROR::2015-05-18 >> 13:35:11,960::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) >> >> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c >> monitoring information >> Thread-36::ERROR::2015-05-18 >> 13:35:21,962::sdc::137::Storage.StorageDomainCache::(_findDomain) >> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:21,965::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >> >> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:22,068::sdc::143::Storage.StorageDomainCache::(_findDomain) domain >> 036b5575-51fa-4f14-8b05-890d7807894c not found >> Thread-36::ERROR::2015-05-18 >> 13:35:22,072::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) >> >> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c >> monitoring information >> Thread-15::ERROR::2015-05-18 >> 13:35:33,821::task::866::TaskManager.Task::(_setError) >> Task=`54bdfc77-f63a-493b-b24e-e5a3bc4977bb`::Unexpected error >> Thread-15::ERROR::2015-05-18 >> 13:35:33,864::dispatcher::65::Storage.Dispatcher.Protect::(run) >> {'status': {'message': "Unknown pool id, pool not connected: >> ('b384b3da-02a6-44f3-a3f6-56751ce8c26d',)", 'code': 309}} >> Thread-13::ERROR::2015-05-18 >> 13:35:33,930::sdc::137::Storage.StorageDomainCache::(_findDomain) >> looking for unfetched domain abc51e26-7175-4b38-b3a8-95c6928fbc2b >> Thread-15::ERROR::2015-05-18 >> 13:35:33,928::task::866::TaskManager.Task::(_setError) >> Task=`fe9bb0fa-cf1e-4b21-af00-0698c6d1718f`::Unexpected error >> Thread-13::ERROR::2015-05-18 >> 13:35:33,932::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >> >> looking for domain abc51e26-7175-4b38-b3a8-95c6928fbc2b >> Thread-15::ERROR::2015-05-18 >> 13:35:33,978::dispatcher::65::Storage.Dispatcher.Protect::(run) >> {'status': {'message': 'Not SPM: ()', 'code': 654}} >> Thread-36::ERROR::2015-05-18 >> 13:35:41,117::sdc::137::Storage.StorageDomainCache::(_findDomain) >> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:41,131::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >> >> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c >> Thread-36::ERROR::2015-05-18 >> 13:35:41,452::sdc::143::Storage.StorageDomainCache::(_findDomain) domain >> 036b5575-51fa-4f14-8b05-890d7807894c not found >> Thread-36::ERROR::2015-05-18 >> 13:35:41,453::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) >> >> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c >> monitoring information >> >> >> Thanks, >> Mario >> >>