[ovirt-users] vdsm storage problem - maybe cache problem?

Maor Lipchuk mlipchuk at redhat.com
Mon May 18 07:25:06 EDT 2015 

Hi ml,

See my comments inline

Regards,
Maor


----- Original Message -----
> From: ml at ohnewald.net
> To: users at ovirt.org
> Sent: Monday, May 18, 2015 2:16:23 PM
> Subject: [ovirt-users] vdsm storage problem - maybe cache problem?
> 
> Hello List,
> 
> i did a simple update in my CentOS 6.6 Engine (3.5) and then i rebootet it.
> 
> After that i ran into trouble.
> 
> My NFS mounts from my clients to my engine got stuck.
> So i unmounted them manually with:
> umount -f -l <path to my mount point>
> 
> Which worked. I hoped vdsm would remount it again. With no luck.
> So i deleted the nfs export and iso domain from the cluster.
> 
> Now i am getting this error in my vdsm log:
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,682::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,683::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,717::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,717::sp::329::Storage.StoragePool::(startSpm) Unexpected error
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,717::sp::330::Storage.StoragePool::(startSpm) failed: Storage
> domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',)
> 2c6b4422-7faa-4847-ab30-fc713d7012af::ERROR::2015-05-18
> 13:13:25,928::task::866::TaskManager.Task::(_setError)
> Task=`2c6b4422-7faa-4847-ab30-fc713d7012af`::Unexpected error
> Thread-37::ERROR::2015-05-18
> 13:13:35,683::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:35,683::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:35,720::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> Thread-37::ERROR::2015-05-18
> 13:13:35,720::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c
> monitoring information
> Thread-482::ERROR::2015-05-18
> 13:13:40,062::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-482::ERROR::2015-05-18
> 13:13:40,063::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-482::ERROR::2015-05-18
> 13:13:40,147::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,149::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,152::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,191::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,191::sp::288::Storage.StoragePool::(startSpm) Backup domain
> validation failed
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,193::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,193::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,228::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,228::sp::329::Storage.StoragePool::(startSpm) Unexpected error
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,228::sp::330::Storage.StoragePool::(startSpm) failed: Storage
> domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',)
> 3140b81f-a434-4877-9a34-3923505e4a1f::ERROR::2015-05-18
> 13:13:40,932::task::866::TaskManager.Task::(_setError)
> Task=`3140b81f-a434-4877-9a34-3923505e4a1f`::Unexpected error
> Thread-37::ERROR::2015-05-18
> 13:13:45,721::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:45,721::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:45,757::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> Thread-37::ERROR::2015-05-18
> 13:13:45,758::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c
> monitoring information
> Thread-501::ERROR::2015-05-18
> 13:13:54,714::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-501::ERROR::2015-05-18
> 13:13:54,715::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-501::ERROR::2015-05-18
> 13:13:54,751::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,751::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,752::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,787::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,788::sp::288::Storage.StoragePool::(startSpm) Backup domain
> validation failed
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,790::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,790::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,824::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,824::sp::329::Storage.StoragePool::(startSpm) Unexpected error
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,824::sp::330::Storage.StoragePool::(startSpm) failed: Storage
> domain does not exist: ('036b5575-51fa-4f14-8b05-890d7807894c',)
> 5893d839-b69b-4d25-816a-67f2c9f24e12::ERROR::2015-05-18
> 13:13:54,937::task::866::TaskManager.Task::(_setError)
> Task=`5893d839-b69b-4d25-816a-67f2c9f24e12`::Unexpected error
> Thread-37::ERROR::2015-05-18
> 13:13:55,758::sdc::137::Storage.StorageDomainCache::(_findDomain)
> looking for unfetched domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:55,759::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
> looking for domain 036b5575-51fa-4f14-8b05-890d7807894c
> Thread-37::ERROR::2015-05-18
> 13:13:55,794::sdc::143::Storage.StorageDomainCache::(_findDomain) domain
> 036b5575-51fa-4f14-8b05-890d7807894c not found
> Thread-37::ERROR::2015-05-18
> 13:13:55,795::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain)
> Error while collecting domain 036b5575-51fa-4f14-8b05-890d7807894c
> monitoring information
> 
> 
> In older engine logs i was able to find out what id
> 036b5575-51fa-4f14-8b05-890d7807894c is:
> ----------------------------
> 
> 2015-05-18 10:58:55,397 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-43) domain
> 036b5575-51fa-4f14-8b05-890d7807894c:EXPORT2 in problem. vds:
> ovirt-node01.stuttgart.imos.net
> 
> 
> Now my Question: Why does the vdsm node not know that i deleted the
> storage? Has the vdsm cached this mount informations? Why does it still
> try to access 036b5575-51fa-4f14-8b05-890d7807894c?


Yes, the vdsm use a cache for Storage Domains, you can try to restart the vdsmd service instead of rebooting the host.

> 
> I can NOT reboot my node because i have active running vms on it.
> 
> Thanks,
> Mario
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>

More information about the Users mailing list