Hi, I just rebooted my NFS server which hosts a ISO domain which is used by RHEV 3.1 and oVirt 3.2. In both environments the ISO domain became inactive and I tried to activate it again. In RHEV this worked fine, but in oVirt it didn't. Just for your information: both - RHEV and oVirt use "Local on host" as datacenter storage type. I normally use a local NFS server and mount the share locally (dc with storage type NFS) as "Local on host" is making troubles in most of my setups (in both - oVirt and RHEV) and the issues are mainly missing (sometimes lost) volume groups. As my oVirt 3.2 setup is a testing environment and my test vms are still running, I'll keep it in this state for finding a solution for this. As said, have seen issues like this more then one time before, so maybe this is a bug and need some attention (can also open a bug report if needed). vdsm.log is telling me that the storage domain doesn't exist (as complained by vdsm daemon): 2013-05-07 16:44:18,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-3-thread-47) [210a0bcb] START, ActivateStorageDomainVDSCommand( storagePoolId = 484e62d7-7a01-4b5e-aec8-59d366100281, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a4c43175-ce34-49a5-8608-cac573bf7647), log id: 33c54af7 2013-05-07 16:44:20,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) [210a0bcb] Failed in ActivateStorageDomainVDS method 2013-05-07 16:44:20,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) [210a0bcb] Error code StorageDomainDoesNotExist and error message IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('a4c43175-ce34-49a5-8608-cac573bf7647',) 2013-05-07 16:44:20,791 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [210a0bcb] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: ('a4c43175-ce34-49a5-8608-cac573bf7647',) More interesting is the behavior of my host (CentOS 6.4) which tries to find a logical volume and is doing some iSCSI-scans: Thread-224450::DEBUG::2013-05-07 16:44:19,333::task::957::TaskManager.Task::(_decref) Task=`ab00697e-32df-43b1-822b-a94bef55909d`::ref 0 aborting False Thread-1129392::DEBUG::2013-05-07 16:44:20,555::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-1129392::DEBUG::2013-05-07 16:44:20,608::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-1129392::DEBUG::2013-05-07 16:44:20,609::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,609::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,609::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,610::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,610::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,610::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,610::misc::1064::SamplingMethod::(__call__) Returning last result Thread-1129392::DEBUG::2013-05-07 16:44:20,610::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-1129392::DEBUG::2013-05-07 16:44:20,612::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\ \"^/dev/mapper/\\" ] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } back up { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_fr ee a4c43175-ce34-49a5-8608-cac573bf7647' (cwd None) Thread-1129392::DEBUG::2013-05-07 16:44:20,759::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "a4c43175-ce34-49a5-8608-cac573bf7647" not found\n'; <rc> = 5 Thread-1129392::WARNING::2013-05-07 16:44:20,760::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a4c43175-ce34-49a5-8608-cac573bf7647" not found'] Thread-1129392::DEBUG::2013-05-07 16:44:20,760::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-1129392::ERROR::2013-05-07 16:44:20,767::task::833::TaskManager.Task::(_setError) Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1144, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper return f(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1042, in activateSD dom = sdCache.produce(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 97, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('a4c43175-ce34-49a5-8608-cac573bf7647',) Thread-1129392::DEBUG::2013-05-07 16:44:20,768::task::852::TaskManager.Task::(_run) Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::Task._run: 589ebcc8-2255-4a29-b995-5e76f3697ec8 ('a4c43175-ce34-49a5-8608-cac573bf7647', '484e62d7-7a01-4b5e-aec8-59d366100281') {} failed - stopping task Thread-1129392::DEBUG::2013-05-07 16:44:20,768::task::1177::TaskManager.Task::(stop) Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::stopping in state preparing (force False) Thread-1129392::DEBUG::2013-05-07 16:44:20,768::task::957::TaskManager.Task::(_decref) Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::ref 1 aborting True Thread-1129392::INFO::2013-05-07 16:44:20,769::task::1134::TaskManager.Task::(prepare) Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::aborting: Task is aborted: 'Storage domain does not exist' - code 358 It's clear to me that my host can't find a volume group named a4c43175-ce34-49a5-8608-cac573bf7647, because this is the ID of my ISO domain which resists on NFS and not iSCSI or FC or local on host. I use RPM for RHEL from ovirt-stable repository on my CentOS 6.4 host: [root@centos-hyp01 ~]# yum list vdsm* | grep '@' vdsm.x86_64 4.10.3-10.el6 @ovirt-stable vdsm-cli.noarch 4.10.3-10.el6 @ovirt-stable vdsm-gluster.noarch 4.10.3-10.el6 @ovirt-stable vdsm-python.x86_64 4.10.3-10.el6 @ovirt-stable vdsm-xmlrpc.noarch 4.10.3-10.el6 @ovirt-stable Please let me know if you need further information. Thanks a lot! -- Best Regards, René Koch Senior Solution Architect ============================================ ovido gmbh - "Das Linux Systemhaus" Brünner Straße 163, A-1210 Wien Phone: +43 720 / 530 670 Mobile: +43 660 / 512 21 31 E-Mail: r.koch(a)ovido.at ============================================