6:06 p.m.
Hi,
I just rebooted my NFS server which hosts a ISO domain which is used by
RHEV 3.1 and oVirt 3.2. In both environments the ISO domain became
inactive and I tried to activate it again.
In RHEV this worked fine, but in oVirt it didn't.
Just for your information: both - RHEV and oVirt use "Local on host" as
datacenter storage type.
I normally use a local NFS server and mount the share locally (dc with
storage type NFS) as "Local on host" is making troubles in most of my
setups (in both - oVirt and RHEV) and the issues are mainly missing
(sometimes lost) volume groups.
As my oVirt 3.2 setup is a testing environment and my test vms are still
running, I'll keep it in this state for finding a solution for this. As
said, have seen issues like this more then one time before, so maybe
this is a bug and need some attention (can also open a bug report if
needed).
vdsm.log is telling me that the storage domain doesn't exist (as
complained by vdsm daemon):
2013-05-07 16:44:18,493 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(pool-3-thread-47) [210a0bcb] START, ActivateStorageDomainVDSCommand( storagePoolId =
484e62d7-7a01-4b5e-aec8-59d366100281, ignoreFailoverLimit = false, compatabilityVersion =
null, storageDomainId = a4c43175-ce34-49a5-8608-cac573bf7647), log id: 33c54af7
2013-05-07 16:44:20,770 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
(pool-3-thread-47) [210a0bcb] Failed in ActivateStorageDomainVDS method
2013-05-07 16:44:20,781 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
(pool-3-thread-47) [210a0bcb] Error code StorageDomainDoesNotExist and
error message IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
('a4c43175-ce34-49a5-8608-cac573bf7647',)
2013-05-07 16:44:20,791 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(pool-3-thread-47) [210a0bcb]
IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSErrorException:
IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
('a4c43175-ce34-49a5-8608-cac573bf7647',)
More interesting is the behavior of my host (CentOS 6.4) which tries to
find a logical volume and is doing some iSCSI-scans:
Thread-224450::DEBUG::2013-05-07
16:44:19,333::task::957::TaskManager.Task::(_decref)
Task=`ab00697e-32df-43b1-822b-a94bef55909d`::ref 0 aborting False
Thread-1129392::DEBUG::2013-05-07
16:44:20,555::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo
-n /sbin/multipath' (cwd None)
Thread-1129392::DEBUG::2013-05-07
16:44:20,608::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> =
''; <rc> = 0
Thread-1129392::DEBUG::2013-05-07
16:44:20,609::lvm::477::OperationMutex::(_invalidateAllPvs) Operation
'lvm invalidate operation' got the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,609::lvm::479::OperationMutex::(_invalidateAllPvs) Operation
'lvm invalidate operation' released the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,609::lvm::488::OperationMutex::(_invalidateAllVgs) Operation
'lvm invalidate operation' got the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,610::lvm::490::OperationMutex::(_invalidateAllVgs) Operation
'lvm invalidate operation' released the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,610::lvm::508::OperationMutex::(_invalidateAllLvs) Operation
'lvm invalidate operation' got the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,610::lvm::510::OperationMutex::(_invalidateAllLvs) Operation
'lvm invalidate operation' released the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,610::misc::1064::SamplingMethod::(__call__) Returning last
result
Thread-1129392::DEBUG::2013-05-07
16:44:20,610::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' got the operation mutex
Thread-1129392::DEBUG::2013-05-07
16:44:20,612::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo
-n /sbin/lvm vgs --config " devices { preferred_names = [\
\"^/dev/mapper/\\"
] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } back
up { retain_min = 50 retain_days = 0 } " --noheadings --units b
--nosuffix --separator | -o
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_fr
ee a4c43175-ce34-49a5-8608-cac573bf7647' (cwd None)
Thread-1129392::DEBUG::2013-05-07
16:44:20,759::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> =
' Volume group "a4c43175-ce34-49a5-8608-cac573bf7647" not found\n';
<rc> =
5
Thread-1129392::WARNING::2013-05-07
16:44:20,760::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
[' Volume group "a4c43175-ce34-49a5-8608-cac573bf7647" not found']
Thread-1129392::DEBUG::2013-05-07
16:44:20,760::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' released the operation mutex
Thread-1129392::ERROR::2013-05-07
16:44:20,767::task::833::TaskManager.Task::(_setError)
Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 840, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1144, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper
return f(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1042, in activateSD
dom = sdCache.produce(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 97, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('a4c43175-ce34-49a5-8608-cac573bf7647',)
Thread-1129392::DEBUG::2013-05-07
16:44:20,768::task::852::TaskManager.Task::(_run)
Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::Task._run:
589ebcc8-2255-4a29-b995-5e76f3697ec8
('a4c43175-ce34-49a5-8608-cac573bf7647',
'484e62d7-7a01-4b5e-aec8-59d366100281') {} failed - stopping task
Thread-1129392::DEBUG::2013-05-07
16:44:20,768::task::1177::TaskManager.Task::(stop)
Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::stopping in state preparing
(force False)
Thread-1129392::DEBUG::2013-05-07
16:44:20,768::task::957::TaskManager.Task::(_decref)
Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::ref 1 aborting True
Thread-1129392::INFO::2013-05-07
16:44:20,769::task::1134::TaskManager.Task::(prepare)
Task=`589ebcc8-2255-4a29-b995-5e76f3697ec8`::aborting: Task is aborted:
'Storage domain does not exist' - code 358
It's clear to me that my host can't find a volume group named
a4c43175-ce34-49a5-8608-cac573bf7647, because this is the ID of my ISO
domain which resists on NFS and not iSCSI or FC or local on host.
I use RPM for RHEL from ovirt-stable repository on my CentOS 6.4 host:
[root@centos-hyp01 ~]# yum list vdsm* | grep '@'
vdsm.x86_64 4.10.3-10.el6
@ovirt-stable
vdsm-cli.noarch 4.10.3-10.el6
@ovirt-stable
vdsm-gluster.noarch 4.10.3-10.el6
@ovirt-stable
vdsm-python.x86_64 4.10.3-10.el6
@ovirt-stable
vdsm-xmlrpc.noarch 4.10.3-10.el6
@ovirt-stable
Please let me know if you need further information.
Thanks a lot!
--
Best Regards,
René Koch
Senior Solution Architect
============================================
ovido gmbh - "Das Linux Systemhaus"
Brünner Straße 163, A-1210 Wien
Phone: +43 720 / 530 670
Mobile: +43 660 / 512 21 31
E-Mail: r.koch(a)ovido.at
============================================