[Users] Fwd: AW: storage domain reactivate not working
Rami Vaknin
rvaknin at redhat.com
Thu Mar 29 15:57:08 UTC 2012
Rene, VDSM can't read the storage domain's metadata, the problem is that
vdsm tries to read the metadata using 'dd' command which applies to the
old version of storage domains as in the new format the metadata is
saved as vg tags. Are you using storage domain version lower that V2?
Can you attach the full log?
Saggi, any thoughts on that?
-------- Original Message --------
Subject: AW: [Users] storage domain reactivate not working
Date: Thu, 29 Mar 2012 06:33:27 -0400
From: Rene Rosenberger <r.rosenberger at netbiscuits.com>
To: rvaknin at redhat.com <rvaknin at redhat.com>, users at ovirt.org
<users at ovirt.org>
Hi,
not sure if the logs i posted is waht you need. The thing ist hat the
iscsi target is connected but in web gui it is locked. Can I unlock it?
Regards, rene
*Von:*Rene Rosenberger
*Gesendet:* Donnerstag, 29. März 2012 12:00
*An:* Rene Rosenberger; rvaknin at redhat.com; users at ovirt.org
*Betreff:* AW: [Users] storage domain reactivate not working
Hi,
This is the roor message:
Thread-5448::DEBUG::2012-03-29
11:57:46,310::misc::1032::SamplingMethod::(__call__) Returning last result
Thread-5448::DEBUG::2012-03-29
11:57:46,313::lvm::349::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' got the operation mutex
Thread-5448::DEBUG::2012-03-29
11:57:46,322::lvm::284::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
/sbin/lvm vgs --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 filter = [
\\"a%360014052dd702d2defc8d459adba02dc|360014057fda80efdcae4d414eda829d7%\\
<file:///%5C%5C%22a%25360014052dd702d2defc8d459adba02dc%7C360014057fda80efdcae4d414eda829d7%25%5C>",
\\"r%.*%\\ <file:///%5C%5C%22r%25.*%25%5C>" ] }global
{locking_type=1prioritise_write_locks=1wait_for_locks=1 }backup
{retain_min = 50retain_days = 0 } " --noheadings --units b --nosuffix
--separator | -o
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free
8ed25a57-f53a-4cf0-bb92-781f3ce36a48' (cwd None)
Thread-5448::DEBUG::2012-03-29
11:57:47,096::lvm::284::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
'/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after 0 of
4096 at 2147483582464: Input/output
error\n/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after
0 of 4096 at 2147483639808: Input/output
error\n/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after
0 of 4096 at 0: Input/output error\nWARNING: Error counts reached a
limit of 3. Device /dev/mapper/360014057fda80efdcae4d414eda829d7 was
disabled\n'; <rc> = 0
Thread-5448::DEBUG::2012-03-29
11:57:47,105::lvm::376::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' released the operation mutex
Thread-5448::DEBUG::2012-03-29
11:57:47,107::persistentDict::175::Storage.PersistentDict::(__init__)
Created a persistant dict with LvMetadataRW backend
Thread-5448::DEBUG::2012-03-29
11:57:47,110::blockSD::177::Storage.Misc.excCmd::(readlines) '/bin/dd
iflag=direct skip=0 bs=2048
if=/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata count=1' (cwd None)
Thread-5448::DEBUG::2012-03-29
11:57:47,155::blockSD::177::Storage.Misc.excCmd::(readlines) FAILED:
<err> = "/bin/dd: reading
`/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata': Input/output
error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied,
0.000525019 s, 0.0 kB/s\n"; <rc> = 1
Thread-5448::ERROR::2012-03-29
11:57:47,158::sdc::113::Storage.StorageDomainCache::(_findDomain) Error
while looking for domain `8ed25a57-f53a-4cf0-bb92-781f3ce36a48`
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain
return mod.findDomain(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1051, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 241, in __init__
metadata = selectMetadata(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 210, in selectMetadata
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 51, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 95, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 92, in __iter__
return ifilter(lambda k: k in self._validatorDict, self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 209, in __iter__
with self._accessWrapper():
File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 137, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh
lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 177, in readlines
m = misc.readblockSUDO(self.metavol, self._offset, self._size)
File "/usr/share/vdsm/storage/misc.py", line 307, in readblockSUDO
raise se.MiscBlockReadException(name, offset, size)
MiscBlockReadException: Internal block device read failure:
'name=/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata, offset=0,
size=2048'
Thread-5448::DEBUG::2012-03-29
11:57:47,177::resourceManager::538::ResourceManager::(releaseResource)
Trying to release resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a'
Thread-5448::DEBUG::2012-03-29
11:57:47,180::resourceManager::553::ResourceManager::(releaseResource)
Released resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a' (0
active users)
Thread-5448::DEBUG::2012-03-29
11:57:47,181::resourceManager::558::ResourceManager::(releaseResource)
Resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a' is free, finding
out if anyone is waiting for it.
Thread-5448::DEBUG::2012-03-29
11:57:47,183::resourceManager::565::ResourceManager::(releaseResource)
No one is waiting for resource
'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a', Clearing records.
Thread-5448::ERROR::2012-03-29
11:57:47,185::task::853::TaskManager.Task::(_setError)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 861, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 813, in connectStoragePool
return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID,
masterVersion, options)
File "/usr/share/vdsm/storage/hsm.py", line 855, in _connectStoragePool
res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 641, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1107, in __rebuild
self.masterDomain = self.getMasterDomain(msdUUID=msdUUID,
masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1442, in getMasterDomain
raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain:
'spUUID=13080edc-77ea-11e1-b6a4-525400c49d2a,
msdUUID=8ed25a57-f53a-4cf0-bb92-781f3ce36a48'
Thread-5448::DEBUG::2012-03-29
11:57:47,188::task::872::TaskManager.Task::(_run)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Task._run:
838b7ca3-9f79-4c87-a2f0-12cad48cc127
('13080edc-77ea-11e1-b6a4-525400c49d2a', 1,
'13080edc-77ea-11e1-b6a4-525400c49d2a',
'8ed25a57-f53a-4cf0-bb92-781f3ce36a48', 1) {} failed - stopping task
Thread-5448::DEBUG::2012-03-29
11:57:47,190::task::1199::TaskManager.Task::(stop)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::stopping in state preparing
(force False)
Thread-5448::DEBUG::2012-03-29
11:57:47,192::task::978::TaskManager.Task::(_decref)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::ref 1 aborting True
Thread-5448::INFO::2012-03-29
11:57:47,193::task::1157::TaskManager.Task::(prepare)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::aborting: Task is aborted:
'Cannot find master domain' - code 304
Thread-5448::DEBUG::2012-03-29
11:57:47,195::task::1162::TaskManager.Task::(prepare)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Prepare: aborted: Cannot
find master domain
Thread-5448::DEBUG::2012-03-29
11:57:47,197::task::978::TaskManager.Task::(_decref)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::ref 0 aborting True
Thread-5448::DEBUG::2012-03-29
11:57:47,198::task::913::TaskManager.Task::(_doAbort)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Task._doAbort: force False
Thread-5448::DEBUG::2012-03-29
11:57:47,200::resourceManager::844::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-5448::DEBUG::2012-03-29
11:57:47,201::task::588::TaskManager.Task::(_updateState)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::moving from state preparing
-> state aborting
Thread-5448::DEBUG::2012-03-29
11:57:47,203::task::537::TaskManager.Task::(__state_aborting)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::_aborting: recover policy none
Thread-5448::DEBUG::2012-03-29
11:57:47,205::task::588::TaskManager.Task::(_updateState)
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::moving from state aborting
-> state failed
Thread-5448::DEBUG::2012-03-29
11:57:47,206::resourceManager::809::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-5448::DEBUG::2012-03-29
11:57:47,208::resourceManager::844::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-5448::ERROR::2012-03-29
11:57:47,209::dispatcher::89::Storage.Dispatcher.Protect::(run)
{'status': {'message': "Cannot find master domain:
'spUUID=13080edc-77ea-11e1-b6a4-525400c49d2a,
msdUUID=8ed25a57-f53a-4cf0-bb92-781f3ce36a48'", 'code': 304}}
Thread-5453::DEBUG::2012-03-29
11:57:48,736::task::588::TaskManager.Task::(_updateState)
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::moving from state init ->
state preparing
Thread-5453::INFO::2012-03-29
11:57:48,739::logUtils::37::dispatcher::(wrapper) Run and protect:
repoStats(options=None)
Thread-5453::INFO::2012-03-29
11:57:48,740::logUtils::39::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {}
Thread-5453::DEBUG::2012-03-29
11:57:48,742::task::1172::TaskManager.Task::(prepare)
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::finished: {}
Thread-5453::DEBUG::2012-03-29
11:57:48,743::task::588::TaskManager.Task::(_updateState)
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::moving from state preparing
-> state finished
Thread-5453::DEBUG::2012-03-29
11:57:48,745::resourceManager::809::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-5453::DEBUG::2012-03-29
11:57:48,746::resourceManager::844::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-5453::DEBUG::2012-03-29
11:57:48,748::task::978::TaskManager.Task::(_decref)
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::ref 0 aborting False
Regards, rene
*Von:*users-bounces at ovirt.org [mailto:users-bounces at ovirt.org] *Im
Auftrag von *Rene Rosenberger
*Gesendet:* Donnerstag, 29. März 2012 11:47
*An:* rvaknin at redhat.com; users at ovirt.org
*Betreff:* Re: [Users] storage domain reactivate not working
Hi,
i did:
[root at KVM-DMZ-04 vdsm]# iscsiadm -m node -T
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d -u
Logging out of session [sid: 2, target:
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal:
192.168.xxx.xxx,3260]
Logout of [sid: 2, target:
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal:
192.168.xxx.xxx,3260] successful.
[root at KVM-DMZ-04 vdsm]# iscsiadm -m node -T
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d -l
Logging in to [iface: default, target:
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal:
192.168.xxx.xxx,3260] (multiple)
Login to [iface: default, target:
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal:
192.168.xxx.xxx,3260] successful.
But the data center is locked in webinterface. System -> default ->
storage -> DMZ-NAS-01 -> Data Center = locked
Regards, rene
*Von:*Rami Vaknin [mailto:rvaknin at redhat.com]
*Gesendet:* Donnerstag, 29. März 2012 11:32
*An:* Rene Rosenberger; users at ovirt.org
*Betreff:* Re: [Users] storage domain reactivate not working
On 03/29/2012 11:28 AM, Rene Rosenberger wrote:
Hi,
ihave rebootet my iscsi device without mainenance mode. Now it is
inactive. When I want to reactivate it again it fails. What can I do?
Depends why it fails, vdsm log can help.
You can check whether or not you're connected from the hosts to the
iscsi target or reconnect using:
iscsiadm -m discoverydb --discover -t st -p your_iscsi_server_fqdn
iscsiadm -m node -T your_target_name -l
Than try to activate it.
Regards, rene
_______________________________________________
Users mailing list
Users at ovirt.org <mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20120329/a6d71f9b/attachment-0001.html>
More information about the Users
mailing list