[Users] Fwd: AW: storage domain reactivate not working

Rami Vaknin rvaknin at redhat.com
Thu Mar 29 15:57:08 UTC 2012


Rene, VDSM can't read the storage domain's metadata, the problem is that 
vdsm tries to read the metadata using 'dd' command which applies to the 
old version of storage domains as in the new format the metadata is 
saved as vg tags. Are you using storage domain version lower that V2? 
Can you attach the full log?

Saggi, any thoughts on that?

-------- Original Message --------
Subject: 	AW: [Users] storage domain reactivate not working
Date: 	Thu, 29 Mar 2012 06:33:27 -0400
From: 	Rene Rosenberger <r.rosenberger at netbiscuits.com>
To: 	rvaknin at redhat.com <rvaknin at redhat.com>, users at ovirt.org 
<users at ovirt.org>



Hi,

not sure if the logs i posted is waht you need. The thing ist hat the 
iscsi target is connected but in web gui it is locked. Can I unlock it?

Regards, rene

*Von:*Rene Rosenberger
*Gesendet:* Donnerstag, 29. März 2012 12:00
*An:* Rene Rosenberger; rvaknin at redhat.com; users at ovirt.org
*Betreff:* AW: [Users] storage domain reactivate not working

Hi,

This is the roor message:

Thread-5448::DEBUG::2012-03-29 
11:57:46,310::misc::1032::SamplingMethod::(__call__) Returning last result

Thread-5448::DEBUG::2012-03-29 
11:57:46,313::lvm::349::OperationMutex::(_reloadvgs) Operation 'lvm 
reload operation' got the operation mutex

Thread-5448::DEBUG::2012-03-29 
11:57:46,322::lvm::284::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n 
/sbin/lvm vgs --config " devices { preferred_names = 
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 
disable_after_error_count=3 filter = [ 
\\"a%360014052dd702d2defc8d459adba02dc|360014057fda80efdcae4d414eda829d7%\\ 
<file:///%5C%5C%22a%25360014052dd702d2defc8d459adba02dc%7C360014057fda80efdcae4d414eda829d7%25%5C>", 
\\"r%.*%\\ <file:///%5C%5C%22r%25.*%25%5C>" ] }global 
{locking_type=1prioritise_write_locks=1wait_for_locks=1 }backup 
{retain_min = 50retain_days = 0 } " --noheadings --units b --nosuffix 
--separator | -o 
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 
8ed25a57-f53a-4cf0-bb92-781f3ce36a48' (cwd None)

Thread-5448::DEBUG::2012-03-29 
11:57:47,096::lvm::284::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = 
'/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after 0 of 
4096 at 2147483582464: Input/output 
error\n/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after 
0 of 4096 at 2147483639808: Input/output 
error\n/dev/mapper/360014057fda80efdcae4d414eda829d7: read failed after 
0 of 4096 at 0: Input/output error\nWARNING: Error counts reached a 
limit of 3. Device /dev/mapper/360014057fda80efdcae4d414eda829d7 was 
disabled\n'; <rc> = 0

Thread-5448::DEBUG::2012-03-29 
11:57:47,105::lvm::376::OperationMutex::(_reloadvgs) Operation 'lvm 
reload operation' released the operation mutex

Thread-5448::DEBUG::2012-03-29 
11:57:47,107::persistentDict::175::Storage.PersistentDict::(__init__) 
Created a persistant dict with LvMetadataRW backend

Thread-5448::DEBUG::2012-03-29 
11:57:47,110::blockSD::177::Storage.Misc.excCmd::(readlines) '/bin/dd 
iflag=direct skip=0 bs=2048 
if=/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata count=1' (cwd None)

Thread-5448::DEBUG::2012-03-29 
11:57:47,155::blockSD::177::Storage.Misc.excCmd::(readlines) FAILED: 
<err> = "/bin/dd: reading 
`/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata': Input/output 
error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 
0.000525019 s, 0.0 kB/s\n"; <rc> = 1

Thread-5448::ERROR::2012-03-29 
11:57:47,158::sdc::113::Storage.StorageDomainCache::(_findDomain) Error 
while looking for domain `8ed25a57-f53a-4cf0-bb92-781f3ce36a48`

Traceback (most recent call last):

File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain

return mod.findDomain(sdUUID)

File "/usr/share/vdsm/storage/blockSD.py", line 1051, in findDomain

return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))

File "/usr/share/vdsm/storage/blockSD.py", line 241, in __init__

metadata = selectMetadata(sdUUID)

File "/usr/share/vdsm/storage/blockSD.py", line 210, in selectMetadata

if len(mdProvider) > 0:

File "/usr/share/vdsm/storage/persistentDict.py", line 51, in __len__

return len(self.keys())

File "/usr/share/vdsm/storage/persistentDict.py", line 95, in keys

return list(self.__iter__())

File "/usr/share/vdsm/storage/persistentDict.py", line 92, in __iter__

return ifilter(lambda k: k in self._validatorDict, self._dict.__iter__())

File "/usr/share/vdsm/storage/persistentDict.py", line 209, in __iter__

with self._accessWrapper():

File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__

return self.gen.next()

File "/usr/share/vdsm/storage/persistentDict.py", line 137, in 
_accessWrapper

self.refresh()

File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh

lines = self._metaRW.readlines()

File "/usr/share/vdsm/storage/blockSD.py", line 177, in readlines

m = misc.readblockSUDO(self.metavol, self._offset, self._size)

File "/usr/share/vdsm/storage/misc.py", line 307, in readblockSUDO

raise se.MiscBlockReadException(name, offset, size)

MiscBlockReadException: Internal block device read failure: 
'name=/dev/8ed25a57-f53a-4cf0-bb92-781f3ce36a48/metadata, offset=0, 
size=2048'

Thread-5448::DEBUG::2012-03-29 
11:57:47,177::resourceManager::538::ResourceManager::(releaseResource) 
Trying to release resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a'

Thread-5448::DEBUG::2012-03-29 
11:57:47,180::resourceManager::553::ResourceManager::(releaseResource) 
Released resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a' (0 
active users)

Thread-5448::DEBUG::2012-03-29 
11:57:47,181::resourceManager::558::ResourceManager::(releaseResource) 
Resource 'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a' is free, finding 
out if anyone is waiting for it.

Thread-5448::DEBUG::2012-03-29 
11:57:47,183::resourceManager::565::ResourceManager::(releaseResource) 
No one is waiting for resource 
'Storage.13080edc-77ea-11e1-b6a4-525400c49d2a', Clearing records.

Thread-5448::ERROR::2012-03-29 
11:57:47,185::task::853::TaskManager.Task::(_setError) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Unexpected error

Traceback (most recent call last):

File "/usr/share/vdsm/storage/task.py", line 861, in _run

return fn(*args, **kargs)

File "/usr/share/vdsm/logUtils.py", line 38, in wrapper

res = f(*args, **kwargs)

File "/usr/share/vdsm/storage/hsm.py", line 813, in connectStoragePool

return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, 
masterVersion, options)

File "/usr/share/vdsm/storage/hsm.py", line 855, in _connectStoragePool

res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)

File "/usr/share/vdsm/storage/sp.py", line 641, in connect

self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)

File "/usr/share/vdsm/storage/sp.py", line 1107, in __rebuild

self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, 
masterVersion=masterVersion)

File "/usr/share/vdsm/storage/sp.py", line 1442, in getMasterDomain

raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)

StoragePoolMasterNotFound: Cannot find master domain: 
'spUUID=13080edc-77ea-11e1-b6a4-525400c49d2a, 
msdUUID=8ed25a57-f53a-4cf0-bb92-781f3ce36a48'

Thread-5448::DEBUG::2012-03-29 
11:57:47,188::task::872::TaskManager.Task::(_run) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Task._run: 
838b7ca3-9f79-4c87-a2f0-12cad48cc127 
('13080edc-77ea-11e1-b6a4-525400c49d2a', 1, 
'13080edc-77ea-11e1-b6a4-525400c49d2a', 
'8ed25a57-f53a-4cf0-bb92-781f3ce36a48', 1) {} failed - stopping task

Thread-5448::DEBUG::2012-03-29 
11:57:47,190::task::1199::TaskManager.Task::(stop) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::stopping in state preparing 
(force False)

Thread-5448::DEBUG::2012-03-29 
11:57:47,192::task::978::TaskManager.Task::(_decref) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::ref 1 aborting True

Thread-5448::INFO::2012-03-29 
11:57:47,193::task::1157::TaskManager.Task::(prepare) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::aborting: Task is aborted: 
'Cannot find master domain' - code 304

Thread-5448::DEBUG::2012-03-29 
11:57:47,195::task::1162::TaskManager.Task::(prepare) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Prepare: aborted: Cannot 
find master domain

Thread-5448::DEBUG::2012-03-29 
11:57:47,197::task::978::TaskManager.Task::(_decref) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::ref 0 aborting True

Thread-5448::DEBUG::2012-03-29 
11:57:47,198::task::913::TaskManager.Task::(_doAbort) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::Task._doAbort: force False

Thread-5448::DEBUG::2012-03-29 
11:57:47,200::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}

Thread-5448::DEBUG::2012-03-29 
11:57:47,201::task::588::TaskManager.Task::(_updateState) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::moving from state preparing 
-> state aborting

Thread-5448::DEBUG::2012-03-29 
11:57:47,203::task::537::TaskManager.Task::(__state_aborting) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::_aborting: recover policy none

Thread-5448::DEBUG::2012-03-29 
11:57:47,205::task::588::TaskManager.Task::(_updateState) 
Task=`838b7ca3-9f79-4c87-a2f0-12cad48cc127`::moving from state aborting 
-> state failed

Thread-5448::DEBUG::2012-03-29 
11:57:47,206::resourceManager::809::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}

Thread-5448::DEBUG::2012-03-29 
11:57:47,208::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}

Thread-5448::ERROR::2012-03-29 
11:57:47,209::dispatcher::89::Storage.Dispatcher.Protect::(run) 
{'status': {'message': "Cannot find master domain: 
'spUUID=13080edc-77ea-11e1-b6a4-525400c49d2a, 
msdUUID=8ed25a57-f53a-4cf0-bb92-781f3ce36a48'", 'code': 304}}

Thread-5453::DEBUG::2012-03-29 
11:57:48,736::task::588::TaskManager.Task::(_updateState) 
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::moving from state init -> 
state preparing

Thread-5453::INFO::2012-03-29 
11:57:48,739::logUtils::37::dispatcher::(wrapper) Run and protect: 
repoStats(options=None)

Thread-5453::INFO::2012-03-29 
11:57:48,740::logUtils::39::dispatcher::(wrapper) Run and protect: 
repoStats, Return response: {}

Thread-5453::DEBUG::2012-03-29 
11:57:48,742::task::1172::TaskManager.Task::(prepare) 
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::finished: {}

Thread-5453::DEBUG::2012-03-29 
11:57:48,743::task::588::TaskManager.Task::(_updateState) 
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::moving from state preparing 
-> state finished

Thread-5453::DEBUG::2012-03-29 
11:57:48,745::resourceManager::809::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}

Thread-5453::DEBUG::2012-03-29 
11:57:48,746::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}

Thread-5453::DEBUG::2012-03-29 
11:57:48,748::task::978::TaskManager.Task::(_decref) 
Task=`b801beea-93db-44e7-ba71-5d1eca7eecae`::ref 0 aborting False

Regards, rene

*Von:*users-bounces at ovirt.org [mailto:users-bounces at ovirt.org] *Im 
Auftrag von *Rene Rosenberger
*Gesendet:* Donnerstag, 29. März 2012 11:47
*An:* rvaknin at redhat.com; users at ovirt.org
*Betreff:* Re: [Users] storage domain reactivate not working

Hi,

i did:

[root at KVM-DMZ-04 vdsm]# iscsiadm -m node -T 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d -u

Logging out of session [sid: 2, target: 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal: 
192.168.xxx.xxx,3260]

Logout of [sid: 2, target: 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal: 
192.168.xxx.xxx,3260] successful.

[root at KVM-DMZ-04 vdsm]# iscsiadm -m node -T 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d -l

Logging in to [iface: default, target: 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal: 
192.168.xxx.xxx,3260] (multiple)

Login to [iface: default, target: 
iqn.2004-04.com.qnap:ts-419uplus:iscsi.dmznas01.c2b74d, portal: 
192.168.xxx.xxx,3260] successful.

But the data center is locked in webinterface. System -> default -> 
storage -> DMZ-NAS-01 -> Data Center = locked

Regards, rene

*Von:*Rami Vaknin [mailto:rvaknin at redhat.com]
*Gesendet:* Donnerstag, 29. März 2012 11:32
*An:* Rene Rosenberger; users at ovirt.org
*Betreff:* Re: [Users] storage domain reactivate not working

On 03/29/2012 11:28 AM, Rene Rosenberger wrote:

Hi,

ihave rebootet my iscsi device without mainenance mode. Now it is 
inactive. When I want to reactivate it again it fails. What can I do?

Depends why it fails, vdsm log can help.

You can check whether or not you're connected from the hosts to the 
iscsi target or reconnect using:
iscsiadm -m discoverydb --discover -t st -p your_iscsi_server_fqdn
iscsiadm -m node -T your_target_name -l

Than try to activate it.

Regards, rene




_______________________________________________

Users mailing list

Users at ovirt.org  <mailto:Users at ovirt.org>

http://lists.ovirt.org/mailman/listinfo/users




-- 

  

Thanks,

  

Rami Vaknin, QE @ Red Hat, TLV, IL.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20120329/a6d71f9b/attachment-0001.html>


More information about the Users mailing list