[Users] HELP, ISCSI Freakin!

--_000_9272B85912E97F4486906561F9257C5701675FF9AUSP01DAG0307co_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hey Guys, I am running oVirt 3.1; and I am getting the following in dmesg, and I have= pasted my VG info below. We are running iSCSI luns and this is a node com= plaining in the log. Our equalogic storage was close to full; and this sta= rted happening. I have extended it, and it is thin provisioned; but I am n= ot sure if a pvresize is needed. If so, can it be done while online? Tryi= ng not to take down this cluster; several heavily used VMs on it. The VMs = seem to be running ok for now; but this is filling up the logs pretty quick= ly; as it appears to be retrying to connections and to release the resource= s. All help is appreciated. Thanks for all your help guys; and feel free to email me directly if I can = provide any more info. Log Snippet: Thread-550892::DEBUG::2013-07-17 19:06:14,335::resourceManager::565::Resour= ceManager::(releaseResource) No one is waiting for resource 'Storage.69d66c= f5-9426-4385-be6c-c55e2bd40617', Clearing records. Thread-550892::DEBUG::2013-07-17 19:06:14,335::task::978::TaskManager.Task:= :(_decref) Task=3D`03810e67-5dab-4193-8cbd-c633c8c25fa4`::ref 0 aborting Fa= lse Thread-550892::DEBUG::2013-07-17 19:06:14,336::task::588::TaskManager.Task:= :(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving from = state init -> state preparing Thread-550892::INFO::2013-07-17 19:06:14,336::logUtils::37::dispatcher::(wr= apper) Run and protect: getVolumeSize(sdUUID=3D'7cf8a585-34a8-470c-9b40-9d4= 86456fcea', spUUID=3D'2ecfa6dd-a1fa-428c-9d04-db17c6c4491d', imgUUID=3D'8cb= c183a-0afe-497a-a54b-ec5f03b7fe16', volUUID=3D'bcb8d500-5aa4-404a-9be2-9fb8= 4bc87300', options=3DNone) Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::175::Resour= ceManager.Request::(__init__) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-9d= 486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Request was made= in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerRes= ource' Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::486::Resour= ceManager::(registerResource) Trying to register resource 'Storage.7cf8a585= -34a8-470c-9b40-9d486456fcea' for lock type 'shared' Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::528::Resour= ceManager::(registerResource) Resource 'Storage.7cf8a585-34a8-470c-9b40-9d4= 86456fcea' is free. Now locking as 'shared' (1 active user) Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::212::Resour= ceManager.Request::(grant) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-9d486= 456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Granted request Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::817::TaskManager.Task:= :(resourceAcquired) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::_resourc= esAcquired: Storage.7cf8a585-34a8-470c-9b40-9d486456fcea (shared) Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::978::TaskManager.Task:= :(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 1 aborting Fa= lse Thread-550892::INFO::2013-07-17 19:06:14,338::logUtils::39::dispatcher::(wr= apper) Run and protect: getVolumeSize, Return response: {'truesize': '31138= 512896', 'apparentsize': '31138512896'} Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::1172::TaskManager.Task= ::(prepare) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::finished: {'true= size': '31138512896', 'apparentsize': '31138512896'} Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::588::TaskManager.Task:= :(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving from = state preparing -> state finished Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::809::Resour= ceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Stor= age.7cf8a585-34a8-470c-9b40-9d486456fcea': < ResourceRef 'Storage.7cf8a585-= 34a8-470c-9b40-9d486456fcea', isValid: 'True' obj: 'None'>} Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::844::Resour= ceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::538::Resour= ceManager::(releaseResource) Trying to release resource 'Storage.7cf8a585-3= 4a8-470c-9b40-9d486456fcea' Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::553::Resour= ceManager::(releaseResource) Released resource 'Storage.7cf8a585-34a8-470c-= 9b40-9d486456fcea' (0 active users) Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::558::Resour= ceManager::(releaseResource) Resource 'Storage.7cf8a585-34a8-470c-9b40-9d48= 6456fcea' is free, finding out if anyone is waiting for it. Thread-550892::DEBUG::2013-07-17 19:06:14,339::resourceManager::565::Resour= ceManager::(releaseResource) No one is waiting for resource 'Storage.7cf8a5= 85-34a8-470c-9b40-9d486456fcea', Clearing records. Thread-550892::DEBUG::2013-07-17 19:06:14,339::task::978::TaskManager.Task:= :(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 0 aborting Fa= lse Dmesg Snippet: __ratelimit: 28 callbacks suppressed end_request: I/O error, dev dm-64, sector 5769088 end_request: I/O error, dev dm-64, sector 5769200 end_request: I/O error, dev dm-64, sector 5507072 end_request: I/O error, dev dm-64, sector 5507080 end_request: I/O error, dev dm-64, sector 5507072 end_request: I/O error, dev dm-64, sector 5506944 end_request: I/O error, dev dm-64, sector 5507056 end_request: I/O error, dev dm-64, sector 1312768 end_request: I/O error, dev dm-64, sector 1312776 end_request: I/O error, dev dm-64, sector 1312768 end_request: I/O error, dev dm-64, sector 6031232 end_request: I/O error, dev dm-64, sector 6031344 end_request: I/O error, dev dm-64, sector 5769216 end_request: I/O error, dev dm-64, sector 5769224 end_request: I/O error, dev dm-64, sector 5769216 end_request: I/O error, dev dm-64, sector 6293376 end_request: I/O error, dev dm-64, sector 6293488 end_request: I/O error, dev dm-64, sector 6031360 end_request: I/O error, dev dm-64, sector 6031368 end_request: I/O error, dev dm-64, sector 6031360 end_request: I/O error, dev dm-64, sector 8390528 end_request: I/O error, dev dm-64, sector 8390640 end_request: I/O error, dev dm-64, sector 6293504 end_request: I/O error, dev dm-64, sector 6293512 end_request: I/O error, dev dm-64, sector 6293504 __ratelimit: 25 callbacks suppressed Thanks, Matt Curry Matt(a)MattCurry.com ________________________________ This is a PRIVATE message. If you are not the intended recipient, please de= lete without copying and kindly advise us by e-mail of the mistake in deliv= ery. NOTE: Regardless of content, this e-mail shall not operate to bind SKO= POS to any order or other contract unless pursuant to explicit written agre= ement or government initiative expressly permitting the use of e-mail for s= uch purpose. --_000_9272B85912E97F4486906561F9257C5701675FF9AUSP01DAG0307co_ Content-Type: text/html; charset="us-ascii" Content-ID: <042E26E9FED4D948ACA4DC2C3F04AFB6(a)collaborationhost.netContent-Transfer-Encoding: quoted-printable <html<head<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= </head<body style=3D"word-wrap:break-word; color:rgb(0,0,0); font-size:14px; font= -family:'Courier New',sans-serif"<div>Hey Guys,</div<div><br</div<div>I am running oVirt 3.1; and I am getting the following in dmesg, and I= have pasted my VG info below. &nbsp;We are running iSCSI luns and this is = a node complaining in the log. &nbsp;Our equalogic storage was close to ful= l; and this started happening. &nbsp;I have extended it, and it is thin provisioned; but I am not sure if a pvresize is needed.= &nbsp;If so, can it be done while online? &nbsp;Trying not to take down th= is cluster; several heavily used VMs on it. &nbsp;The VMs seem to be runnin= g ok for now; but this is filling up the logs pretty quickly; as it appears to be retrying to connections and to release the re= sources. <b>All help is appreciated</b>.</div<div><br</div<div>Thanks for all your help guys; and feel free to email me directly if I= can provide any more info.</div<div><br</div<div><b>Log Snippet:</b></div<div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,335::resourceManager::565::R= esourceManager::(releaseResource) No one is waiting for resource 'Storage.6= 9d66cf5-9426-4385-be6c-c55e2bd40617', Clearing records.</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,335::task::978::TaskManager.= Task::(_decref) Task=3D`03810e67-5dab-4193-8cbd-c633c8c25fa4`::ref 0 aborti= ng False</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::task::588::TaskManager.= Task::(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving = from state init -&gt; state preparing</div<div>Thread-550892::INFO::2013-07-17 19:06:14,336::logUtils::37::dispatcher= ::(wrapper) Run and protect: getVolumeSize(sdUUID=3D'7cf8a585-34a8-470c-9b4= 0-9d486456fcea', spUUID=3D'2ecfa6dd-a1fa-428c-9d04-db17c6c4491d', imgUUID= =3D'8cbc183a-0afe-497a-a54b-ec5f03b7fe16', volUUID=3D'bcb8d500-5aa4-404a-9be2-9fb84bc87300', options=3DNone)</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::175::R= esourceManager.Request::(__init__) ResName=3D`Storage.7cf8a585-34a8-470c-9b= 40-9d486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Request was= made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource'</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::486::R= esourceManager::(registerResource) Trying to register resource 'Storage.7cf= 8a585-34a8-470c-9b40-9d486456fcea' for lock type 'shared'</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::528::R= esourceManager::(registerResource) Resource 'Storage.7cf8a585-34a8-470c-9b4= 0-9d486456fcea' is free. Now locking as 'shared' (1 active user)</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::212::R= esourceManager.Request::(grant) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-= 9d486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Granted reques= t</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::817::TaskManager.= Task::(resourceAcquired) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::_re= sourcesAcquired: Storage.7cf8a585-34a8-470c-9b40-9d486456fcea (shared)</div= <div>Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::978::TaskManager.= Task::(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 1 aborti= ng False</div<div>Thread-550892::INFO::2013-07-17 19:06:14,338::logUtils::39::dispatcher= ::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '= 31138512896', 'apparentsize': '31138512896'}</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::1172::TaskManager= .Task::(prepare) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::finished: {= 'truesize': '31138512896', 'apparentsize': '31138512896'}</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::588::TaskManager.= Task::(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving = from state preparing -&gt; state finished</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::809::R= esourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {= 'Storage.7cf8a585-34a8-470c-9b40-9d486456fcea': &lt; ResourceRef 'Storage.7= cf8a585-34a8-470c-9b40-9d486456fcea', isValid: 'True' obj: 'None'&gt;}</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::844::R= esourceManager.Owner::(cancelAll) Owner.cancelAll requests {}</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::538::R= esourceManager::(releaseResource) Trying to release resource 'Storage.7cf8a= 585-34a8-470c-9b40-9d486456fcea'</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::553::R= esourceManager::(releaseResource) Released resource 'Storage.7cf8a585-34a8-= 470c-9b40-9d486456fcea' (0 active users)</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::558::R= esourceManager::(releaseResource) Resource 'Storage.7cf8a585-34a8-470c-9b40= -9d486456fcea' is free, finding out if anyone is waiting for it.</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,339::resourceManager::565::R= esourceManager::(releaseResource) No one is waiting for resource 'Storage.7= cf8a585-34a8-470c-9b40-9d486456fcea', Clearing records.</div<div>Thread-550892::DEBUG::2013-07-17 19:06:14,339::task::978::TaskManager.= Task::(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 0 aborti= ng False</div</div<div><br</div<div><br</div<div><b><br</b></div<div><b>Dmesg Snippet:</b></div<div<div>__ratelimit: 28 callbacks suppressed</div<div>end_request: I/O error, dev dm-64, sector 5769088</div<div>end_request: I/O error, dev dm-64, sector 5769200</div<div>end_request: I/O error, dev dm-64, sector 5507072</div<div>end_request: I/O error, dev dm-64, sector 5507080</div<div>end_request: I/O error, dev dm-64, sector 5507072</div<div>end_request: I/O error, dev dm-64, sector 5506944</div<div>end_request: I/O error, dev dm-64, sector 5507056</div<div>end_request: I/O error, dev dm-64, sector 1312768</div<div>end_request: I/O error, dev dm-64, sector 1312776</div<div>end_request: I/O error, dev dm-64, sector 1312768</div<div>end_request: I/O error, dev dm-64, sector 6031232</div<div>end_request: I/O error, dev dm-64, sector 6031344</div<div>end_request: I/O error, dev dm-64, sector 5769216</div<div>end_request: I/O error, dev dm-64, sector 5769224</div<div>end_request: I/O error, dev dm-64, sector 5769216</div<div>end_request: I/O error, dev dm-64, sector 6293376</div<div>end_request: I/O error, dev dm-64, sector 6293488</div<div>end_request: I/O error, dev dm-64, sector 6031360</div<div>end_request: I/O error, dev dm-64, sector 6031368</div<div>end_request: I/O error, dev dm-64, sector 6031360</div<div>end_request: I/O error, dev dm-64, sector 8390528</div<div>end_request: I/O error, dev dm-64, sector 8390640</div<div>end_request: I/O error, dev dm-64, sector 6293504</div<div>end_request: I/O error, dev dm-64, sector 6293512</div<div>end_request: I/O error, dev dm-64, sector 6293504</div<div>__ratelimit: 25 callbacks suppressed</div</div<div><br</div<div><br</div<div>Thanks,</div<div>Matt Curry</div<div&gt;Matt(a)MattCurry.com&lt;/div<div><br</div<br<br<hr<font color=3D"Gray" face=3D"Arial" size=3D"1">This is a PRIVATE message. I= f you are not the intended recipient, please delete without copying and kin= dly advise us by e-mail of the mistake in delivery. NOTE: Regardless of con= tent, this e-mail shall not operate to bind SKOPOS to any order or other contract unless pursuant to explicit wri= tten agreement or government initiative expressly permitting the use of e-m= ail for such purpose.</font</body</html --_000_9272B85912E97F4486906561F9257C5701675FF9AUSP01DAG0307co_--