8:09 p.m.
--_000_9272B85912E97F4486906561F9257C570167600AAUSP01DAG0307co_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Forgot one thing, the paste bin link!
http://pastebin.com/3t7E41JK
Date: Wednesday, July 17, 2013 7:08 PM
To: users <users@ovirt.org<mailto:users@ovirt.org> Subject:
HELP, ISCSI Freakin!
Hey Guys,
I am running oVirt 3.1; and I am getting the following in dmesg, and I have=
pasted my VG info below. We are running iSCSI luns and this is a node com=
plaining in the log. Our equalogic storage was close to full; and this sta=
rted happening. I have extended it, and it is thin provisioned; but I am n=
ot sure if a pvresize is needed. If so, can it be done while online? Tryi=
ng not to take down this cluster; several heavily used VMs on it. The VMs =
seem to be running ok for now; but this is filling up the logs pretty quick=
ly; as it appears to be retrying to connections and to release the resource=
s. All help is appreciated.
Thanks for all your help guys; and feel free to email me directly if I can =
provide any more info.
Log Snippet:
Thread-550892::DEBUG::2013-07-17 19:06:14,335::resourceManager::565::Resour=
ceManager::(releaseResource) No one is waiting for resource 'Storage.69d66c=
f5-9426-4385-be6c-c55e2bd40617', Clearing records.
Thread-550892::DEBUG::2013-07-17 19:06:14,335::task::978::TaskManager.Task:=
:(_decref) Task=3D`03810e67-5dab-4193-8cbd-c633c8c25fa4`::ref 0 aborting Fa=
lse
Thread-550892::DEBUG::2013-07-17 19:06:14,336::task::588::TaskManager.Task:=
:(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving from =
state init -> state preparing
Thread-550892::INFO::2013-07-17 19:06:14,336::logUtils::37::dispatcher::(wr=
apper) Run and protect: getVolumeSize(sdUUID=3D'7cf8a585-34a8-470c-9b40-9d4=
86456fcea', spUUID=3D'2ecfa6dd-a1fa-428c-9d04-db17c6c4491d',
imgUUID=3D'8cb=
c183a-0afe-497a-a54b-ec5f03b7fe16', volUUID=3D'bcb8d500-5aa4-404a-9be2-9fb8=
4bc87300', options=3DNone)
Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::175::Resour=
ceManager.Request::(__init__) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-9d=
486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Request was made=
in '/usr/share/vdsm/storage/resourceManager.py' line '485' at
'registerRes=
ource'
Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::486::Resour=
ceManager::(registerResource) Trying to register resource 'Storage.7cf8a585=
-34a8-470c-9b40-9d486456fcea' for lock type 'shared'
Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::528::Resour=
ceManager::(registerResource) Resource 'Storage.7cf8a585-34a8-470c-9b40-9d4=
86456fcea' is free. Now locking as 'shared' (1 active user)
Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::212::Resour=
ceManager.Request::(grant) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-9d486=
456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Granted request
Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::817::TaskManager.Task:=
:(resourceAcquired) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::_resourc=
esAcquired: Storage.7cf8a585-34a8-470c-9b40-9d486456fcea (shared)
Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::978::TaskManager.Task:=
:(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 1 aborting Fa=
lse
Thread-550892::INFO::2013-07-17 19:06:14,338::logUtils::39::dispatcher::(wr=
apper) Run and protect: getVolumeSize, Return response: {'truesize': '31138=
512896', 'apparentsize': '31138512896'}
Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::1172::TaskManager.Task=
::(prepare) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::finished: {'true=
size': '31138512896', 'apparentsize': '31138512896'}
Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::588::TaskManager.Task:=
:(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving from =
state preparing -> state finished
Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::809::Resour=
ceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Stor=
age.7cf8a585-34a8-470c-9b40-9d486456fcea': < ResourceRef 'Storage.7cf8a585-=
34a8-470c-9b40-9d486456fcea', isValid: 'True' obj: 'None'>}
Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::844::Resour=
ceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::538::Resour=
ceManager::(releaseResource) Trying to release resource 'Storage.7cf8a585-3=
4a8-470c-9b40-9d486456fcea'
Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::553::Resour=
ceManager::(releaseResource) Released resource 'Storage.7cf8a585-34a8-470c-=
9b40-9d486456fcea' (0 active users)
Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::558::Resour=
ceManager::(releaseResource) Resource 'Storage.7cf8a585-34a8-470c-9b40-9d48=
6456fcea' is free, finding out if anyone is waiting for it.
Thread-550892::DEBUG::2013-07-17 19:06:14,339::resourceManager::565::Resour=
ceManager::(releaseResource) No one is waiting for resource 'Storage.7cf8a5=
85-34a8-470c-9b40-9d486456fcea', Clearing records.
Thread-550892::DEBUG::2013-07-17 19:06:14,339::task::978::TaskManager.Task:=
:(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 0 aborting Fa=
lse
Dmesg Snippet:
__ratelimit: 28 callbacks suppressed
end_request: I/O error, dev dm-64, sector 5769088
end_request: I/O error, dev dm-64, sector 5769200
end_request: I/O error, dev dm-64, sector 5507072
end_request: I/O error, dev dm-64, sector 5507080
end_request: I/O error, dev dm-64, sector 5507072
end_request: I/O error, dev dm-64, sector 5506944
end_request: I/O error, dev dm-64, sector 5507056
end_request: I/O error, dev dm-64, sector 1312768
end_request: I/O error, dev dm-64, sector 1312776
end_request: I/O error, dev dm-64, sector 1312768
end_request: I/O error, dev dm-64, sector 6031232
end_request: I/O error, dev dm-64, sector 6031344
end_request: I/O error, dev dm-64, sector 5769216
end_request: I/O error, dev dm-64, sector 5769224
end_request: I/O error, dev dm-64, sector 5769216
end_request: I/O error, dev dm-64, sector 6293376
end_request: I/O error, dev dm-64, sector 6293488
end_request: I/O error, dev dm-64, sector 6031360
end_request: I/O error, dev dm-64, sector 6031368
end_request: I/O error, dev dm-64, sector 6031360
end_request: I/O error, dev dm-64, sector 8390528
end_request: I/O error, dev dm-64, sector 8390640
end_request: I/O error, dev dm-64, sector 6293504
end_request: I/O error, dev dm-64, sector 6293512
end_request: I/O error, dev dm-64, sector 6293504
__ratelimit: 25 callbacks suppressed
Thanks,
Matt Curry
Matt@MattCurry.com<mailto:Matt@MattCurry.com
________________________________
This is a PRIVATE message. If you are not the intended recipient, please de=
lete without copying and kindly advise us by e-mail of the mistake in deliv=
ery. NOTE: Regardless of content, this e-mail shall not operate to bind SKO=
POS to any order or other contract unless pursuant to explicit written agre=
ement or government initiative expressly permitting the use of e-mail for s=
uch purpose.
--_000_9272B85912E97F4486906561F9257C570167600AAUSP01DAG0307co_
Content-Type: text/html; charset="us-ascii"
Content-ID: <2D58FAEDAE23E44BB4A9811BDA48B79A(a)collaborationhost.net Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
</head <body
style=3D"word-wrap:break-word; color:rgb(0,0,0); font-size:14px; font=
-family:'Courier New',sans-serif"
<div <div
<div>Forgot one thing, the paste bin link!</div </div
</div <div><br </div
<div><a
href=3D"http://pastebin.com/3t7E41JK">http://pastebin.com/3t...
/a></div <div><span
style=3D"font-family:Calibri; font-size:11pt; text-align:left; f=
ont-weight:bold"><br </span></div <div><br
</div <span
id=3D"OLK_SRC_BODY_SECTION" <div
style=3D"font-family:Calibri; font-size:11pt; text-align:left; color:b=
lack; border-bottom:medium none; border-left:medium none; padding-bottom:0i=
n; padding-left:0in; padding-right:0in; border-top:#b5c4df 1pt solid; borde=
r-right:medium none; padding-top:3pt" <span
style=3D"font-weight:bold">Date: </span>Wednesday, July 17, 2013 7:08=
PM<br <span
style=3D"font-weight:bold">To: </span>users <<a
href=3D"mailto:use=
rs@ovirt.org">users(a)ovirt.org</a>&gt;<br <span
style=3D"font-weight:bold">Subject: </span>HELP, ISCSI
Freakin!<br </div <div><br
</div <div <div
style=3D"word-wrap:break-word; color:rgb(0,0,0); font-size:14px; font-=
family:'Courier New',sans-serif"
<div>Hey Guys,</div <div><br </div
<div>I am running oVirt 3.1; and I am getting the following in dmesg, and I=
have pasted my VG info below. We are running iSCSI luns and this is =
a node complaining in the log. Our equalogic storage was close to ful=
l; and this started happening. I have extended
it, and it is thin provisioned; but I am not sure if a pvresize is needed.=
If so, can it be done while online? Trying not to take down th=
is cluster; several heavily used VMs on it. The VMs seem to be runnin=
g ok for now; but this is filling up the logs pretty
quickly; as it appears to be retrying to connections and to release the re=
sources.
<b>All help is appreciated</b>.</div
<div><br </div <div>Thanks for all your help guys; and feel free to
email me directly if I=
can provide any more info.</div
<div><br </div <div><b>Log Snippet:</b></div <div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,335::resourceManager::565::R=
esourceManager::(releaseResource) No one is waiting for resource 'Storage.6=
9d66cf5-9426-4385-be6c-c55e2bd40617', Clearing records.</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,335::task::978::TaskManager.=
Task::(_decref) Task=3D`03810e67-5dab-4193-8cbd-c633c8c25fa4`::ref 0 aborti=
ng False</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::task::588::TaskManager.=
Task::(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving =
from state init -> state preparing</div
<div>Thread-550892::INFO::2013-07-17 19:06:14,336::logUtils::37::dispatcher=
::(wrapper) Run and protect: getVolumeSize(sdUUID=3D'7cf8a585-34a8-470c-9b4=
0-9d486456fcea', spUUID=3D'2ecfa6dd-a1fa-428c-9d04-db17c6c4491d', imgUUID=
=3D'8cbc183a-0afe-497a-a54b-ec5f03b7fe16',
volUUID=3D'bcb8d500-5aa4-404a-9be2-9fb84bc87300', options=3DNone)</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,336::resourceManager::175::R=
esourceManager.Request::(__init__) ResName=3D`Storage.7cf8a585-34a8-470c-9b=
40-9d486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Request was=
made in '/usr/share/vdsm/storage/resourceManager.py'
line '485' at 'registerResource'</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,336::resourceManager::486::R=
esourceManager::(registerResource) Trying to register resource 'Storage.7cf=
8a585-34a8-470c-9b40-9d486456fcea' for lock type 'shared'</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,336::resourceManager::528::R=
esourceManager::(registerResource) Resource 'Storage.7cf8a585-34a8-470c-9b4=
0-9d486456fcea' is free. Now locking as 'shared' (1 active user)</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,336::resourceManager::212::R=
esourceManager.Request::(grant) ResName=3D`Storage.7cf8a585-34a8-470c-9b40-=
9d486456fcea`ReqID=3D`7411edba-20b8-4855-8271-b6aa52a9203a`::Granted reques=
t</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,337::task::817::TaskManager.=
Task::(resourceAcquired) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::_re=
sourcesAcquired: Storage.7cf8a585-34a8-470c-9b40-9d486456fcea (shared)</div=
<div>Thread-550892::DEBUG::2013-07-17
19:06:14,337::task::978::TaskManager.=
Task::(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 1 aborti=
ng False</div
<div>Thread-550892::INFO::2013-07-17 19:06:14,338::logUtils::39::dispatcher=
::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '=
31138512896', 'apparentsize': '31138512896'}</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,338::task::1172::TaskManager=
.Task::(prepare) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::finished: {=
'truesize': '31138512896', 'apparentsize':
'31138512896'}</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::task::588::TaskManager.=
Task::(_updateState) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::moving =
from state preparing -> state finished</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::809::R=
esourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {=
'Storage.7cf8a585-34a8-470c-9b40-9d486456fcea': < ResourceRef
'Storage.7=
cf8a585-34a8-470c-9b40-9d486456fcea',
isValid: 'True' obj: 'None'>}</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::844::R=
esourceManager.Owner::(cancelAll) Owner.cancelAll requests {}</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,338::resourceManager::538::R=
esourceManager::(releaseResource) Trying to release resource 'Storage.7cf8a=
585-34a8-470c-9b40-9d486456fcea'</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::553::R=
esourceManager::(releaseResource) Released resource 'Storage.7cf8a585-34a8-=
470c-9b40-9d486456fcea' (0 active users)</div
<div>Thread-550892::DEBUG::2013-07-17 19:06:14,338::resourceManager::558::R=
esourceManager::(releaseResource) Resource 'Storage.7cf8a585-34a8-470c-9b40=
-9d486456fcea' is free, finding out if anyone is waiting for it.</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,339::resourceManager::565::R=
esourceManager::(releaseResource) No one is waiting for resource 'Storage.7=
cf8a585-34a8-470c-9b40-9d486456fcea', Clearing records.</div <div>Thread-550892::DEBUG::2013-07-17
19:06:14,339::task::978::TaskManager.=
Task::(_decref) Task=3D`7379fc72-5137-4432-861c-85b1f33ba843`::ref 0 aborti=
ng False</div </div <div><br
</div <div><br </div
<div><b><br </b></div <div><b>Dmesg Snippet:</b></div <div <div>__ratelimit: 28 callbacks
suppressed</div <div>end_request: I/O error,
dev dm-64, sector 5769088</div <div>end_request: I/O error,
dev dm-64, sector 5769200</div <div>end_request: I/O error,
dev dm-64, sector 5507072</div <div>end_request: I/O error,
dev dm-64, sector 5507080</div <div>end_request: I/O error,
dev dm-64, sector 5507072</div <div>end_request: I/O error,
dev dm-64, sector 5506944</div <div>end_request: I/O error,
dev dm-64, sector 5507056</div <div>end_request: I/O error,
dev dm-64, sector 1312768</div <div>end_request: I/O error,
dev dm-64, sector 1312776</div <div>end_request: I/O error,
dev dm-64, sector 1312768</div <div>end_request: I/O error,
dev dm-64, sector 6031232</div <div>end_request: I/O error,
dev dm-64, sector 6031344</div <div>end_request: I/O error,
dev dm-64, sector 5769216</div <div>end_request: I/O error,
dev dm-64, sector 5769224</div <div>end_request: I/O error,
dev dm-64, sector 5769216</div <div>end_request: I/O error,
dev dm-64, sector 6293376</div <div>end_request: I/O error,
dev dm-64, sector 6293488</div <div>end_request: I/O error,
dev dm-64, sector 6031360</div <div>end_request: I/O error,
dev dm-64, sector 6031368</div <div>end_request: I/O error,
dev dm-64, sector 6031360</div <div>end_request: I/O error,
dev dm-64, sector 8390528</div <div>end_request: I/O error,
dev dm-64, sector 8390640</div <div>end_request: I/O error,
dev dm-64, sector 6293504</div <div>end_request: I/O error,
dev dm-64, sector 6293512</div <div>end_request: I/O error,
dev dm-64, sector 6293504</div <div>__ratelimit: 25 callbacks
suppressed</div </div <div><br
</div <div><br </div
<div>Thanks,</div <div>Matt Curry</div <div><a
href=3D"mailto:Matt@MattCurry.com">Matt@MattCurry.com</a></div <div><br
</div <br
</div </div </span><br
<hr <font color=3D"Gray"
face=3D"Arial" size=3D"1">This is a PRIVATE message. I=
f you are not the intended recipient, please delete without copying and kin=
dly advise us by e-mail of the mistake in delivery. NOTE: Regardless of con=
tent, this e-mail shall not operate to
bind SKOPOS to any order or other contract unless pursuant to explicit wri=
tten agreement or government initiative expressly permitting the use of e-m=
ail for such purpose.</font </body </html
--_000_9272B85912E97F4486906561F9257C570167600AAUSP01DAG0307co_--