<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Attached<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thx Christian<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="DE" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Von:</span></b><span lang="DE" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Nir Soffer [mailto:nsoffer@redhat.com]
<br>
<b>Gesendet:</b> Freitag, 10. Februar 2017 17:43<br>
<b>An:</b> Grundmann, Christian <Christian.Grundmann@fabasoft.com><br>
<b>Cc:</b> users@ovirt.org<br>
<b>Betreff:</b> Re: [ovirt-users] Storage domain experienced a high latency<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Feb 9, 2017 at 10:03 AM, Grundmann, Christian <<a href="mailto:Christian.Grundmann@fabasoft.com" target="_blank">Christian.Grundmann@fabasoft.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Hi,<br>
<br>
@ Can also be low level issue in kernel, hba, switch, server.<br>
I have the old storage on the same cable so I don’t think its hba or switch related<br>
On the same Switch I have a few ESXi Server with same storage setup which are working without problems.<br>
<br>
@multipath<br>
I use stock ng-node multipath configuration<br>
<br>
# VDSM REVISION 1.3<br>
<br>
defaults {<br>
polling_interval 5<br>
no_path_retry fail<br>
user_friendly_names no<br>
flush_on_last_del yes<br>
fast_io_fail_tmo 5<br>
dev_loss_tmo 30<br>
max_fds 4096<br>
}<br>
<br>
# Remove devices entries when overrides section is available.<br>
devices {<br>
device {<br>
# These settings overrides built-in devices settings. It does not apply<br>
# to devices without built-in settings (these use the settings in the<br>
# "defaults" section), or to devices defined in the "devices" section.<br>
# Note: This is not available yet on Fedora 21. For more info see<br>
# <a href="https://bugzilla.redhat.com/1253799" target="_blank">https://bugzilla.redhat.com/1253799</a><br>
all_devs yes<br>
no_path_retry fail<br>
}<br>
}<br>
<br>
# Enable when this section is available on all supported platforms.<br>
# Options defined here override device specific options embedded into<br>
# multipathd.<br>
#<br>
# overrides {<br>
# no_path_retry fail<br>
# }<br>
<br>
<br>
multipath -r v3<br>
has no output<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">My mistake, the correct command is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">multipath -r -v3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It creates tons of output, so better redirect to file and attach the file:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">multipath -r -v3 > multiapth-r-v3.out<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
<br>
Thx Christian<br>
<br>
<br>
Von: Nir Soffer [mailto:<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>]<br>
Gesendet: Mittwoch, 08. Februar 2017 20:44<br>
An: Grundmann, Christian <<a href="mailto:Christian.Grundmann@fabasoft.com">Christian.Grundmann@fabasoft.com</a>><br>
Cc: <a href="mailto:users@ovirt.org">users@ovirt.org</a><br>
Betreff: Re: [ovirt-users] Storage domain experienced a high latency<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
On Wed, Feb 8, 2017 at 6:11 PM, Grundmann, Christian <mailto:<a href="mailto:Christian.Grundmann@fabasoft.com">Christian.Grundmann@fabasoft.com</a>> wrote:<br>
Hi,<br>
got a new FC Storage (EMC Unity 300F) which is seen by my Hosts additional to my old Storage for Migration.<br>
New Storage has only on PATH until Migration is done.<br>
I already have a few VMs running on the new Storage without Problem.<br>
But after starting some VMs (don’t really no whats the difference to working ones), the Path for new Storage fails.<br>
<br>
Engine tells me: Storage Domain <storagedomain> experienced a high latency of 22.4875 seconds from host <host><br>
<br>
Where can I start looking?<br>
<br>
In /var/log/messages I found:<br>
<br>
Feb 8 09:03:53 ovirtnode01 multipathd: 360060160422143002a38935800ae2760: sdd - emc_clariion_checker: Active path is healthy.<br>
Feb 8 09:03:53 ovirtnode01 multipathd: 8:48: reinstated<br>
Feb 8 09:03:53 ovirtnode01 multipathd: 360060160422143002a38935800ae2760: remaining active paths: 1<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 8<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 5833475<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 5833475<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<br>
Feb 8 09:03:53 ovirtnode01 kernel: Buffer I/O error on dev dm-207, logical block 97, async page read<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<br>
Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<br>
Feb 8 09:03:53 ovirtnode01 kernel: device-mapper: multipath: Reinstating path 8:48.<br>
Feb 8 09:03:53 ovirtnode01 kernel: sd 3:0:0:22: alua: port group 01 state A preferred supports tolUsNA<br>
Feb 8 09:03:53 ovirtnode01 sanlock[5192]: 2017-02-08 09:03:53+0100 151809 [11772]: s59 add_lockspace fail result -202<br>
Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: remove map (uevent)<br>
Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: devmap not registered, can't remove<br>
Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: remove map (uevent)<br>
Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: remove map (uevent)<br>
Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: devmap not registered, can't remove<br>
Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: remove map (uevent)<br>
Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: remove map (uevent)<br>
Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: devmap not registered, can't remove<br>
Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: remove map (uevent)<br>
Feb 8 09:04:08 ovirtnode01 kernel: dd: sending ioctl 80306d02 to a partition!<br>
Feb 8 09:04:24 ovirtnode01 sanlock[5192]: 2017-02-08 09:04:24+0100 151840 [15589]: read_sectors delta_leader offset 2560 rv -202 /dev/f9b70017-0a34-47bc-bf2f-dfc70200a347/ids<br>
Feb 8 09:04:34 ovirtnode01 sanlock[5192]: 2017-02-08 09:04:34+0100 151850 [15589]: f9b70017 close_task_aio 0 0x7fd78c0008c0 busy<br>
Feb 8 09:04:39 ovirtnode01 multipathd: 360060160422143002a38935800ae2760: sdd - emc_clariion_checker: Read error for WWN 60060160422143002a38935800ae2760. Sense data are 0x0/0x0/0x0.<br>
Feb 8 09:04:39 ovirtnode01 multipathd: checker failed path 8:48 in map 360060160422143002a38935800ae2760<br>
Feb 8 09:04:39 ovirtnode01 multipathd: 360060160422143002a38935800ae2760: remaining active paths: 0<br>
Feb 8 09:04:39 ovirtnode01 kernel: qla2xxx [0000:11:00.0]-801c:3: Abort command issued nexus=3:0:22 -- 1 2002.<br>
Feb 8 09:04:39 ovirtnode01 kernel: device-mapper: multipath: Failing path 8:48.<br>
Feb 8 09:04:40 ovirtnode01 kernel: qla2xxx [0000:11:00.0]-801c:3: Abort command issued nexus=3:0:22 -- 1 2002.<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: 8 callbacks suppressed<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<br>
Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<br>
<br>
Maybe you should consult the storage vendor about this?<br>
<br>
Can be also incorrect multipath configuration, maybe multipatch checker,<br>
fail, and because you have one path the device moved to faulty state, and <br>
sanlock fail to access the device.<br>
<br>
Can also be low level issue in kernel, hba, switch, server.<br>
<br>
Lets start by inspecting multipath configuration, can you share<br>
output of:<br>
<br>
cat /etc/multiapth.conf<br>
multipath -r v3<br>
<br>
Maybe you can expose one lun for testing, and blacklist this lun in <br>
multipath.conf. You will not be able to use this lun in ovirt, but it can<br>
be used to validate the layers below multipath. If a plain lun is ok, <br>
and same lun used a multipath device fails, the problem is likely to be<br>
multipath configuration.<br>
<br>
Nir<br>
<br>
<br>
<br>
multipath -ll output for this Domain<br>
<br>
360060160422143002a38935800ae2760 dm-10 DGC ,VRAID<br>
size=2.0T features='1 retain_attached_hw_handler' hwhandler='1 alua' wp=rw<br>
`-+- policy='service-time 0' prio=50 status=active<br>
`- 3:0:0:22 sdd 8:48 active ready running<br>
<br>
<br>
Thx Christian<br>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">mailto:<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>