<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Feb 8, 2017 at 6:11 PM, Grundmann, Christian <span dir="ltr"><<a href="mailto:Christian.Grundmann@fabasoft.com" target="_blank">Christian.Grundmann@fabasoft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="DE-AT" link="#0563C1" vlink="#954F72">
<div class="m_1985796589679391807WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">got a new FC Storage (EMC Unity 300F) which is seen by my Hosts additional to my old Storage for Migration.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">New Storage has only on PATH until Migration is done.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I already have a few VMs running on the new Storage without Problem.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">But after starting some VMs (don’t really no whats the difference to working ones), the Path for new Storage fails.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Engine tells me: Storage Domain <storagedomain> experienced a high latency of 22.4875 seconds from host <host><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Where can I start looking?</span></p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="DE-AT" link="#0563C1" vlink="#954F72"><div class="m_1985796589679391807WordSection1"><p class="MsoNormal"><span lang="EN-US"> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">In /var/log/messages I found:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 multipathd: 360060160422143002a38935800ae2<wbr>760: sdd - emc_clariion_checker: Active path is healthy.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 multipathd: 8:48: reinstated<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 multipathd: 360060160422143002a38935800ae2<wbr>760: remaining active paths: 1<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 8<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 5833475<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 5833475<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: Buffer I/O error on dev dm-207, logical block 97, async page read<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: device-mapper: multipath: Reinstating path 8:48.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 kernel: sd 3:0:0:22: alua: port group 01 state A preferred supports tolUsNA<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:03:53 ovirtnode01 sanlock[5192]: 2017-02-08 09:03:53+0100 151809 [11772]: s59 add_lockspace fail result -202<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: devmap not registered, can't remove<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:05 ovirtnode01 multipathd: dm-33: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: devmap not registered, can't remove<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:06 ovirtnode01 multipathd: dm-34: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: devmap not registered, can't remove<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:08 ovirtnode01 multipathd: dm-33: remove map (uevent)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:08 ovirtnode01 kernel: dd: sending ioctl 80306d02 to a partition!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:24 ovirtnode01 sanlock[5192]: 2017-02-08 09:04:24+0100 151840 [15589]: read_sectors delta_leader offset 2560 rv -202 /dev/f9b70017-0a34-47bc-bf2f-<wbr>dfc70200a347/ids<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:34 ovirtnode01 sanlock[5192]: 2017-02-08 09:04:34+0100 151850 [15589]: f9b70017 close_task_aio 0 0x7fd78c0008c0 busy<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:39 ovirtnode01 multipathd: 360060160422143002a38935800ae2<wbr>760: sdd - emc_clariion_checker: Read error for WWN 60060160422143002a38935800ae27<wbr>60. Sense data are 0x0/0x0/0x0.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:39 ovirtnode01 multipathd: checker failed path 8:48 in map 360060160422143002a38935800ae2<wbr>760<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:39 ovirtnode01 multipathd: 360060160422143002a38935800ae2<wbr>760: remaining active paths: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:39 ovirtnode01 kernel: qla2xxx [0000:11:00.0]-801c:3: Abort command issued nexus=3:0:22 -- 1 2002.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:39 ovirtnode01 kernel: device-mapper: multipath: Failing path 8:48.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:40 ovirtnode01 kernel: qla2xxx [0000:11:00.0]-801c:3: Abort command issued nexus=3:0:22 -- 1 2002.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: 8 callbacks suppressed<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967168<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 4294967280<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Feb 8 09:04:42 ovirtnode01 kernel: blk_update_request: I/O error, dev dm-10, sector 0</span></p></div></div></blockquote><div><br></div><div>Maybe you should consult the storage vendor about this?</div><div><br></div><div>Can be also incorrect multipath configuration, maybe multipatch checker,</div><div>fail, and because you have one path the device moved to faulty state, and </div><div>sanlock fail to access the device.</div><div><br></div><div>Can also be low level issue in kernel, hba, switch, server.</div><div><br></div><div>Lets start by inspecting multipath configuration, can you share</div><div>output of:</div><div><br></div><div>cat /etc/multiapth.conf</div><div>multipath -r v3</div><div><br></div><div>Maybe you can expose one lun for testing, and blacklist this lun in </div><div>multipath.conf. You will not be able to use this lun in ovirt, but it can</div><div>be used to validate the layers below multipath. If a plain lun is ok, </div><div>and same lun used a multipath device fails, the problem is likely to be</div><div>multipath configuration.</div><div> </div><div>Nir</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="DE-AT" link="#0563C1" vlink="#954F72"><div class="m_1985796589679391807WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">multipath -ll output for this Domain<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">360060160422143002a38935800ae2<wbr>760 dm-10 DGC ,VRAID<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">size=2.0T features='1 retain_attached_hw_handler' hwhandler='1 alua' wp=rw<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">`-+- policy='service-time 0' prio=50 status=active<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> `- 3:0:0:22 sdd 8:48 active ready running<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thx Christian<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>