<div dir="ltr">I asked on the Dell Storage Forum and they recommend the following:<div><br></div><div><p style="font-family:arial,helvetica,sans-serif;font-size:13.3333px"><i>I recommend not using a numeric value for the "no_path_retry" variable within /etc/multipath.conf as once that numeric value is reached, if no healthy LUNs were discovered during that defined time multipath will disable the I/O queue altogether.</i></p><p style="font-family:arial,helvetica,sans-serif;font-size:13.3333px"><i>I do recommend, however, changing the variable value from "12" (or even "60") to "queue" which will then allow multipathd to continue queing I/O until a healthy LUN is discovered (time of fail-over between controllers) and I/O is allowed to flow once again.</i></p><p style="font-family:arial,helvetica,sans-serif;font-size:13.3333px"><span style="font-size:13.3333px">Can you see any issues with this recommendation as far as Ovirt is concerned ?</span></p><p style="font-family:arial,helvetica,sans-serif;font-size:13.3333px">Thanks again</p></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><i>Gary Lloyd</i><div><span style="color:rgb(136,136,136)">______________________________</span><span style="color:rgb(136,136,136)">__________________</span><br><div><span style="color:rgb(136,136,136)">I.T. Systems:Keele University</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">Finance & IT Directorate</span></div><div><span style="color:rgb(136,136,136)">Keele:Staffs:IC1 Building:ST5 5NB:UK</span><br style="color:rgb(136,136,136)"><a href="tel:%2B44%201782%20733073" value="+441782733073" style="color:rgb(17,85,204)" target="_blank">+44 1782 733063</a><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">______________________________</span><span style="color:rgb(136,136,136)">__________________</span></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On 4 October 2016 at 19:11, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 4, 2016 at 10:51 AM, Gary Lloyd <span dir="ltr"><<a href="mailto:g.lloyd@keele.ac.uk" target="_blank">g.lloyd@keele.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi</div><div><br></div><div>We have Ovirt 3.65 with a Dell Equallogic SAN and we use Direct Luns for all our VMs.</div><div>At the weekend during early hours an Equallogic controller failed over to its standby on one of our arrays and this caused about 20 of our VMs to be paused due to IO problems.</div><div><br></div><div>I have also noticed that this happens during Equallogic firmware upgrades since we moved onto Ovirt 3.65.</div><div><br></div><div>As recommended by Dell disk timeouts within the VMs are set to 60 seconds when they are hosted on an EqualLogic SAN.</div><div><br></div><div>Is there any other timeout value that we can configure in vdsm.conf to stop VMs from getting paused when a controller fails over ?</div></div></blockquote><div><br></div><div>You can set the timeout in multipath.conf.</div><div><br></div><div>With current multipath configuration (deployed by vdsm), when all paths to a device</div><div>are lost (e.g. you take down all ports on the server during upgrade), all io will fail</div><div>immediately.</div><div><br></div><div>If you want to allow 60 seconds gracetime in such case, you can configure:</div><div><br></div><div> no_path_retry 12</div><div><br></div><div>This will continue to monitor the paths 12 times, each 5 seconds </div><div>(assuming polling_interval=5). If some path recover during this time, the io</div><div>can complete and the vm will not be paused.</div><div><br></div><div>If no path is available after these retries, io will fail and vms with pending io</div><div>will pause.</div><div><br></div><div>Note that this will also cause delays in vdsm in various flows, increasing the chance</div><div>of timeouts in engine side, or delays in storage domain monitoring.</div><div><br></div><div>However, the 60 seconds delay is expected only on the first time all paths become</div><div>faulty. Once the timeout has expired, any access to the device will fail immediately.</div><div><br></div><div>To configure this, you must add the # VDSM PRIVATE tag at the second line of</div><div>multipath.conf, otherwise vdsm will override your configuration in the next time</div><div>you run vdsm-tool configure.</div><div><br></div><div>multipath.conf should look like this:</div><div><br></div><div><font face="monospace, monospace"># VDSM REVISION 1.3<br></font></div><div><font face="monospace, monospace"># VDSM PRIVATE<br></font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">defaults {</font></div><div><font face="monospace, monospace"> polling_interval 5</font></div><div><font face="monospace, monospace"> no_path_retry 12</font></div><div><font face="monospace, monospace"> user_friendly_names no</font></div><div><font face="monospace, monospace"> flush_on_last_del yes</font></div><div><font face="monospace, monospace"> fast_io_fail_tmo 5</font></div><div><font face="monospace, monospace"> dev_loss_tmo 30</font></div><div><font face="monospace, monospace"> max_fds 4096</font></div><div><font face="monospace, monospace">}</font></div><div><font face="monospace, monospace"><br></font></div><div><span style="font-family:monospace,monospace">devices {</span><br></div><div><font face="monospace, monospace"> device {</font></div><div><font face="monospace, monospace"> all_devs yes</font></div><div><font face="monospace, monospace"> no_path_retry 12</font></div><div><font face="monospace, monospace"> }</font></div><div><font face="monospace, monospace">}</font></div></div><div><br></div><div><font face="arial, helvetica, sans-serif">This will use 12 retries (60 seconds) timeout for any device. If you like to </font></div><div><font face="arial, helvetica, sans-serif">configure only your specific device, you can add a device section for</font></div><div><font face="arial, helvetica, sans-serif">your specific server instead.</font></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Also is there anything that we can tweak to automatically unpause the VMs once connectivity with the arrays is re-established ?</div></div></blockquote><div><br></div><div>Vdsm will resume the vms when storage monitor detect that storage became available again.</div><div>However we cannot guarantee that storage monitoring will detect that storage was down.</div><div>This should be improved in 4.0.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>At the moment we are running a customized version of storageServer.py, as Ovirt has yet to include iscsi multipath support for Direct Luns out of the box.</div></div></blockquote><div><br></div><div>Would you like to share this code?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Nir</div></font></span></div></div></div>
</blockquote></div><br></div></div>