<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 11, 2017 at 7:35 PM, Mark Greenall <span dir="ltr"><<a href="mailto:m.greenall@iontrading.com" target="_blank">m.greenall@iontrading.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div class="m_-6490983004819797624WordSection1">
<p class="MsoNormal">Hi Ovirt Champions,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am pulling my hair out and in need of advice / help.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Host server: Dell PowerEdge R815 (40 cores and 768GB memory)<u></u><u></u></p>
<p class="MsoNormal">Stoage: Dell Equallogic (Firmware V8.1.4)<u></u><u></u></p>
<p class="MsoNormal">OS: Centos 7.3 (although the same thing happens on 7.2)<u></u><u></u></p>
<p class="MsoNormal">Ovirt: 4.0.6.3-1 (although also happens on 4.0.5)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I can’t exactly pinpoint when this started happening but it’s certainly been happening with Ovirt 4.0.5 and CentOS 7.2. Today I updated Hosted Engine and one host to 4.0.6 and CentOS 7.3 but we still see the same problem. Our hosts are
connected to Dell iSCSI Eqallogic storage. We have one storage domain defined per VM guest, so do have quite a few LUN’s presented to the cluster (around 45 in total).</p></div></div></blockquote><div><br></div><div>Why do you have 1 SD per VM?</div><div><br></div><div>Can you try and disable (mask) the lvmetad service on the hosts and see if it improves matters?</div><div>Also /var/log/messages from the host may give us some clues.</div><div>TIA,</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-GB" link="blue" vlink="purple"><div class="m_-6490983004819797624WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Problem Description:<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Reboot a host.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Activate a host in Ovirt Admin Gui.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>3)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>A few minutes later host is shown as activated.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>4)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Approx 10-15 mins later host goes offline complaining that it can’t connect to storage.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>5)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Constantly then loops around (activating, non operational, connecting, initialising) and the host ends up with a high CPU load and large number of lvm commands in the process tree.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>6)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Multipath and iscsi show all storage is available and logged in.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>7)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Equallogic shows host connected and no errors.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>8)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Admin GUI ends up saying the host can’t connect to storage ‘UNKNOWN’.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The strange thing is that every now and again step 5 doesn’t happen and the host will actually activate again and then stays up. However, it still takes step 4 to take the host offline first.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Expected Behaviour:<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Reboot a host.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Activate a host in Ovirt Admin Gui.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>3)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>A few minutes later host is shown as activated.<u></u><u></u></p>
<p class="m_-6490983004819797624MsoListParagraph"><u></u><span>4)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Begin using host with confidence.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’ve attached the engine.log from Hosted Engine and vdsm.log from the host. The following is a timeline of the latest event.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Host Activation : 15:07<u></u><u></u></p>
<p class="MsoNormal">Host Up: 15:10<u></u><u></u></p>
<p class="MsoNormal">Non-Operational: 15:17<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Seriously hoping someone can spot something obvious as this is making the clusters somewhat unstable and unreliable.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Many Thanks,<u></u><u></u></p>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>