<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 20 Sep 2017, at 21:08, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" class="">ykaul@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote">On Sep 20, 2017 9:50 PM, <<a href="mailto:support@jac-properties.com" class="">support@jac-properties.com</a>> wrote:<br type="attribution" class=""><blockquote class="quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><u class=""></u><div class=""><p class="">This matches about with what we were thinking, thank you!</p><p class="">To answer your questions</p><p class="">We do not have power management configured due to it causing a cascading failure early in our deployment. The host was not fenced and "confirm host rebooted" was never used. The VMs were powered on via virsh (this shouldn't have happened)</p><p class="">The way they were powered on is most likely why they were corrupted is our thought</p><div class=""><br class=""></div></div></blockquote></div></div></div></div></blockquote><div><br class=""></div>yes. </div><div>That’s why we put a basic password protection to the plain virsh access. Easy to circumvent, but then you’re on your own…</div><div><br class=""></div><div>Hm, how exactly were they powered on by virsh? Normally this is not possible for oVirt VMs at all due to initial set up of host-specific things(disk paths), we also use transient libvirt domains so stopped VMs are not defined in libvirt once they stop. So I wonder how exactly was this done?</div><div>Unless they were in Paused state where you indeed can simply continue the execution. </div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">We'd be happy if you could share both engine and host logs, including vdsm.log, engine.log and /var/log/messages from both. </div><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Y. </div><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><font color="#888888" class=""><p class=""><br class=""></p><p class="">Logan</p></font><div class="elided-text"><blockquote type="cite" class=""><div id="m_-4156457467377991239ox-3145df7df0" style="word-wrap: break-word;" class="">On September 20, 2017 at 12:03 PM Michal Skrivanek <<a href="mailto:michal.skrivanek@redhat.com" target="_blank" class="">michal.skrivanek@redhat.com</a>> wrote:<br class=""><br class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 20 Sep 2017, at 18:06, Logan Kuhn <<a href="mailto:support@jac-properties.com" target="_blank" class="">support@jac-properties.com</a>> wrote:</div><br class="m_-4156457467377991239ox-3145df7df0-Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class="">We had an incident where a VM hosts' disk filled up, the VMs all went unknown in the web console, but were fully functional if you were to login or use the services of one.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Hi,</div>yes, that can happen since the VM’s storage is on NAS whereas the server itself is non-functional as the management and all other local processes are using local resources</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class=""> <span class="Apple-converted-space"> </span>We couldn't migrate them so we powered them down on that host and powered them up and let ovirt choose the host for it, same as always. </div></div></div></blockquote><div class=""><br class=""></div><div class="">that’s a mistake. The host should be fenced in that case, you likely do not have a power management configured, do you? Even when you do not have a fencing device available it should have been resolved manually by rebooting it manually(after fixing the disk problem), or in case of permanent damage (e.g. server needs to be replaced, that takes a week, you need to run those VMs in the meantime elsewhere) it should have been powered off and VM states should be reset by “confirm host has been rebooted” manual action.</div><div class=""><br class=""></div><div class="">Normally you should now be able to run those VMs while the status of the host is still Not Responding - was it not the case? How exactly you get to the situation that you were able to power up the VMs?</div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote><div><br class=""></div>sorry, I meant "Normally you should not be able to run those VMs”</div><div><br class=""></div><div>Thanks,</div><div>michal</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="auto" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class="elided-text"><blockquote type="cite" class=""><div id="m_-4156457467377991239ox-3145df7df0" style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class="">However the disk image on a few of them were corrupted because once we fixed the host with the full disk, it still thought it should be running the VM. Which promptly corrupted the disk, the error seems to be this in the logs:</div></div></div></blockquote><div class=""><br class=""></div>this can only happen for VMs flagged as HA, is it a case?</div><div class=""><br class=""></div><div class=""><div class="">Thanks,</div><div class="">michal</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class=""><br class=""></div><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class=""><span style="font-family: monospace;" class=""><span style="background-color: rgb(255, 255, 255);" class=""><span class="m_-4156457467377991239ox-3145df7df0-gmail-Object" id="m_-4156457467377991239ox-3145df7df0-gmail-OBJ_PREFIX_DWT446_com_zimbra_date" style="color: rgb(111, 22, 22);"><span class="m_-4156457467377991239ox-3145df7df0-gmail-Object" id="m_-4156457467377991239ox-3145df7df0-gmail-OBJ_PREFIX_DWT447_com_zimbra_date">2017-09-19</span></span> 21:59:11,058 INFO [org.ovirt.engine.core.<wbr class="">vdsbroker.monitoring.<wbr class="">VmAnalyzer] (DefaultQuartzScheduler3) [36c806f6] VM '70cf75c7-0fc2-4bbe-958e-<wbr class="">7d0095f70960'(testhub) is </span><span style="font-weight: bold; color: rgb(255, 84, 84); background-color: rgb(255, 255, 255);" class="">running</span><span style="background-color: rgb(255, 255, 255);" class=""> in db and not </span><span style="font-weight: bold; color: rgb(255, 84, 84); background-color: rgb(255, 255, 255);" class="">running</span><span style="background-color: rgb(255, 255, 255);" class=""> on VDS 'ef6dc2a3-af6e-4e00-aa4</span><br class="">0-493b31263417'(vm-int7)<br class=""></span></div><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class=""><br class=""></div><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class="">We upgraded to 4.1.6 from 4.0.6 earlier in the day, I don't really think it's anything more than coincidence, but it's worrying enough to send to the community.</div><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class=""><br class=""></div><div style="font-family: arial; font-size: 16px; background-color: rgb(253, 253, 253);" class="">Regards,<br class="">Logan</div></div>______________________________<wbr class="">_________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a><br class=""><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/users</a><br class=""></div></blockquote></div><br class=""></div></blockquote></div></div><br class="">______________________________<wbr class="">_________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class=""><a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/users</a></blockquote></div></div></div></div></blockquote></div><br class=""></body></html>