<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 14, 2016 at 1:12 AM, Steve Dainard <span dir="ltr"><<a href="mailto:sdainard@spd1.com" target="_blank">sdainard@spd1.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I had a hypervisor semi-crash this week, 4 of ~10 VM's continued to run, but the others were killed off somehow and all VM's running on this host had '?' status in the ovirt UI.</div><div><br></div><div>This appears to have been caused by vdsm logs filling up disk space on the logging partition.</div><div><br></div><div>I've attached the log file vdsm.log.27.xz which shows this error:</div><div><br></div><div>vdsm.Scheduler::DEBUG::2016-<wbr>10-11 16:42:09,318::executor::216::<wbr>Executor::(_discard) Worker discarded: <Worker name=periodic/3017 running <Operation action=<VmDispatcher operation=<class 'virt.periodic.<wbr>DriveWatermarkMonitor'> at 0x7f8e90021210> at 0x7f8e90021250> discarded at 0x7f8dd123e850><br></div><div><br></div><div>which happens more and more frequently throughout the log.</div><div><br></div><div>It was a bit difficult to understand what caused the failure, but the logs were getting really large, then being xz'd which compressed 11G+ into a few MB. Once this happened the disk space would be freed, and nagios wouldn't hit the 3rd check to throw a warning, until pretty much right at the crash.</div><div><br></div><div>I was able to restart vdsmd to resolve the issue, but I still need to know why these logs started to stack up so I can avoid this issue in the future.</div></div></blockquote><div><br></div><div>We had this one: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1383259">https://bugzilla.redhat.com/show_bug.cgi?id=1383259</a></div><div>but in your case the logs are rotating.</div><div>Francesco?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Hypervisor host info:</div><div>CentOS 7</div><div><div># rpm -qa | grep vdsm</div><div>vdsm-yajsonrpc-4.17.32-1.el7.<wbr>noarch</div><div>vdsm-xmlrpc-4.17.32-1.el7.<wbr>noarch</div><div>vdsm-infra-4.17.32-1.el7.<wbr>noarch</div><div>vdsm-hook-vmfex-dev-4.17.32-1.<wbr>el7.noarch</div><div>vdsm-python-4.17.32-1.el7.<wbr>noarch</div><div>vdsm-4.17.32-1.el7.noarch</div><div>vdsm-cli-4.17.32-1.el7.noarch</div><div>vdsm-jsonrpc-4.17.32-1.el7.<wbr>noarch</div></div><div><br></div><div>Engine host info:</div><div>CentOS 7</div><div><div>$ rpm -qa | grep ovirt</div><div>ovirt-engine-lib-3.6.7.5-1.<wbr>el7.centos.noarch</div><div>ovirt-iso-uploader-3.6.0-1.<wbr>el7.centos.noarch</div><div>ovirt-engine-wildfly-overlay-<wbr>8.0.5-1.el7.noarch</div><div>ovirt-engine-webadmin-portal-<wbr>3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-jboss-as-7.1.1-1.<wbr>el7.x86_64</div><div>ovirt-engine-setup-plugin-<wbr>vmconsole-proxy-helper-3.6.7.<wbr>5-1.el7.centos.noarch</div><div>ovirt-host-deploy-1.4.1-1.el7.<wbr>centos.noarch</div><div>ovirt-engine-vmconsole-proxy-<wbr>helper-3.6.7.5-1.el7.centos.<wbr>noarch</div><div>ovirt-engine-backend-3.6.7.5-<wbr>1.el7.centos.noarch</div><div>ovirt-setup-lib-1.0.1-1.el7.<wbr>centos.noarch</div><div>ovirt-engine-setup-plugin-<wbr>websocket-proxy-3.6.7.5-1.el7.<wbr>centos.noarch</div><div>ovirt-engine-websocket-proxy-<wbr>3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-tools-3.6.7.5-1.<wbr>el7.centos.noarch</div><div>ovirt-engine-setup-base-3.6.7.<wbr>5-1.el7.centos.noarch</div><div>ovirt-engine-setup-3.6.7.5-1.<wbr>el7.centos.noarch</div><div>ovirt-vmconsole-1.0.2-1.el7.<wbr>centos.noarch</div><div>ovirt-engine-wildfly-8.2.1-1.<wbr>el7.x86_64</div><div>ovirt-engine-tools-backup-3.6.<wbr>7.5-1.el7.centos.noarch</div><div>ovirt-engine-userportal-3.6.7.<wbr>5-1.el7.centos.noarch</div><div>ovirt-engine-3.6.7.5-1.el7.<wbr>centos.noarch</div><div>ovirt-release35-006-1.noarch</div><div>ovirt-engine-extension-aaa-<wbr>ldap-1.1.0-0.0.master.<wbr>20151021074904.git92c5c31.el7.<wbr>noarch</div><div>ovirt-release36-3.6.7-1.noarch</div><div>ovirt-engine-setup-plugin-<wbr>ovirt-engine-3.6.7.5-1.el7.<wbr>centos.noarch</div><div>ovirt-host-deploy-java-1.4.1-<wbr>1.el7.centos.noarch</div><div>ovirt-image-uploader-3.6.0-1.<wbr>el7.centos.noarch</div><div>ovirt-engine-dbscripts-3.6.7.<wbr>5-1.el7.centos.noarch</div><div>ovirt-engine-sdk-python-3.6.3.<wbr>0-1.el7.noarch</div><div>ovirt-engine-extension-aaa-<wbr>jdbc-1.0.7-1.el7.noarch</div><div>ovirt-engine-extensions-api-<wbr>impl-3.6.7.5-1.el7.centos.<wbr>noarch</div><div>ovirt-engine-restapi-3.6.7.5-<wbr>1.el7.centos.noarch</div><div>ovirt-engine-setup-plugin-<wbr>ovirt-engine-common-3.6.7.5-1.<wbr>el7.centos.noarch</div><div>ovirt-vmconsole-proxy-1.0.2-1.<wbr>el7.centos.noarch</div><div>ovirt-engine-cli-3.6.2.0-1.<wbr>el7.centos.noarch</div></div><div><br></div><div><br></div><div>Thanks,</div><div>Steve</div></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>