<div dir="ltr">Hello,<div><br></div><div>I had a hypervisor semi-crash this week, 4 of ~10 VM&#39;s continued to run, but the others were killed off somehow and all VM&#39;s running on this host had &#39;?&#39; status in the ovirt UI.</div><div><br></div><div>This appears to have been caused by vdsm logs filling up disk space on the logging partition.</div><div><br></div><div>I&#39;ve attached the log file vdsm.log.27.xz which shows this error:</div><div><br></div><div>vdsm.Scheduler::DEBUG::2016-10-11 16:42:09,318::executor::216::Executor::(_discard) Worker discarded: &lt;Worker name=periodic/3017 running &lt;Operation action=&lt;VmDispatcher operation=&lt;class &#39;virt.periodic.DriveWatermarkMonitor&#39;&gt; at 0x7f8e90021210&gt; at 0x7f8e90021250&gt; discarded at 0x7f8dd123e850&gt;<br></div><div><br></div><div>which happens more and more frequently throughout the log.</div><div><br></div><div>It was a bit difficult to understand what caused the failure, but the logs were getting really large, then being xz&#39;d which compressed 11G+ into a few MB. Once this happened the disk space would be freed, and nagios wouldn&#39;t hit the 3rd check to throw a warning, until pretty much right at the crash.</div><div><br></div><div>I was able to restart vdsmd to resolve the issue, but I still need to know why these logs started to stack up so I can avoid this issue in the future.</div><div><br></div><div>Hypervisor host info:</div><div>CentOS 7</div><div><div># rpm -qa | grep vdsm</div><div>vdsm-yajsonrpc-4.17.32-1.el7.noarch</div><div>vdsm-xmlrpc-4.17.32-1.el7.noarch</div><div>vdsm-infra-4.17.32-1.el7.noarch</div><div>vdsm-hook-vmfex-dev-4.17.32-1.el7.noarch</div><div>vdsm-python-4.17.32-1.el7.noarch</div><div>vdsm-4.17.32-1.el7.noarch</div><div>vdsm-cli-4.17.32-1.el7.noarch</div><div>vdsm-jsonrpc-4.17.32-1.el7.noarch</div></div><div><br></div><div>Engine host info:</div><div>CentOS 7</div><div><div>$ rpm -qa | grep ovirt</div><div>ovirt-engine-lib-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-iso-uploader-3.6.0-1.el7.centos.noarch</div><div>ovirt-engine-wildfly-overlay-8.0.5-1.el7.noarch</div><div>ovirt-engine-webadmin-portal-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-jboss-as-7.1.1-1.el7.x86_64</div><div>ovirt-engine-setup-plugin-vmconsole-proxy-helper-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-host-deploy-1.4.1-1.el7.centos.noarch</div><div>ovirt-engine-vmconsole-proxy-helper-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-backend-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-setup-lib-1.0.1-1.el7.centos.noarch</div><div>ovirt-engine-setup-plugin-websocket-proxy-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-websocket-proxy-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-tools-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-setup-base-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-setup-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-vmconsole-1.0.2-1.el7.centos.noarch</div><div>ovirt-engine-wildfly-8.2.1-1.el7.x86_64</div><div>ovirt-engine-tools-backup-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-userportal-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-release35-006-1.noarch</div><div>ovirt-engine-extension-aaa-ldap-1.1.0-0.0.master.20151021074904.git92c5c31.el7.noarch</div><div>ovirt-release36-3.6.7-1.noarch</div><div>ovirt-engine-setup-plugin-ovirt-engine-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-host-deploy-java-1.4.1-1.el7.centos.noarch</div><div>ovirt-image-uploader-3.6.0-1.el7.centos.noarch</div><div>ovirt-engine-dbscripts-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-sdk-python-3.6.3.0-1.el7.noarch</div><div>ovirt-engine-extension-aaa-jdbc-1.0.7-1.el7.noarch</div><div>ovirt-engine-extensions-api-impl-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-restapi-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-engine-setup-plugin-ovirt-engine-common-3.6.7.5-1.el7.centos.noarch</div><div>ovirt-vmconsole-proxy-1.0.2-1.el7.centos.noarch</div><div>ovirt-engine-cli-3.6.2.0-1.el7.centos.noarch</div></div><div><br></div><div><br></div><div>Thanks,</div><div>Steve</div></div>