Since upgrading our ovirt setup from 3.2 to 3.3 we have run into some issues with the ovirt-engine becoming non responsive. The first few times we just cycled the server and moved on with things, but the problem kept re-occurring. After digging into the server.log we identified a number of errors, some samples below
2014-09-18 03:20:06,115 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015004: Caught exception writing deployment marker file /var/tmp/ovirt-engine/deployments/engine.ear.pending: java.io.FileNotFoundException: /var/tmp/ovirt-engine/deployments/engine.ear.pending (No such file or directory)
2014-09-18 03:20:06,121 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) JBAS015004: Caught exception writing deployment marker file /var/tmp/ovirt-engine/deployments/engine.ear.isundeploying: java.io.FileNotFoundException: /var/tmp/ovirt-engine/deployments/engine.ear.isundeploying (No such file or directory)
I didn't see any similar messages posted to the forum. After doing some digging we found that the OS delivered /etc/cron.daily/tmpwatch was the culprit. It has an line that cleans up /var/tmp.
flags=-umc
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
The tmpwatch combined with the noatime that we have set in fstab causes these files to be deleted. Our immediate solution will be to remove the noatime from fstab. Should these files be moved to a different directory, or can the engine recreate them if they get removed from the tmp directory?