<div dir="ltr"><span style="font-size:12.8px">Yup, ovirttest1 ran out of disk space on Friday, we recovered it and everything seemed completely normal.</span><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif"><br></font></div><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif">the postgres service is down on the HEVM, but that is because it's on our postgresql cluster, has been for weeks. I can connect to it's database from within the HEVM using the credentials stored at <span style="color:rgb(0,0,0)">/etc/ovirt-engine/engine.<wbr>conf.d/10-setup-database.conf I can tail the logs on the postgres master and ovirt can and does connect to it.</span></font></div><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0)"><br></span></font></div><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0)">However, trying from ovirttest1 I cannot connect to the engine database using those same credentails, should I be able to? It'd make sense to be able to connect to it....</span></font></div><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0)"><br></span></font></div><div style="font-size:12.8px"><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0)">Logan</span></font></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 20, 2017 at 10:14 AM, Alexander Wels <span dir="ltr"><<a href="mailto:awels@redhat.com" target="_blank">awels@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-HOEnZb"><div class="gmail-h5">On Monday, March 20, 2017 9:14:51 AM EDT Logan Kuhn wrote:<br>
> Starting at 1:09am on Saturday the Hosted Engine has been rebooting because<br>
> it failed it's liveliness check. This is due to the webadmin not loading.<br>
> Nothing changed as far as I can tell on the engine since it's last<br>
> successful reboot on Friday afternoon.<br>
><br>
> The engine, dwhd and httpd are all up and do not seem to be reporting<br>
> anything unusual in their respective logs. The engine can talk to the<br>
> database as I can login using the credentials in /etc/ovirt-engine/<a href="http://engine.co" rel="noreferrer" target="_blank">engine.co</a><br>
> nf.d/10-setup-database.conf and the logs on the postgres server are showing<br>
> activity.<br>
><br>
> I tried to run engine-setup but it says it's not in global maintenance even<br>
> though the hosted engine hosts agree that it is. We are on version 4.0.6.3<br>
><br>
> Server, engine and agent logs are attached<br>
><br>
> Regards,<br>
> Logan<br>
<br>
</div></div>Looking at our logs, it appears that on Friday one of your hosts ran out of<br>
disk space in its logs or temp directory. At which point connectivity started<br>
to be spotty. I see a bunch of attempts to migrate VMs away from that host<br>
(ovirttest1). All of them fail. That repeats a ton of times, I forwarded to<br>
Saturday where it appears you had a bunch of stale locks which also repeates a<br>
bunch of time until the engine VM gets restarted.<br>
<br>
Then I see nothing but restarts of the engine and no apparent errors in the<br>
engine log.<br>
<br>
The server log does however reveal this:<br>
2017-03-20 07:04:27,282 ERROR [org.quartz.core.ErrorLogger]<br>
(QuartzOvirtDBScheduler_<wbr>QuartzSchedulerThread) An error occurred while<br>
scanning for the next triggers to fire.: org.quartz.<wbr>JobPersistenceException:<br>
Failed to obtain DB connection from data source 'NMEngineDS':<br>
java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/<br>
ENGINEDataSourceNoJTA' java.sql.SQLException:<br>
javax.resource.<wbr>ResourceException: IJ000470: You are trying to use a connection<br>
factory that has been shut down: java:/ENGINEDataSourceNoJTA [See nested<br>
exception: java.sql.SQLException: Could not retrieve datasource via JNDI url<br>
'java:/ENGINEDataSourceNoJTA' java.sql.SQLException:<br>
javax.resource.<wbr>ResourceException: IJ000470: You are trying to use a connection<br>
factory that has been shut down: java:/ENGINEDataSourceNoJTA]<br>
at<br>
org.quartz.impl.jdbcjobstore.<wbr>JobStoreCMT.<wbr>getNonManagedTXConnection(<wbr>JobStoreCMT.java:<br>
168) [quartz.jar:]<br>
at<br>
org.quartz.impl.jdbcjobstore.<wbr>JobStoreSupport.<wbr>executeInNonManagedTXLock(<wbr>JobStoreSupport.java:<br>
3807) [quartz.jar:]<br>
at<br>
org.quartz.impl.jdbcjobstore.<wbr>JobStoreSupport.<wbr>acquireNextTriggers(<wbr>JobStoreSupport.java:<br>
2751) [quartz.jar:]<br>
at org.quartz.core.<wbr>QuartzSchedulerThread.run(<wbr>QuartzSchedulerThread.java:<br>
264) [quartz.jar:]<br>
Caused by: java.sql.SQLException: Could not retrieve datasource via JNDI url<br>
'java:/ENGINEDataSourceNoJTA' java.sql.SQLException:<br>
javax.resource.<wbr>ResourceException: IJ000470: You are trying to use a connection<br>
factory that has been shut down: java:/ENGINEDataSourceNoJTA<br>
at<br>
org.quartz.utils.<wbr>JNDIConnectionProvider.<wbr>getConnection(<wbr>JNDIConnectionProvider.java:<br>
163) [quartz.jar:]<br>
at<br>
org.quartz.utils.<wbr>DBConnectionManager.<wbr>getConnection(<wbr>DBConnectionManager.java:<br>
108) [quartz.jar:]<br>
at<br>
org.quartz.impl.jdbcjobstore.<wbr>JobStoreCMT.<wbr>getNonManagedTXConnection(<wbr>JobStoreCMT.java:<br>
165) [quartz.jar:]<br>
... 3 more<br>
<br>
Is your postgresql service running? That is the most likely source of the<br>
engine not coming up.<br>
</blockquote></div><br></div></div>