Hello Guys,<br><br>Hope you can help on this one.<br><br>I have a fresh install of ovirt-engine 3.1, it was working fine since the installation 4 days ago, now the Ovirt-Engine is experiencing issue while I tried to shutdown a VM on the management interfaces. Since then the is not going further after I try to login with the right credentials.<br>
<br>I try to reboot several times<br><br>here the log that I have <br><br>engine.log :<br>2012-10-28 08:55:00,023 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 storage domains<br>
2012-10-28 08:55:00,023 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains done<br>2012-10-28 08:55:14,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [23d9d0ed] Domain 8da021a8-d5d7-4dd0-bdaf-d9f317757705 recovered from problem. vds: <a href="http://hypervisor02-b.in.miromedia.ca">hypervisor02-b.in.miromedia.ca</a><br>
2012-10-28 08:55:14,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [23d9d0ed] Domain 8da021a8-d5d7-4dd0-bdaf-d9f317757705 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.<br>
2012-10-28 09:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable hosts<br>2012-10-28 09:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 hosts<br>
2012-10-28 09:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable hosts done<br>2012-10-28 09:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains<br>
2012-10-28 09:00:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 storage domains<br>2012-10-28 09:00:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains done<br>
<br>server.log :<br>2012-10-28 08:06:32,260 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService]] (ajp--0.0.0.0-8009-13) Servlet.service() for servlet org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService threw exception: java.lang.RuntimeException: Unable to report failure<br>
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doUnexpectedFailure(AbstractRemoteServiceServlet.java:107) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:67) [gwt-servlet-2.3.0.jar:]<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-3.0-api.jar:1.0.1.Final]<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-3.0-api.jar:1.0.1.Final]<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jboss-web.jar:]<br> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jboss-web.jar:]<br>
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jboss-web.jar:]<br> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jboss-web.jar:]<br>
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]<br> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jboss-web.jar:]<br>
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jboss-web.jar:]<br> at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jboss-web.jar:]<br> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jboss-web.jar:]<br>
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jboss-web.jar:]<br> at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jboss-web.jar:]<br> at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jboss-web.jar:]<br>
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jboss-web.jar:]<br> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]<br>Caused by: java.lang.RuntimeException: Unable to continue<br>
at com.google.gwt.rpc.client.ast.RpcCommandVisitor.halt(RpcCommandVisitor.java:337) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:520) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:493) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.visit(WebModePayloadSink.java:407) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.client.ast.ReturnCommand.traverse(ReturnCommand.java:44) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.client.ast.RpcCommandVisitor.doAccept(RpcCommandVisitor.java:320) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.client.ast.RpcCommandVisitor.accept(RpcCommandVisitor.java:42) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.server.WebModePayloadSink.accept(WebModePayloadSink.java:883) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.server.RPC.streamResponse(RPC.java:472) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:198) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222) [gwt-servlet-2.3.0.jar:]<br> at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) [gwt-servlet-2.3.0.jar:]<br>
... 16 more<br>Caused by: com.google.gwt.user.client.rpc.SerializationException: Could not send data<br> at com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:923) [gwt-servlet-2.3.0.jar:]<br>
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:518) [gwt-servlet-2.3.0.jar:]<br> ... 27 more<br>Caused by: ClientAbortException: java.net.SocketException: Broken pipe<br>
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) [jboss-web.jar:]<br> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449) [jboss-web.jar:]<br> at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349) [jboss-web.jar:]<br>
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426) [jboss-web.jar:]<br> at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415) [jboss-web.jar:]<br> at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) [jboss-web.jar:]<br>
at com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:921) [gwt-servlet-2.3.0.jar:]<br> ... 28 more<br>Caused by: java.net.SocketException: Broken pipe<br> at java.net.SocketOutputStream.socketWrite0(Native Method) [rt.jar:1.7.0_09-icedtea]<br>
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) [rt.jar:1.7.0_09-icedtea]<br> at java.net.SocketOutputStream.write(SocketOutputStream.java:153) [rt.jar:1.7.0_09-icedtea]<br> at org.apache.coyote.ajp.AjpProcessor$SocketOutputBuffer.doWrite(AjpProcessor.java:1337) [jboss-web.jar:]<br>
at org.apache.coyote.Response.doWrite(Response.java:594) [jboss-web.jar:]<br> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398) [jboss-web.jar:]<br> ... 34 more<br><br>
<br>httpd error_log :<br>[Sun Oct 28 09:08:06 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>[Sun Oct 28 09:09:26 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>
[Sun Oct 28 09:09:31 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>[Sun Oct 28 09:09:46 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>
[Sun Oct 28 09:10:46 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>[Sun Oct 28 09:10:51 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>
[Sun Oct 28 09:11:11 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>[Sun Oct 28 09:11:27 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>
[Sun Oct 28 09:12:07 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>[Sun Oct 28 09:12:11 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header<br>
<br>iptables rules :<br>Chain INPUT (policy ACCEPT 0 packets, 0 bytes)<br> pkts bytes target prot opt in out source destination <br>45397 17M RH-Firewall-1-INPUT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> <br>
<br>Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)<br> pkts bytes target prot opt in out source destination <br> 0 0 RH-Firewall-1-INPUT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> <br>
<br>Chain OUTPUT (policy ACCEPT 48291 packets, 15M bytes)<br> pkts bytes target prot opt in out source destination <br><br>Chain RH-Firewall-1-INPUT (2 references)<br> pkts bytes target prot opt in out source destination <br>
24226 8281K ACCEPT all -- lo * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> <br> 0 0 ACCEPT icmp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> icmptype 255<br>
20372 9116K ACCEPT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state RELATED,ESTABLISHED<br> 1 60 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:22<br>
0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:80<br> 103 6180 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:443<br>
0 0 ACCEPT udp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW udp dpt:111<br> 0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:111<br>
0 0 ACCEPT udp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW udp dpt:892<br> 0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:892<br>
0 0 ACCEPT udp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW udp dpt:875<br> 0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:875<br>
0 0 ACCEPT udp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW udp dpt:662<br> 0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:662<br>
1 60 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:2049<br> 0 0 ACCEPT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW tcp dpt:32803<br>
0 0 ACCEPT udp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> state NEW udp dpt:32769<br> 694 76288 REJECT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-host-prohibited<br>
<br><br>LISTEN PORT :<br>tcp 0 0 <a href="http://0.0.0.0:4447">0.0.0.0:4447</a> 0.0.0.0:* LISTEN 663/-server <br>tcp 0 0 <a href="http://0.0.0.0:2049">0.0.0.0:2049</a> 0.0.0.0:* LISTEN - <br>
tcp 0 0 <a href="http://0.0.0.0:32803">0.0.0.0:32803</a> 0.0.0.0:* LISTEN - <br>tcp 0 0 <a href="http://127.0.0.1:9990">127.0.0.1:9990</a> 0.0.0.0:* LISTEN 663/-server <br>
tcp 0 0 <a href="http://0.0.0.0:8009">0.0.0.0:8009</a> 0.0.0.0:* LISTEN 663/-server <br>tcp 0 0 <a href="http://127.0.0.1:9999">127.0.0.1:9999</a> 0.0.0.0:* LISTEN 663/-server <br>
tcp 0 0 <a href="http://0.0.0.0:111">0.0.0.0:111</a> 0.0.0.0:* LISTEN 589/rpcbind <br>tcp 0 0 <a href="http://0.0.0.0:8080">0.0.0.0:8080</a> 0.0.0.0:* LISTEN 663/-server <br>
tcp 0 0 <a href="http://0.0.0.0:20048">0.0.0.0:20048</a> 0.0.0.0:* LISTEN 641/rpc.mountd <br>tcp 0 0 <a href="http://0.0.0.0:22">0.0.0.0:22</a> 0.0.0.0:* LISTEN 600/sshd <br>
tcp 0 0 <a href="http://127.0.0.1:5432">127.0.0.1:5432</a> 0.0.0.0:* LISTEN 620/postgres <br>tcp 0 0 <a href="http://0.0.0.0:53144">0.0.0.0:53144</a> 0.0.0.0:* LISTEN 619/rpc.statd <br>
tcp 0 0 <a href="http://127.0.0.1:25">127.0.0.1:25</a> 0.0.0.0:* LISTEN 890/sendmail: accep <br>tcp 0 0 <a href="http://0.0.0.0:8443">0.0.0.0:8443</a> 0.0.0.0:* LISTEN 663/-server <br>
tcp6 0 0 :::2049 :::* LISTEN - <br>tcp6 0 0 :::32803 :::* LISTEN - <br>tcp6 0 0 :::111 :::* LISTEN 589/rpcbind <br>
tcp6 0 0 :::80 :::* LISTEN 645/httpd <br>tcp6 0 0 :::20048 :::* LISTEN 641/rpc.mountd <br>tcp6 0 0 :::22 :::* LISTEN 600/sshd <br>
tcp6 0 0 ::1:5432 :::* LISTEN 620/postgres <br>tcp6 0 0 :::443 :::* LISTEN 645/httpd <br>tcp6 0 0 :::44413 :::* LISTEN 619/rpc.statd <br>