Hello Guys,
Hope you can help on this one.
I have a fresh install of ovirt-engine 3.1, it was working fine since the installation 4 days ago, now the Ovirt-Engine is experiencing issue while I tried to shutdown a VM on the management interfaces. Since then the is not going further after I try to login with the right credentials.
I try to reboot several times
here the log that I have
engine.log :
2012-10-28 08:55:00,023 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 storage domains
2012-10-28 08:55:00,023 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains done
2012-10-28 08:55:14,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [23d9d0ed] Domain 8da021a8-d5d7-4dd0-bdaf-d9f317757705 recovered from problem. vds: hypervisor02-b.in.miromedia.ca
2012-10-28 08:55:14,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [23d9d0ed] Domain 8da021a8-d5d7-4dd0-bdaf-d9f317757705 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2012-10-28 09:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable hosts
2012-10-28 09:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 hosts
2012-10-28 09:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable hosts done
2012-10-28 09:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains
2012-10-28 09:00:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 0 storage domains
2012-10-28 09:00:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Checking autorecoverable storage domains done
server.log :
2012-10-28 08:06:32,260 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService]] (ajp--0.0.0.0-8009-13) Servlet.service() for servlet org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService threw exception: java.lang.RuntimeException: Unable to report failure
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doUnexpectedFailure(AbstractRemoteServiceServlet.java:107) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:67) [gwt-servlet-2.3.0.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-3.0-api.jar:1.0.1.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-3.0-api.jar:1.0.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jboss-web.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jboss-web.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jboss-web.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jboss-web.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jboss-web.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jboss-web.jar:]
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jboss-web.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jboss-web.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jboss-web.jar:]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jboss-web.jar:]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jboss-web.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jboss-web.jar:]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.lang.RuntimeException: Unable to continue
at com.google.gwt.rpc.client.ast.RpcCommandVisitor.halt(RpcCommandVisitor.java:337) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:520) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:493) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.visit(WebModePayloadSink.java:407) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.client.ast.ReturnCommand.traverse(ReturnCommand.java:44) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.client.ast.RpcCommandVisitor.doAccept(RpcCommandVisitor.java:320) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.client.ast.RpcCommandVisitor.accept(RpcCommandVisitor.java:42) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink.accept(WebModePayloadSink.java:883) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.RPC.streamResponse(RPC.java:472) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:198) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) [gwt-servlet-2.3.0.jar:]
... 16 more
Caused by: com.google.gwt.user.client.rpc.SerializationException: Could not send data
at com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:923) [gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:518) [gwt-servlet-2.3.0.jar:]
... 27 more
Caused by: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) [jboss-web.jar:]
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449) [jboss-web.jar:]
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349) [jboss-web.jar:]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426) [jboss-web.jar:]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415) [jboss-web.jar:]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) [jboss-web.jar:]
at com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:921) [gwt-servlet-2.3.0.jar:]
... 28 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) [rt.jar:1.7.0_09-icedtea]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) [rt.jar:1.7.0_09-icedtea]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153) [rt.jar:1.7.0_09-icedtea]
at org.apache.coyote.ajp.AjpProcessor$SocketOutputBuffer.doWrite(AjpProcessor.java:1337) [jboss-web.jar:]
at org.apache.coyote.Response.doWrite(Response.java:594) [jboss-web.jar:]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398) [jboss-web.jar:]
... 34 more
httpd error_log :
[Sun Oct 28 09:08:06 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:26 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:31 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:46 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:46 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:51 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:11 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:27 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:07 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:11 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
iptables rules :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
45397 17M RH-Firewall-1-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 RH-Firewall-1-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 48291 packets, 15M bytes)
pkts bytes target prot opt in out source destination
Chain RH-Firewall-1-INPUT (2 references)
pkts bytes target prot opt in out source destination
24226 8281K ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 255
20372 9116K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
103 6180 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:111
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:111
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:892
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:892
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:875
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:875
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:662
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:662
1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2049
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:32803
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:32769
694 76288 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
LISTEN PORT :
tcp 0 0 0.0.0.0:4447 0.0.0.0:* LISTEN 663/-server
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:32803 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:9990 0.0.0.0:* LISTEN 663/-server
tcp 0 0 0.0.0.0:8009 0.0.0.0:* LISTEN 663/-server
tcp 0 0 127.0.0.1:9999 0.0.0.0:* LISTEN 663/-server
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 589/rpcbind
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 663/-server
tcp 0 0 0.0.0.0:20048 0.0.0.0:* LISTEN 641/rpc.mountd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 600/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 620/postgres
tcp 0 0 0.0.0.0:53144 0.0.0.0:* LISTEN 619/rpc.statd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 890/sendmail: accep
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 663/-server
tcp6 0 0 :::2049 :::* LISTEN -
tcp6 0 0 :::32803 :::* LISTEN -
tcp6 0 0 :::111 :::* LISTEN 589/rpcbind
tcp6 0 0 :::80 :::* LISTEN 645/httpd
tcp6 0 0 :::20048 :::* LISTEN 641/rpc.mountd
tcp6 0 0 :::22 :::* LISTEN 600/sshd
tcp6 0 0 ::1:5432 :::* LISTEN 620/postgres
tcp6 0 0 :::443 :::* LISTEN 645/httpd
tcp6 0 0 :::44413 :::* LISTEN 619/rpc.statd