12:13 p.m.
Hello Guys,
Hope you can help on this one.
I have a fresh install of ovirt-engine 3.1, it was working fine since the
installation 4 days ago, now the Ovirt-Engine is experiencing issue while I
tried to shutdown a VM on the management interfaces. Since then the is not
going further after I try to login with the right credentials.
I try to reboot several times
here the log that I have
engine.log :
2012-10-28 08:55:00,023 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 storage domains
2012-10-28 08:55:00,023 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains done
2012-10-28 08:55:14,330 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-23) [23d9d0ed] Domain
8da021a8-d5d7-4dd0-bdaf-d9f317757705 recovered from problem. vds:
hypervisor02-b.in.miromedia.ca
2012-10-28 08:55:14,332 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-23) [23d9d0ed] Domain
8da021a8-d5d7-4dd0-bdaf-d9f317757705 has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain
recovery timer.
2012-10-28 09:00:00,001 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable hosts
2012-10-28 09:00:00,004 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 hosts
2012-10-28 09:00:00,004 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable hosts done
2012-10-28 09:00:00,005 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains
2012-10-28 09:00:00,007 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 storage domains
2012-10-28 09:00:00,008 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains done
server.log :
2012-10-28 08:06:32,260 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[org.ovirt.engine.ui.frontend.WebAdmin
GenericApiGWTService]] (ajp--0.0.0.0-8009-13) Servlet.service() for servlet
org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService threw exception:
java.lang.RuntimeException: Unable to report failure
at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doUnexpectedFailure(AbstractRemoteServiceServlet.java:107)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:67)
[gwt-servlet-2.3.0.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
[jboss-servlet-3.0-api.jar:1.0.1.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
[jboss-servlet-3.0-api.jar:1.0.1.Final]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
[jboss-web.jar:]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jboss-web.jar:]
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
[jboss-web.jar:]
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
[jboss-web.jar:]
at
org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
[jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
[jboss-web.jar:]
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[jboss-web.jar:]
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
[jboss-web.jar:]
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[jboss-web.jar:]
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[jboss-web.jar:]
at
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
[jboss-web.jar:]
at
org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
[jboss-web.jar:]
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
[jboss-web.jar:]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.lang.RuntimeException: Unable to continue
at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.halt(RpcCommandVisitor.java:337)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:520)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:493)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.visit(WebModePayloadSink.java:407)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.client.ast.ReturnCommand.traverse(ReturnCommand.java:44)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.doAccept(RpcCommandVisitor.java:320)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.accept(RpcCommandVisitor.java:42)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink.accept(WebModePayloadSink.java:883)
[gwt-servlet-2.3.0.jar:]
at com.google.gwt.rpc.server.RPC.streamResponse(RPC.java:472)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:198)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
[gwt-servlet-2.3.0.jar:]
... 16 more
Caused by: com.google.gwt.user.client.rpc.SerializationException: Could not
send data
at
com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:923)
[gwt-servlet-2.3.0.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:518)
[gwt-servlet-2.3.0.jar:]
... 27 more
Caused by: ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
[jboss-web.jar:]
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
[jboss-web.jar:]
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
[jboss-web.jar:]
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426)
[jboss-web.jar:]
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415)
[jboss-web.jar:]
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
[jboss-web.jar:]
at
com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:921)
[gwt-servlet-2.3.0.jar:]
... 28 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
[rt.jar:1.7.0_09-icedtea]
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
[rt.jar:1.7.0_09-icedtea]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
[rt.jar:1.7.0_09-icedtea]
at
org.apache.coyote.ajp.AjpProcessor$SocketOutputBuffer.doWrite(AjpProcessor.java:1337)
[jboss-web.jar:]
at org.apache.coyote.Response.doWrite(Response.java:594)
[jboss-web.jar:]
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398)
[jboss-web.jar:]
... 34 more
httpd error_log :
[Sun Oct 28 09:08:06 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:26 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:31 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:46 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:46 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:51 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:11 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:27 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:07 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:11 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
iptables rules :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
45397 17M RH-Firewall-1-INPUT all -- * * 0.0.0.0/0
0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
0 0 RH-Firewall-1-INPUT all -- * * 0.0.0.0/0
0.0.0.0/0
Chain OUTPUT (policy ACCEPT 48291 packets, 15M bytes)
pkts bytes target prot opt in out source
destination
Chain RH-Firewall-1-INPUT (2 references)
pkts bytes target prot opt in out source
destination
24226 8281K ACCEPT all -- lo * 0.0.0.0/0
0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0
0.0.0.0/0 icmptype 255
20372 9116K ACCEPT all -- * * 0.0.0.0/0
0.0.0.0/0 state RELATED,ESTABLISHED
1 60 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:22
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:80
103 6180 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:443
0 0 ACCEPT udp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW udp dpt:111
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:111
0 0 ACCEPT udp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW udp dpt:892
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:892
0 0 ACCEPT udp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW udp dpt:875
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:875
0 0 ACCEPT udp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW udp dpt:662
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:662
1 60 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:2049
0 0 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW tcp dpt:32803
0 0 ACCEPT udp -- * * 0.0.0.0/0
0.0.0.0/0 state NEW udp dpt:32769
694 76288 REJECT all -- * * 0.0.0.0/0
0.0.0.0/0 reject-with icmp-host-prohibited
LISTEN PORT :
tcp 0 0 0.0.0.0:4447 0.0.0.0:*
LISTEN 663/-server
tcp 0 0 0.0.0.0:2049 0.0.0.0:*
LISTEN -
tcp 0 0 0.0.0.0:32803 0.0.0.0:*
LISTEN -
tcp 0 0 127.0.0.1:9990 0.0.0.0:*
LISTEN 663/-server
tcp 0 0 0.0.0.0:8009 0.0.0.0:*
LISTEN 663/-server
tcp 0 0 127.0.0.1:9999 0.0.0.0:*
LISTEN 663/-server
tcp 0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 589/rpcbind
tcp 0 0 0.0.0.0:8080 0.0.0.0:*
LISTEN 663/-server
tcp 0 0 0.0.0.0:20048 0.0.0.0:*
LISTEN 641/rpc.mountd
tcp 0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 600/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:*
LISTEN 620/postgres
tcp 0 0 0.0.0.0:53144 0.0.0.0:*
LISTEN 619/rpc.statd
tcp 0 0 127.0.0.1:25 0.0.0.0:*
LISTEN 890/sendmail: accep
tcp 0 0 0.0.0.0:8443 0.0.0.0:*
LISTEN 663/-server
tcp6 0 0 :::2049 :::*
LISTEN -
tcp6 0 0 :::32803 :::*
LISTEN -
tcp6 0 0 :::111 :::*
LISTEN 589/rpcbind
tcp6 0 0 :::80 :::*
LISTEN 645/httpd
tcp6 0 0 :::20048 :::*
LISTEN 641/rpc.mountd
tcp6 0 0 :::22 :::*
LISTEN 600/sshd
tcp6 0 0 ::1:5432 :::*
LISTEN 620/postgres
tcp6 0 0 :::443 :::*
LISTEN 645/httpd
tcp6 0 0 :::44413 :::*
LISTEN 619/rpc.statd