[Users] Ovirt-Engine stop working

28 Oct 2012

      Hello Guys,

Hope you can help on this one.

I have a fresh install of ovirt-engine 3.1, it was working fine since the
installation 4 days ago, now the Ovirt-Engine is experiencing issue while I
tried to shutdown a VM on the management interfaces. Since then the is not
going further after I try to login with the right credentials.

I try to reboot several times

here the log that I have

engine.log :
2012-10-28 08:55:00,023 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 storage domains
2012-10-28 08:55:00,023 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains done
2012-10-28 08:55:14,330 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-23) [23d9d0ed] Domain
8da021a8-d5d7-4dd0-bdaf-d9f317757705 recovered from problem. vds:
hypervisor02-b.in.miromedia.ca
2012-10-28 08:55:14,332 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-23) [23d9d0ed] Domain
8da021a8-d5d7-4dd0-bdaf-d9f317757705 has recovered from problem. No active
host in the DC is reporting it as problematic, so clearing the domain
recovery timer.
2012-10-28 09:00:00,001 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable hosts
2012-10-28 09:00:00,004 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 hosts
2012-10-28 09:00:00,004 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable hosts done
2012-10-28 09:00:00,005 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains
2012-10-28 09:00:00,007 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Autorecovering 0 storage domains
2012-10-28 09:00:00,008 INFO
[org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90)
Checking autorecoverable storage domains done

server.log :
2012-10-28 08:06:32,260 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/webadmin].[org.ovirt.engine.ui.frontend.WebAdmin
GenericApiGWTService]] (ajp--0.0.0.0-8009-13) Servlet.service() for servlet
org.ovirt.engine.ui.frontend.WebAdmin GenericApiGWTService threw exception:
java.lang.RuntimeException: Unable to report failure
        at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doUnexpectedFailure(AbstractRemoteServiceServlet.java:107)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:67)
[gwt-servlet-2.3.0.jar:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
[jboss-servlet-3.0-api.jar:1.0.1.Final]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
[jboss-servlet-3.0-api.jar:1.0.1.Final]
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
[jboss-web.jar:]
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jboss-web.jar:]
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
[jboss-web.jar:]
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
[jboss-web.jar:]
        at
org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
[jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
[jboss-web.jar:]
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[jboss-web.jar:]
        at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
[jboss-web.jar:]
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[jboss-web.jar:]
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[jboss-web.jar:]
        at
org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
[jboss-web.jar:]
        at
org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
[jboss-web.jar:]
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
[jboss-web.jar:]
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.lang.RuntimeException: Unable to continue
        at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.halt(RpcCommandVisitor.java:337)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:520)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:493)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.visit(WebModePayloadSink.java:407)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.client.ast.ReturnCommand.traverse(ReturnCommand.java:44)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.doAccept(RpcCommandVisitor.java:320)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.client.ast.RpcCommandVisitor.accept(RpcCommandVisitor.java:42)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink.accept(WebModePayloadSink.java:883)
[gwt-servlet-2.3.0.jar:]
        at com.google.gwt.rpc.server.RPC.streamResponse(RPC.java:472)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:198)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
[gwt-servlet-2.3.0.jar:]
        ... 16 more
Caused by: com.google.gwt.user.client.rpc.SerializationException: Could not
send data
        at
com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:923)
[gwt-servlet-2.3.0.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink$PayloadVisitor.commit(WebModePayloadSink.java:518)
[gwt-servlet-2.3.0.jar:]
        ... 27 more
Caused by: ClientAbortException:  java.net.SocketException: Broken pipe
        at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
[jboss-web.jar:]
        at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
[jboss-web.jar:]
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
[jboss-web.jar:]
        at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426)
[jboss-web.jar:]
        at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415)
[jboss-web.jar:]
        at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
[jboss-web.jar:]
        at
com.google.gwt.rpc.server.WebModePayloadSink.send(WebModePayloadSink.java:921)
[gwt-servlet-2.3.0.jar:]
        ... 28 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
[rt.jar:1.7.0_09-icedtea]
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
[rt.jar:1.7.0_09-icedtea]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
[rt.jar:1.7.0_09-icedtea]
        at
org.apache.coyote.ajp.AjpProcessor$SocketOutputBuffer.doWrite(AjpProcessor.java:1337)
[jboss-web.jar:]
        at org.apache.coyote.Response.doWrite(Response.java:594)
[jboss-web.jar:]
        at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398)
[jboss-web.jar:]
        ... 34 more

httpd error_log :
[Sun Oct 28 09:08:06 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:26 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:31 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:09:46 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:46 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:10:51 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:11 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:11:27 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:07 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Sun Oct 28 09:12:11 2012] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header

iptables rules :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination
45397   17M RH-Firewall-1-INPUT  all  --  *      *       0.0.0.0/0
0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination
    0     0 RH-Firewall-1-INPUT  all  --  *      *       0.0.0.0/0
0.0.0.0/0

Chain OUTPUT (policy ACCEPT 48291 packets, 15M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain RH-Firewall-1-INPUT (2 references)
 pkts bytes target     prot opt in     out     source
destination
24226 8281K ACCEPT     all  --  lo     *       0.0.0.0/0
0.0.0.0/0
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0
0.0.0.0/0            icmptype 255
20372 9116K ACCEPT     all  --  *      *       0.0.0.0/0
0.0.0.0/0            state RELATED,ESTABLISHED
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:80
  103  6180 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:443
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW udp dpt:111
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:111
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW udp dpt:892
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:892
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW udp dpt:875
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:875
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW udp dpt:662
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:662
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:2049
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW tcp dpt:32803
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            state NEW udp dpt:32769
  694 76288 REJECT     all  --  *      *       0.0.0.0/0
0.0.0.0/0            reject-with icmp-host-prohibited

LISTEN PORT :
tcp        0      0 0.0.0.0:4447            0.0.0.0:*
LISTEN      663/-server
tcp        0      0 0.0.0.0:2049            0.0.0.0:*
LISTEN      -
tcp        0      0 0.0.0.0:32803           0.0.0.0:*
LISTEN      -
tcp        0      0 127.0.0.1:9990          0.0.0.0:*
LISTEN      663/-server
tcp        0      0 0.0.0.0:8009            0.0.0.0:*
LISTEN      663/-server
tcp        0      0 127.0.0.1:9999          0.0.0.0:*
LISTEN      663/-server
tcp        0      0 0.0.0.0:111             0.0.0.0:*
LISTEN      589/rpcbind
tcp        0      0 0.0.0.0:8080            0.0.0.0:*
LISTEN      663/-server
tcp        0      0 0.0.0.0:20048           0.0.0.0:*
LISTEN      641/rpc.mountd
tcp        0      0 0.0.0.0:22              0.0.0.0:*
LISTEN      600/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*
LISTEN      620/postgres
tcp        0      0 0.0.0.0:53144           0.0.0.0:*
LISTEN      619/rpc.statd
tcp        0      0 127.0.0.1:25            0.0.0.0:*
LISTEN      890/sendmail: accep
tcp        0      0 0.0.0.0:8443            0.0.0.0:*
LISTEN      663/-server
tcp6       0      0 :::2049                 :::*
LISTEN      -
tcp6       0      0 :::32803                :::*
LISTEN      -
tcp6       0      0 :::111                  :::*
LISTEN      589/rpcbind
tcp6       0      0 :::80                   :::*
LISTEN      645/httpd
tcp6       0      0 :::20048                :::*
LISTEN      641/rpc.mountd
tcp6       0      0 :::22                   :::*
LISTEN      600/sshd
tcp6       0      0 ::1:5432                :::*
LISTEN      620/postgres
tcp6       0      0 :::443                  :::*
LISTEN      645/httpd
tcp6       0      0 :::44413                :::*
LISTEN      619/rpc.statd

[Users] Ovirt-Engine stop working

EricD