Moving a Hosted Engine from Fedora 20 to CentOS 7
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original. -- John Florian
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
-- John Florian
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <rgolan@redhat.com> wrote:
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate. #install the appliance yum install ovirt-engine-appliance #and then run the setup ovirt-hosted-engine-setup choose "disk" in this stage Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk it will suggest the downloaded appliance automatically See this wiki for reference http://www.ovirt.org/Features/HEApplianceFlow#Testing --
John Florian
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------050301070802050405070805 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit On 10/29/2015 05:49 AM, Roy Golan wrote:
On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org <mailto:jflorian@doubledog.org>> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate.
#install the appliance yum install ovirt-engine-appliance
#and then run the setup ovirt-hosted-engine-setup
choose "disk" in this stage Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk
it will suggest the downloaded appliance automatically See this wiki for reference http://www.ovirt.org/Features/HEApplianceFlow#Testing
Neat! I was unaware of the HEA so I'm happy to learn of this offering. However, my concern is focused more on a procedure that I presently can only imagine looking something akin to: 1. hosted-engine --set-maintenance --mode=global 2. hosted-engine --vm-poweroff 3. ovirt-hosted-engine-setup ... 4. use the exact same host name and IP address 5. hosted-engine --vm-start 6. hosted-engine --set-maintenance --mode=none Or perhaps: 1. create a new guest 2. inside run foo to deploy the HE packages 3. configure it be aware of existing HE 4. run bar manually or wait for the two HE's to sync their state, sort of like a RAID mirror 5. power off the original HE guest and allow the new one to be the sole provider going forward My confusion may simply stem from having forgot much of the process when I deployed my HE back in January this year. But I thought it best to get started on the right foot for the simplest and/or best practice. -- John Florian --------------050301070802050405070805 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 10/29/2015 05:49 AM, Roy Golan wrote:<br> </div> <blockquote cite="mid:CAC_JqcmkCAy91h8z_X1YA5eneyEQSRjqAJ4DU_FbBVxhV09Tig@mail.gmail.com" type="cite"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rgolan@redhat.com" target="_blank">rgolan@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"><span class="">On Wed, Oct 28, 2015 at 10:23 PM, John Florian <span dir="ltr"><<a moz-do-not-send="true" href="mailto:jflorian@doubledog.org" target="_blank">jflorian@doubledog.org</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can somebody please point me to documentation or describe how I should<br> proceed with this task? I see lots of pages for moving from a physical<br> engine to a VM and vice-versa but am having no luck finding how to go<br> about building a new HE to obsolete my original.<br> <span><font color="#888888"><br> </font></span></blockquote> <div><br> </div> </span> <div>using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM <br> </div> </div> </div> </div> </blockquote> <div><br> <br> </div> <div>BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate. <br> <br> </div> <div>#install the appliance <br> </div> <div>yum install ovirt-engine-appliance<br> <br> </div> <div>#and then run the setup<br> </div> <div>ovirt-hosted-engine-setup<br> <br> </div> <div>choose "disk" in this stage<br> <pre>Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk </pre> <pre><span style="font-family:arial,helvetica,sans-serif">it will suggest the downloaded appliance automatically</span> </pre> <pre><span style="font-family:arial,helvetica,sans-serif">See this wiki for reference <a moz-do-not-send="true" href="http://www.ovirt.org/Features/HEApplianceFlow#Testing">http://www.ovirt.org/Features/HEApplianceFlow#Testing</a></span> </pre> <br> </div> </div> </div> </div> </blockquote> <br> Neat! I was unaware of the HEA so I'm happy to learn of this offering. However, my concern is focused more on a procedure that I presently can only imagine looking something akin to:<br> <br> 1. hosted-engine --set-maintenance --mode=global<br> 2. hosted-engine --vm-poweroff<br> 3. ovirt-hosted-engine-setup ...<br> 4. use the exact same host name and IP address<br> 5. hosted-engine --vm-start<br> 6. hosted-engine --set-maintenance --mode=none<br> <br> Or perhaps:<br> <br> 1. create a new guest<br> 2. inside run foo to deploy the HE packages<br> 3. configure it be aware of existing HE<br> 4. run bar manually or wait for the two HE's to sync their state, sort of like a RAID mirror<br> 5. power off the original HE guest and allow the new one to be the sole provider going forward<br> <br> My confusion may simply stem from having forgot much of the process when I deployed my HE back in January this year. But I thought it best to get started on the right foot for the simplest and/or best practice.<br> <pre class="moz-signature" cols="72">-- John Florian </pre> </body> </html> --------------050301070802050405070805--
This is a multi-part message in MIME format. --------------070507020507000903020509 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit On 10/29/2015 05:49 AM, Roy Golan wrote:
On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org <mailto:jflorian@doubledog.org>> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate.
#install the appliance yum install ovirt-engine-appliance
#and then run the setup ovirt-hosted-engine-setup
choose "disk" in this stage Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk
it will suggest the downloaded appliance automatically See this wiki for reference http://www.ovirt.org/Features/HEApplianceFlow#Testing
I'm afraid I'm lost here. Here's a map of my setup: oVirt 3.5.5 hosted engine is named enceladus-f20 (on Fedora 20) I have one oVirt 3.5.5 Host named oberon-f20 (also on Fedora 20) I previously had one other oVirt 3.5.5 Host named ophelia-f20 I took opehlia down, installed CentOS 7 on it and attempted a "hosted-engine --deploy". I can't remember if that was 3.5.5 or 3.6, but I could not add it to my cluster. From what I could gather enceladus-f20 provided an emulation type of pc1.0 while ophelia-c7 didn't seem to have anything that matched, the closest being just "pc". That looked hopelessly complicated to resolve so I thought I'd try again, but this time putting F22 on the ophelia and doing a 3.6 HE deploy saying yes to the redeploy prompt. This time I was met with: Checking for oVirt-Engine status at enceladus-f20.doubledog.org... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ ERROR ] Cannot automatically add the host to cluster Default: Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details. Please check Engine VM configuration. On enceladus-f20, I see: ==> /var/log/ovirt-engine/engine.log <== 2015-11-06 19:17:12,085 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. 2015-11-06 19:17:12,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. ==> /var/log/ovirt-engine/server.log <== 2015-11-06 19:17:13,654 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-1) Client session created 2015-11-06 19:17:13,663 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-2) Server version string: SSH-2.0-OpenSSH_6.9 2015-11-06 19:17:13,667 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1098) at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:357) at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295) at org.apache.sshd.client.session.ClientSessionImpl.handleMessage(ClientSessionImpl.java:266) at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:720) at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277) at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:188) at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:174) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.7.0_79] at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.7.0_79] at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] ==> /var/log/ovirt-engine/engine.log <== 2015-11-06 19:17:13,672 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] Failed to establish session with host hosted_engine_2: java.io.IOException: SSH session closed during connection 'root@ophelia.doubledog.org' at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:309) [uutils.jar:] at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:] at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:465) [bll.jar:] at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:364) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:768) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:347) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:435) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:416) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:374) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.7.0_79] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final] at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view6.runAction(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:264) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] 2015-11-06 19:17:13,708 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] CanDoAction of action AddVds failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server ophelia.doubledog.org,VDS_CANNOT_CONNECT_TO_SERVER 2015-11-06 19:17:13,728 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-9) Operation Failed: [Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.] ==> /var/log/ovirt-engine/server.log <== 2015-11-06 19:17:13,729 WARN [org.jboss.resteasy.core.SynchronousDispatcher] (ajp--127.0.0.1-8702-9) failed to execute: org.ovirt.engine.api.restapi.resource.BaseBackendResource$WebFaultException at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:255) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:230) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:245) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) [restapi-jaxrs.jar:] at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) [restapi-jaxrs.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:] at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79] I don't understand the ssh failure. From enceladus-f20, I can ssh as root into ophelia-f22 and vice versa. I really don't care if it's all Fedora or CentOS or a mix, my primary objective is to just be on 3.6... somehow. -- John Florian --------------070507020507000903020509 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 10/29/2015 05:49 AM, Roy Golan wrote:<br> </div> <blockquote cite="mid:CAC_JqcmkCAy91h8z_X1YA5eneyEQSRjqAJ4DU_FbBVxhV09Tig@mail.gmail.com" type="cite"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rgolan@redhat.com" target="_blank">rgolan@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"><span class="">On Wed, Oct 28, 2015 at 10:23 PM, John Florian <span dir="ltr"><<a moz-do-not-send="true" href="mailto:jflorian@doubledog.org" target="_blank">jflorian@doubledog.org</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can somebody please point me to documentation or describe how I should<br> proceed with this task? I see lots of pages for moving from a physical<br> engine to a VM and vice-versa but am having no luck finding how to go<br> about building a new HE to obsolete my original.<br> <span><font color="#888888"><br> </font></span></blockquote> <div><br> </div> </span> <div>using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM <br> </div> </div> </div> </div> </blockquote> <div><br> <br> </div> <div>BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate. <br> <br> </div> <div>#install the appliance <br> </div> <div>yum install ovirt-engine-appliance<br> <br> </div> <div>#and then run the setup<br> </div> <div>ovirt-hosted-engine-setup<br> <br> </div> <div>choose "disk" in this stage<br> <pre>Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk </pre> <pre><span style="font-family:arial,helvetica,sans-serif">it will suggest the downloaded appliance automatically</span> </pre> <pre><span style="font-family:arial,helvetica,sans-serif">See this wiki for reference <a moz-do-not-send="true" href="http://www.ovirt.org/Features/HEApplianceFlow#Testing">http://www.ovirt.org/Features/HEApplianceFlow#Testing</a></span> </pre> <br> </div> </div> </div> </div> </blockquote> <br> I'm afraid I'm lost here. Here's a map of my setup:<br> <br> oVirt 3.5.5 hosted engine is named enceladus-f20 (on Fedora 20)<br> I have one oVirt 3.5.5 Host named oberon-f20 (also on Fedora 20)<br> I previously had one other oVirt 3.5.5 Host named ophelia-f20<br> <br> I took opehlia down, installed CentOS 7 on it and attempted a "hosted-engine --deploy". I can't remember if that was 3.5.5 or 3.6, but I could not add it to my cluster. From what I could gather enceladus-f20 provided an emulation type of pc1.0 while ophelia-c7 didn't seem to have anything that matched, the closest being just "pc". That looked hopelessly complicated to resolve so I thought I'd try again, but this time putting F22 on the ophelia and doing a 3.6 HE deploy saying yes to the redeploy prompt. This time I was met with:<br> <br> <pre> Checking for oVirt-Engine status at enceladus-f20.doubledog.org...</pre> <pre>[ INFO ] Engine replied: DB Up!Welcome to Health Status!</pre> <pre>[ ERROR ] Cannot automatically add the host to cluster Default: Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details. </pre> <pre> </pre> <pre> Please check Engine VM configuration.</pre> On enceladus-f20, I see:<br> <br> <pre>==> /var/log/ovirt-engine/engine.log <==</pre> <pre>2015-11-06 19:17:12,085 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.</pre> <pre>2015-11-06 19:17:12,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.</pre> <pre> ==> /var/log/ovirt-engine/server.log <==</pre> <pre>2015-11-06 19:17:13,654 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-1) Client session created</pre> <pre>2015-11-06 19:17:13,663 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-2) Server version string: SSH-2.0-OpenSSH_6.9</pre> <pre>2015-11-06 19:17:13,667 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: <a class="moz-txt-link-abbreviated" href="mailto:curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1">curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1</a>)</pre> <pre> at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1098)</pre> <pre> at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:357)</pre> <pre> at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295)</pre> <pre> at org.apache.sshd.client.session.ClientSessionImpl.handleMessage(ClientSessionImpl.java:266)</pre> <pre> at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:720)</pre> <pre> at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277)</pre> <pre> at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)</pre> <pre> at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:188)</pre> <pre> at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:174)</pre> <pre> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.7.0_79]</pre> <pre> at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.7.0_79]</pre> <pre> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.7.0_79]</pre> <pre> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]</pre> <pre> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]</pre> <pre> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]</pre> <pre> </pre> <pre>==> /var/log/ovirt-engine/engine.log <==</pre> <pre>2015-11-06 19:17:13,672 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] Failed to establish session with host hosted_engine_2: java.io.IOException: SSH session closed during connection '<a class="moz-txt-link-abbreviated" href="mailto:root@ophelia.doubledog.org">root@ophelia.doubledog.org</a>'</pre> <pre> at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:309) [uutils.jar:]</pre> <pre> at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:465) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:364) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:768) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:347) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:435) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:416) [bll.jar:]</pre> <pre> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:374) [bll.jar:]</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]</pre> <pre> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]</pre> <pre> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]</pre> <pre> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.7.0_79]</pre> <pre> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]</pre> <pre> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]</pre> <pre> at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]</pre> <pre> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view6.runAction(Unknown Source) [common.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:264)</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243)</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120)</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110)</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129)</pre> <pre> at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100)</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]</pre> <pre> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]</pre> <pre> at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110)</pre> <pre> at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)</pre> <pre> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)</pre> <pre> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)</pre> <pre> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)</pre> <pre> at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)</pre> <pre> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)</pre> <pre> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)</pre> <pre> at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)</pre> <pre> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)</pre> <pre> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)</pre> <pre> at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)</pre> <pre> at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)</pre> <pre> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)</pre> <pre> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]</pre> <pre> 2015-11-06 19:17:13,708 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] CanDoAction of action AddVds failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server ophelia.doubledog.org,VDS_CANNOT_CONNECT_TO_SERVER</pre> <pre>2015-11-06 19:17:13,728 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-9) Operation Failed: [Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.]</pre> <pre> ==> /var/log/ovirt-engine/server.log <==</pre> <pre>2015-11-06 19:17:13,729 WARN [org.jboss.resteasy.core.SynchronousDispatcher] (ajp--127.0.0.1-8702-9) failed to execute: org.ovirt.engine.api.restapi.resource.BaseBackendResource$WebFaultException</pre> <pre> at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:255) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:230) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:245) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) [restapi-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) [restapi-jaxrs.jar:]</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]</pre> <pre> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]</pre> <pre> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]</pre> <pre> at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]</pre> <pre> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:]</pre> <pre> at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]</pre> <pre> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]</pre> <pre> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]</pre> <br> <br> I don't understand the ssh failure. From enceladus-f20, I can ssh as root into ophelia-f22 and vice versa. I really don't care if it's all Fedora or CentOS or a mix, my primary objective is to just be on 3.6... somehow.<br> <pre class="moz-signature" cols="72">-- John Florian </pre> </body> </html> --------------070507020507000903020509--
On Sat, Nov 7, 2015 at 2:38 AM, John Florian <jflorian@doubledog.org> wrote:
On 10/29/2015 05:49 AM, Roy Golan wrote:
On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <rgolan@redhat.com> wrote:
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate.
#install the appliance yum install ovirt-engine-appliance
#and then run the setup ovirt-hosted-engine-setup
choose "disk" in this stage
Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk
it will suggest the downloaded appliance automatically
See this wiki for reference http://www.ovirt.org/Features/HEApplianceFlow#Testing
I'm afraid I'm lost here. Here's a map of my setup:
oVirt 3.5.5 hosted engine is named enceladus-f20 (on Fedora 20) I have one oVirt 3.5.5 Host named oberon-f20 (also on Fedora 20) I previously had one other oVirt 3.5.5 Host named ophelia-f20
I took opehlia down, installed CentOS 7 on it and attempted a "hosted-engine --deploy". I can't remember if that was 3.5.5 or 3.6, but I could not add it to my cluster. From what I could gather enceladus-f20 provided an emulation type of pc1.0 while ophelia-c7 didn't seem to have anything that matched, the closest being just "pc". That looked hopelessly complicated to resolve so I thought I'd try again, but this time putting F22 on the ophelia and doing a 3.6 HE deploy saying yes to the redeploy prompt. This time I was met with:
Checking for oVirt-Engine status at enceladus-f20.doubledog.org...
[ INFO ] Engine replied: DB Up!Welcome to Health Status!
[ ERROR ] Cannot automatically add the host to cluster Default: Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.
Please check Engine VM configuration.
On enceladus-f20, I see:
==> /var/log/ovirt-engine/engine.log <==
2015-11-06 19:17:12,085 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
2015-11-06 19:17:12,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
==> /var/log/ovirt-engine/server.log <==
2015-11-06 19:17:13,654 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-1) Client session created
2015-11-06 19:17:13,663 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-2) Server version string: SSH-2.0-OpenSSH_6.9
2015-11-06 19:17:13,667 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1)
at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1098)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:357)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295)
at org.apache.sshd.client.session.ClientSessionImpl.handleMessage(ClientSessionImpl.java:266)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:720)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:188)
at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:174)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.7.0_79]
at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.7.0_79]
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
==> /var/log/ovirt-engine/engine.log <==
2015-11-06 19:17:13,672 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] Failed to establish session with host hosted_engine_2: java.io.IOException: SSH session closed during connection 'root@ophelia.doubledog.org'
at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:309) [uutils.jar:]
at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]
at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:465) [bll.jar:]
at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:364) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:768) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:347) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:435) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:416) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:374) [bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view6.runAction(Unknown Source) [common.jar:]
at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:264)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129)
at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110)
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
2015-11-06 19:17:13,708 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] CanDoAction of action AddVds failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server ophelia.doubledog.org,VDS_CANNOT_CONNECT_TO_SERVER
2015-11-06 19:17:13,728 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-9) Operation Failed: [Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.]
==> /var/log/ovirt-engine/server.log <==
2015-11-06 19:17:13,729 WARN [org.jboss.resteasy.core.SynchronousDispatcher] (ajp--127.0.0.1-8702-9) failed to execute: org.ovirt.engine.api.restapi.resource.BaseBackendResource$WebFaultException
at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:255) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:230) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:245) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) [restapi-jaxrs.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:]
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
I don't understand the ssh failure. From enceladus-f20, I can ssh as root into ophelia-f22 and vice versa. I really don't care if it's all Fedora or CentOS or a mix, my primary objective is to just be on 3.6... somehow.
The failure is in the engine vm - it fails to ssh back to the host it's running on. Most probably due to name resolution and not due to network access. See also: https://bugzilla.redhat.com/show_bug.cgi?id=1251968 Best,
-- John Florian
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
On 11/08/2015 04:16 AM, Yedidyah Bar David wrote:
On Sat, Nov 7, 2015 at 2:38 AM, John Florian <jflorian@doubledog.org> wrote:
On 10/29/2015 05:49 AM, Roy Golan wrote:
On Thu, Oct 29, 2015 at 11:39 AM, Roy Golan <rgolan@redhat.com> wrote:
On Wed, Oct 28, 2015 at 10:23 PM, John Florian <jflorian@doubledog.org> wrote:
Can somebody please point me to documentation or describe how I should proceed with this task? I see lots of pages for moving from a physical engine to a VM and vice-versa but am having no luck finding how to go about building a new HE to obsolete my original.
using ovirt-hosted-engine-setup you can choose a setup without using the appliance. So you can scratch install your VM
BTW the ovirt-engine-appliacnce we build [1] is Centos based. Seems like perfect candidate.
#install the appliance yum install ovirt-engine-appliance
#and then run the setup ovirt-hosted-engine-setup
choose "disk" in this stage
Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: disk
it will suggest the downloaded appliance automatically
See this wiki for reference http://www.ovirt.org/Features/HEApplianceFlow#Testing
I'm afraid I'm lost here. Here's a map of my setup:
oVirt 3.5.5 hosted engine is named enceladus-f20 (on Fedora 20) I have one oVirt 3.5.5 Host named oberon-f20 (also on Fedora 20) I previously had one other oVirt 3.5.5 Host named ophelia-f20
I took opehlia down, installed CentOS 7 on it and attempted a "hosted-engine --deploy". I can't remember if that was 3.5.5 or 3.6, but I could not add it to my cluster. From what I could gather enceladus-f20 provided an emulation type of pc1.0 while ophelia-c7 didn't seem to have anything that matched, the closest being just "pc". That looked hopelessly complicated to resolve so I thought I'd try again, but this time putting F22 on the ophelia and doing a 3.6 HE deploy saying yes to the redeploy prompt. This time I was met with:
Checking for oVirt-Engine status at enceladus-f20.doubledog.org...
[ INFO ] Engine replied: DB Up!Welcome to Health Status!
[ ERROR ] Cannot automatically add the host to cluster Default: Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.
Please check Engine VM configuration.
On enceladus-f20, I see:
==> /var/log/ovirt-engine/engine.log <==
2015-11-06 19:17:12,085 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false.
2015-11-06 19:17:12,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
==> /var/log/ovirt-engine/server.log <==
2015-11-06 19:17:13,654 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-1) Client session created
2015-11-06 19:17:13,663 INFO [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-2) Server version string: SSH-2.0-OpenSSH_6.9
2015-11-06 19:17:13,667 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-18-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1)
at org.apache.sshd.common.session.AbstractSession.negotiate(AbstractSession.java:1098)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:357)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295)
at org.apache.sshd.client.session.ClientSessionImpl.handleMessage(ClientSessionImpl.java:266)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:720)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:188)
at org.apache.sshd.common.io.nio2.Nio2Session$1.completed(Nio2Session.java:174)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.7.0_79]
at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.7.0_79]
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
==> /var/log/ovirt-engine/engine.log <==
2015-11-06 19:17:13,672 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] Failed to establish session with host hosted_engine_2: java.io.IOException: SSH session closed during connection 'root@ophelia.doubledog.org'
at org.ovirt.engine.core.uutils.ssh.SSHClient.connect(SSHClient.java:309) [uutils.jar:]
at org.ovirt.engine.core.bll.utils.EngineSSHClient.connect(EngineSSHClient.java:59) [bll.jar:]
at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:465) [bll.jar:]
at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:364) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:768) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:347) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:435) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:416) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:374) [bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:114) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:125) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:135) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:]
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:82) [jboss-as-weld-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view6.runAction(Unknown Source) [common.jar:]
at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:264)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:243)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110)
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129)
at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110)
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
2015-11-06 19:17:13,708 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp--127.0.0.1-8702-9) [65094082] CanDoAction of action AddVds failed for user admin@internal. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server ophelia.doubledog.org,VDS_CANNOT_CONNECT_TO_SERVER
2015-11-06 19:17:13,728 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp--127.0.0.1-8702-9) Operation Failed: [Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.]
==> /var/log/ovirt-engine/server.log <==
2015-11-06 19:17:13,729 WARN [org.jboss.resteasy.core.SynchronousDispatcher] (ajp--127.0.0.1-8702-9) failed to execute: org.ovirt.engine.api.restapi.resource.BaseBackendResource$WebFaultException
at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:255) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.BaseBackendResource.handleError(BaseBackendResource.java:230) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:245) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:120) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:110) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:129) [restapi-jaxrs.jar:]
at org.ovirt.engine.api.restapi.resource.BackendHostsResource.add(BackendHostsResource.java:100) [restapi-jaxrs.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_79]
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:74) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:113) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:90) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:73) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:32) [aaa.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:110) [interface-common-jaxrs.jar:]
at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:101) [interface-common-jaxrs.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
I don't understand the ssh failure. From enceladus-f20, I can ssh as root into ophelia-f22 and vice versa. I really don't care if it's all Fedora or CentOS or a mix, my primary objective is to just be on 3.6... somehow. The failure is in the engine vm - it fails to ssh back to the host it's running on. Most probably due to name resolution and not due to network access.
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot. -- John Florian
On 11/09/2015 06:25 PM, John Florian wrote:
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot.
I got past the above problem by adding the following line to the /etc/ssh/sshd_config of the new F22 host: KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 This represents the defaults for F22 -- at least according to sshd_config(5) -- but with the addition of diffie-hellman-group1-sha1 that the Java-based ssh client seems insistent on using. However, all is not rosy. The deploy script ground to a halt with: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... The host hosted_engine_2 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]? So I did as suggested and tried to activate the host from the webadmin UI. That didn't work either. The status message at the bottom of the browser page shows: Host hosted_engine_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.16, 4.11, 4.15, 4.12, 4.10]. The attempt to activate the host via the web UI also caused the following to be logged on the engine: 2015-11-09 19:12:39,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [4bf460e8] Lock Acquired to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:39,838 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-11-09 19:12:39,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Before acquiring lock in order to prevent monitoring for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,856 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Lock acquired, from now a monitoring of host will be skipped for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d206899 2015-11-09 19:12:39,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] FINISH, SetVdsStatusVDSCommand, log id: 1d206899 2015-11-09 19:12:39,888 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Activate finished. Lock released. Monitoring can run now for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 4bf460e8, Job ID: 08a2b1ad-1c1c-425c-b657-7739df72b764, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 was activated by admin@internal. 2015-11-09 19:12:39,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:40,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 7be846bb 2015-11-09 19:12:40,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] FINISH, GetHardwareInfoVDSCommand, log id: 7be846bb 2015-11-09 19:12:40,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 56456697 2015-11-09 19:12:40,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] FINISH, SetVdsStatusVDSCommand, log id: 56456697 2015-11-09 19:12:40,332 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' 2015-11-09 19:12:40,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 3cf70a78 2015-11-09 19:12:40,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] FINISH, GetHardwareInfoVDSCommand, log id: 3cf70a78 2015-11-09 19:12:40,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 17fa8ac0 2015-11-09 19:12:40,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] FINISH, SetVdsStatusVDSCommand, log id: 17fa8ac0 2015-11-09 19:12:40,413 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host hosted_engine_2 was set to NonOperational. 2015-11-09 19:12:40,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] ResourceManager::refreshVdsRunTimeInfo: Error: DataIntegrityViolationException: CallableStatementCallback; SQL [{call updatevdsdynamic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: value too long for type character varying(255) Where: SQL statement "UPDATE vds_dynamic SET cpu_cores = v_cpu_cores,cpu_threads = v_cpu_threads, cpu_model = v_cpu_model,cpu_speed_mh = v_cpu_speed_mh, if_total_speed = v_if_total_speed,kvm_enabled = v_kvm_enabled, mem_commited = v_mem_commited,physical_mem_mb = v_physical_mem_mb, status = v_status,vm_active = v_vm_active,vm_count = v_vm_count, vm_migrating = v_vm_migrating,reserved_mem = v_reserved_mem, guest_overhead = v_guest_overhead,rpm_version = v_rpm_version, software_version = v_software_version, version_name = v_version_name,build_name = v_build_name,previous_status = v_previous_status, cpu_flags = v_cpu_flags, vms_cores_count = v_vms_cores_count,pending_vcpus_count = v_pending_vcpus_count, pending_vmem_size = v_pending_vmem_size, cpu_sockets = v_cpu_sockets,net_config_dirty = v_net_config_dirty, supported_cluster_levels = v_supported_cluster_levels, supported_engines = v_supported_engines,host_os = v_host_os, kvm_version = v_kvm_version,libvirt_version = v_libvirt_version,spice_version = v_spice_version, gluster_version = v_gluster_version, kernel_version = v_kernel_version,iscsi_initiator_name = v_iscsi_initiator_name, transparent_hugepages_state = v_transparent_hugepages_state, hooks = v_hooks, _update_date = LOCALTIMESTAMP,non_operational_reason = v_non_operational_reason, hw_manufacturer = v_hw_manufacturer, hw_product_name = v_hw_product_name, hw_version = v_hw_version, hw_serial_number = v_hw_serial_number, hw_uuid = v_hw_uuid, hw_family = v_hw_family, hbas = v_hbas, supported_emulated_machines = v_supported_emulated_machines, kdump_status = v_kdump_status, selinux_enforce_mode = v_selinux_enforce_mode, auto_numa_balancing = v_auto_numa_balancing, is_numa_supported = v_is_numa_supported, supported_rng_sources = v_supported_rng_sources, is_live_snapshot_supported = v_is_live_snapshot_supported, is_live_merge_supported = v_is_live_merge_supported, online_cpus = v_online_cpus WHERE vds_id = v_vds_id" The SQL error then seems to repeat several more times. -- John Florian
On Tue, Nov 10, 2015 at 2:16 AM, John Florian <jflorian@doubledog.org> wrote:
On 11/09/2015 06:25 PM, John Florian wrote:
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot.
I got past the above problem by adding the following line to the /etc/ssh/sshd_config of the new F22 host:
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
This represents the defaults for F22 -- at least according to sshd_config(5) -- but with the addition of diffie-hellman-group1-sha1 that the Java-based ssh client seems insistent on using.
Adding Alon for this. Not sure if we can configure the java ssh client and how.
However, all is not rosy. The deploy script ground to a halt with: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... The host hosted_engine_2 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]?
So I did as suggested and tried to activate the host from the webadmin UI. That didn't work either. The status message at the bottom of the browser page shows:
Host hosted_engine_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.16, 4.11, 4.15, 4.12, 4.10].
The attempt to activate the host via the web UI also caused the following to be logged on the engine:
2015-11-09 19:12:39,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [4bf460e8] Lock Acquired to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:39,838 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-11-09 19:12:39,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Before acquiring lock in order to prevent monitoring for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,856 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Lock acquired, from now a monitoring of host will be skipped for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d206899 2015-11-09 19:12:39,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] FINISH, SetVdsStatusVDSCommand, log id: 1d206899 2015-11-09 19:12:39,888 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Activate finished. Lock released. Monitoring can run now for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 4bf460e8, Job ID: 08a2b1ad-1c1c-425c-b657-7739df72b764, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 was activated by admin@internal. 2015-11-09 19:12:39,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:40,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 7be846bb 2015-11-09 19:12:40,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] FINISH, GetHardwareInfoVDSCommand, log id: 7be846bb 2015-11-09 19:12:40,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 56456697 2015-11-09 19:12:40,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] FINISH, SetVdsStatusVDSCommand, log id: 56456697 2015-11-09 19:12:40,332 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt'
Didn't you see this one anywhere? There was some problem creating the bridge, or something else making the engine think so. If it seems ok to you, perhaps check also vdsm logs.
2015-11-09 19:12:40,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 3cf70a78 2015-11-09 19:12:40,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] FINISH, GetHardwareInfoVDSCommand, log id: 3cf70a78 2015-11-09 19:12:40,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 17fa8ac0 2015-11-09 19:12:40,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] FINISH, SetVdsStatusVDSCommand, log id: 17fa8ac0 2015-11-09 19:12:40,413 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host hosted_engine_2 was set to NonOperational. 2015-11-09 19:12:40,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] ResourceManager::refreshVdsRunTimeInfo: Error: DataIntegrityViolationException: CallableStatementCallback; SQL [{call updatevdsdynamic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: value too long for type character varying(255)
That's ugly too, not sure why it happened and how reproducible, but I think you should first take care of the ovirtmgmt network. Adding Eli anyway.
Where: SQL statement "UPDATE vds_dynamic SET cpu_cores = v_cpu_cores,cpu_threads = v_cpu_threads, cpu_model = v_cpu_model,cpu_speed_mh = v_cpu_speed_mh, if_total_speed = v_if_total_speed,kvm_enabled = v_kvm_enabled, mem_commited = v_mem_commited,physical_mem_mb = v_physical_mem_mb, status = v_status,vm_active = v_vm_active,vm_count = v_vm_count, vm_migrating = v_vm_migrating,reserved_mem = v_reserved_mem, guest_overhead = v_guest_overhead,rpm_version = v_rpm_version, software_version = v_software_version, version_name = v_version_name,build_name = v_build_name,previous_status = v_previous_status, cpu_flags = v_cpu_flags, vms_cores_count = v_vms_cores_count,pending_vcpus_count = v_pending_vcpus_count, pending_vmem_size = v_pending_vmem_size, cpu_sockets = v_cpu_sockets,net_config_dirty = v_net_config_dirty, supported_cluster_levels = v_supported_cluster_levels, supported_engines = v_supported_engines,host_os = v_host_os, kvm_version = v_kvm_version,libvirt_version = v_libvirt_version,spice_version = v_spice_version, gluster_version = v_gluster_version, kernel_version = v_kernel_version,iscsi_initiator_name = v_iscsi_initiator_name, transparent_hugepages_state = v_transparent_hugepages_state, hooks = v_hooks, _update_date = LOCALTIMESTAMP,non_operational_reason = v_non_operational_reason, hw_manufacturer = v_hw_manufacturer, hw_product_name = v_hw_product_name, hw_version = v_hw_version, hw_serial_number = v_hw_serial_number, hw_uuid = v_hw_uuid, hw_family = v_hw_family, hbas = v_hbas, supported_emulated_machines = v_supported_emulated_machines, kdump_status = v_kdump_status, selinux_enforce_mode = v_selinux_enforce_mode, auto_numa_balancing = v_auto_numa_balancing, is_numa_supported = v_is_numa_supported, supported_rng_sources = v_supported_rng_sources, is_live_snapshot_supported = v_is_live_snapshot_supported, is_live_merge_supported = v_is_live_merge_supported, online_cpus = v_online_cpus WHERE vds_id = v_vds_id"
The SQL error then seems to repeat several more times.
-- John Florian
Thanks for the report. Best, -- Didi
----- Original Message -----
From: "Yedidyah Bar David" <didi@redhat.com> To: "John Florian" <jflorian@doubledog.org>, "Alon Bar-Lev" <alonbl@redhat.com>, "Roy Golan" <rgolan@redhat.com>, "Eli Mesika" <emesika@redhat.com> Cc: "users" <users@ovirt.org> Sent: Tuesday, November 10, 2015 10:39:27 AM Subject: Re: [ovirt-users] Moving a Hosted Engine from Fedora 20 to CentOS 7
On Tue, Nov 10, 2015 at 2:16 AM, John Florian <jflorian@doubledog.org> wrote:
On 11/09/2015 06:25 PM, John Florian wrote:
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot.
I got past the above problem by adding the following line to the /etc/ssh/sshd_config of the new F22 host:
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
This represents the defaults for F22 -- at least according to sshd_config(5) -- but with the addition of diffie-hellman-group1-sha1 that the Java-based ssh client seems insistent on using.
Adding Alon for this. Not sure if we can configure the java ssh client and how.
I think that newer than apache-sshd-0.14 altered its behavior, can you please try to downgrade to apache-sshd-0.13 and see if it helps, if it does we will enforce it. Only in apache-sshd-1.1.0 (unreleased) we will be able to migrate properly (I hope).
However, all is not rosy. The deploy script ground to a halt with: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... The host hosted_engine_2 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]?
So I did as suggested and tried to activate the host from the webadmin UI. That didn't work either. The status message at the bottom of the browser page shows:
Host hosted_engine_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.16, 4.11, 4.15, 4.12, 4.10].
The attempt to activate the host via the web UI also caused the following to be logged on the engine:
2015-11-09 19:12:39,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [4bf460e8] Lock Acquired to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:39,838 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-11-09 19:12:39,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Before acquiring lock in order to prevent monitoring for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,856 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Lock acquired, from now a monitoring of host will be skipped for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d206899 2015-11-09 19:12:39,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] FINISH, SetVdsStatusVDSCommand, log id: 1d206899 2015-11-09 19:12:39,888 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Activate finished. Lock released. Monitoring can run now for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 4bf460e8, Job ID: 08a2b1ad-1c1c-425c-b657-7739df72b764, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 was activated by admin@internal. 2015-11-09 19:12:39,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:40,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 7be846bb 2015-11-09 19:12:40,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] FINISH, GetHardwareInfoVDSCommand, log id: 7be846bb 2015-11-09 19:12:40,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 56456697 2015-11-09 19:12:40,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] FINISH, SetVdsStatusVDSCommand, log id: 56456697 2015-11-09 19:12:40,332 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt'
Didn't you see this one anywhere? There was some problem creating the bridge, or something else making the engine think so.
If it seems ok to you, perhaps check also vdsm logs.
2015-11-09 19:12:40,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 3cf70a78 2015-11-09 19:12:40,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] FINISH, GetHardwareInfoVDSCommand, log id: 3cf70a78 2015-11-09 19:12:40,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 17fa8ac0 2015-11-09 19:12:40,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] FINISH, SetVdsStatusVDSCommand, log id: 17fa8ac0 2015-11-09 19:12:40,413 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host hosted_engine_2 was set to NonOperational. 2015-11-09 19:12:40,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] ResourceManager::refreshVdsRunTimeInfo: Error: DataIntegrityViolationException: CallableStatementCallback; SQL [{call updatevdsdynamic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: value too long for type character varying(255)
That's ugly too, not sure why it happened and how reproducible, but I think you should first take care of the ovirtmgmt network.
Adding Eli anyway.
Where: SQL statement "UPDATE vds_dynamic SET cpu_cores = v_cpu_cores,cpu_threads = v_cpu_threads, cpu_model = v_cpu_model,cpu_speed_mh = v_cpu_speed_mh, if_total_speed = v_if_total_speed,kvm_enabled = v_kvm_enabled, mem_commited = v_mem_commited,physical_mem_mb = v_physical_mem_mb, status = v_status,vm_active = v_vm_active,vm_count = v_vm_count, vm_migrating = v_vm_migrating,reserved_mem = v_reserved_mem, guest_overhead = v_guest_overhead,rpm_version = v_rpm_version, software_version = v_software_version, version_name = v_version_name,build_name = v_build_name,previous_status = v_previous_status, cpu_flags = v_cpu_flags, vms_cores_count = v_vms_cores_count,pending_vcpus_count = v_pending_vcpus_count, pending_vmem_size = v_pending_vmem_size, cpu_sockets = v_cpu_sockets,net_config_dirty = v_net_config_dirty, supported_cluster_levels = v_supported_cluster_levels, supported_engines = v_supported_engines,host_os = v_host_os, kvm_version = v_kvm_version,libvirt_version = v_libvirt_version,spice_version = v_spice_version, gluster_version = v_gluster_version, kernel_version = v_kernel_version,iscsi_initiator_name = v_iscsi_initiator_name, transparent_hugepages_state = v_transparent_hugepages_state, hooks = v_hooks, _update_date = LOCALTIMESTAMP,non_operational_reason = v_non_operational_reason, hw_manufacturer = v_hw_manufacturer, hw_product_name = v_hw_product_name, hw_version = v_hw_version, hw_serial_number = v_hw_serial_number, hw_uuid = v_hw_uuid, hw_family = v_hw_family, hbas = v_hbas, supported_emulated_machines = v_supported_emulated_machines, kdump_status = v_kdump_status, selinux_enforce_mode = v_selinux_enforce_mode, auto_numa_balancing = v_auto_numa_balancing, is_numa_supported = v_is_numa_supported, supported_rng_sources = v_supported_rng_sources, is_live_snapshot_supported = v_is_live_snapshot_supported, is_live_merge_supported = v_is_live_merge_supported, online_cpus = v_online_cpus WHERE vds_id = v_vds_id"
The SQL error then seems to repeat several more times.
-- John Florian
Thanks for the report.
Best, -- Didi
On 11/10/2015 03:45 AM, Alon Bar-Lev wrote:
----- Original Message -----
From: "Yedidyah Bar David" <didi@redhat.com> To: "John Florian" <jflorian@doubledog.org>, "Alon Bar-Lev" <alonbl@redhat.com>, "Roy Golan" <rgolan@redhat.com>, "Eli Mesika" <emesika@redhat.com> Cc: "users" <users@ovirt.org> Sent: Tuesday, November 10, 2015 10:39:27 AM Subject: Re: [ovirt-users] Moving a Hosted Engine from Fedora 20 to CentOS 7
On Tue, Nov 10, 2015 at 2:16 AM, John Florian <jflorian@doubledog.org> wrote:
On 11/09/2015 06:25 PM, John Florian wrote:
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot. I got past the above problem by adding the following line to the /etc/ssh/sshd_config of the new F22 host:
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
This represents the defaults for F22 -- at least according to sshd_config(5) -- but with the addition of diffie-hellman-group1-sha1 that the Java-based ssh client seems insistent on using.
Adding Alon for this. Not sure if we can configure the java ssh client and how.
I think that newer than apache-sshd-0.14 altered its behavior, can you please try to downgrade to apache-sshd-0.13 and see if it helps, if it does we will enforce it. Only in apache-sshd-1.1.0 (unreleased) we will be able to migrate properly (I hope).
This is my fault. I see now that this was mentioned in the 3.6.0 release notes as a Fedora 22 specific issue (see also https://bugzilla.redhat.com/show_bug.cgi?id=1225531). The work around is just exactly as I mentioned above, though I figured it out, then read the release notes; doh! I don't even see an apache-sshd package installed on my engine (or hosts).
However, all is not rosy. The deploy script ground to a halt with: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... The host hosted_engine_2 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]?
So I did as suggested and tried to activate the host from the webadmin UI. That didn't work either. The status message at the bottom of the browser page shows:
Host hosted_engine_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.16, 4.11, 4.15, 4.12, 4.10].
The attempt to activate the host via the web UI also caused the following to be logged on the engine:
2015-11-09 19:12:39,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [4bf460e8] Lock Acquired to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:39,838 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-11-09 19:12:39,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Before acquiring lock in order to prevent monitoring for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,856 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Lock acquired, from now a monitoring of host will be skipped for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d206899 2015-11-09 19:12:39,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] FINISH, SetVdsStatusVDSCommand, log id: 1d206899 2015-11-09 19:12:39,888 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Activate finished. Lock released. Monitoring can run now for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 4bf460e8, Job ID: 08a2b1ad-1c1c-425c-b657-7739df72b764, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 was activated by admin@internal. 2015-11-09 19:12:39,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:40,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 7be846bb 2015-11-09 19:12:40,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] FINISH, GetHardwareInfoVDSCommand, log id: 7be846bb 2015-11-09 19:12:40,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 56456697 2015-11-09 19:12:40,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] FINISH, SetVdsStatusVDSCommand, log id: 56456697 2015-11-09 19:12:40,332 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' Didn't you see this one anywhere? There was some problem creating the bridge, or something else making the engine think so.
If it seems ok to you, perhaps check also vdsm logs.
2015-11-09 19:12:40,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 3cf70a78 2015-11-09 19:12:40,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] FINISH, GetHardwareInfoVDSCommand, log id: 3cf70a78 2015-11-09 19:12:40,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 17fa8ac0 2015-11-09 19:12:40,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] FINISH, SetVdsStatusVDSCommand, log id: 17fa8ac0 2015-11-09 19:12:40,413 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host hosted_engine_2 was set to NonOperational. 2015-11-09 19:12:40,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] ResourceManager::refreshVdsRunTimeInfo: Error: DataIntegrityViolationException: CallableStatementCallback; SQL [{call updatevdsdynamic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: value too long for type character varying(255) That's ugly too, not sure why it happened and how reproducible, but I think you should first take care of the ovirtmgmt network.
Adding Eli anyway.
Where: SQL statement "UPDATE vds_dynamic SET cpu_cores = v_cpu_cores,cpu_threads = v_cpu_threads, cpu_model = v_cpu_model,cpu_speed_mh = v_cpu_speed_mh, if_total_speed = v_if_total_speed,kvm_enabled = v_kvm_enabled, mem_commited = v_mem_commited,physical_mem_mb = v_physical_mem_mb, status = v_status,vm_active = v_vm_active,vm_count = v_vm_count, vm_migrating = v_vm_migrating,reserved_mem = v_reserved_mem, guest_overhead = v_guest_overhead,rpm_version = v_rpm_version, software_version = v_software_version, version_name = v_version_name,build_name = v_build_name,previous_status = v_previous_status, cpu_flags = v_cpu_flags, vms_cores_count = v_vms_cores_count,pending_vcpus_count = v_pending_vcpus_count, pending_vmem_size = v_pending_vmem_size, cpu_sockets = v_cpu_sockets,net_config_dirty = v_net_config_dirty, supported_cluster_levels = v_supported_cluster_levels, supported_engines = v_supported_engines,host_os = v_host_os, kvm_version = v_kvm_version,libvirt_version = v_libvirt_version,spice_version = v_spice_version, gluster_version = v_gluster_version, kernel_version = v_kernel_version,iscsi_initiator_name = v_iscsi_initiator_name, transparent_hugepages_state = v_transparent_hugepages_state, hooks = v_hooks, _update_date = LOCALTIMESTAMP,non_operational_reason = v_non_operational_reason, hw_manufacturer = v_hw_manufacturer, hw_product_name = v_hw_product_name, hw_version = v_hw_version, hw_serial_number = v_hw_serial_number, hw_uuid = v_hw_uuid, hw_family = v_hw_family, hbas = v_hbas, supported_emulated_machines = v_supported_emulated_machines, kdump_status = v_kdump_status, selinux_enforce_mode = v_selinux_enforce_mode, auto_numa_balancing = v_auto_numa_balancing, is_numa_supported = v_is_numa_supported, supported_rng_sources = v_supported_rng_sources, is_live_snapshot_supported = v_is_live_snapshot_supported, is_live_merge_supported = v_is_live_merge_supported, online_cpus = v_online_cpus WHERE vds_id = v_vds_id"
The SQL error then seems to repeat several more times.
-- John Florian Thanks for the report.
Best, -- Didi
-- John Florian
On 11/10/2015 03:39 AM, Yedidyah Bar David wrote:
On Tue, Nov 10, 2015 at 2:16 AM, John Florian <jflorian@doubledog.org> wrote:
On 11/09/2015 06:25 PM, John Florian wrote:
I don't think it has anything to do with name resolution either. I believe the telltale clue is this bit... 2015-11-09 18:22:31,738 WARN [org.apache.sshd.client.session.ClientSessionImpl] (pool-20-thread-3) Exception caught: java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: diffie-hellman-group1-sha1 / server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1) As mentioned, I can ssh from my engine to the host just fine. It appears that the Java-based ssh client however cannot. I got past the above problem by adding the following line to the /etc/ssh/sshd_config of the new F22 host:
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
This represents the defaults for F22 -- at least according to sshd_config(5) -- but with the addition of diffie-hellman-group1-sha1 that the Java-based ssh client seems insistent on using.
Adding Alon for this. Not sure if we can configure the java ssh client and how.
However, all is not rosy. The deploy script ground to a halt with: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... The host hosted_engine_2 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]?
So I did as suggested and tried to activate the host from the webadmin UI. That didn't work either. The status message at the bottom of the browser page shows:
Host hosted_engine_2 is installed with VDSM version (<UNKNOWN>) and cannot join cluster Default which is compatible with VDSM versions [4.13, 4.14, 4.9, 4.16, 4.11, 4.15, 4.12, 4.10].
The attempt to activate the host via the web UI also caused the following to be logged on the engine:
2015-11-09 19:12:39,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-7) [4bf460e8] Lock Acquired to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:39,838 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2015-11-09 19:12:39,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Before acquiring lock in order to prevent monitoring for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,856 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] Lock acquired, from now a monitoring of host will be skipped for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1d206899 2015-11-09 19:12:39,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-49) [4bf460e8] FINISH, SetVdsStatusVDSCommand, log id: 1d206899 2015-11-09 19:12:39,888 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Activate finished. Lock released. Monitoring can run now for host hosted_engine_2 from data-center Default 2015-11-09 19:12:39,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-49) Correlation ID: 4bf460e8, Job ID: 08a2b1ad-1c1c-425c-b657-7739df72b764, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 was activated by admin@internal. 2015-11-09 19:12:39,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df value: VDS , sharedLocks= ] 2015-11-09 19:12:40,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 7be846bb 2015-11-09 19:12:40,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [79b24fed] FINISH, GetHardwareInfoVDSCommand, log id: 7be846bb 2015-11-09 19:12:40,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 56456697 2015-11-09 19:12:40,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] FINISH, SetVdsStatusVDSCommand, log id: 56456697 2015-11-09 19:12:40,332 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' Didn't you see this one anywhere? There was some problem creating the bridge, or something else making the engine think so.
If it seems ok to you, perhaps check also vdsm logs. I might have missed that; I don't recall for certain as there is a lot of logging going on in various places, it's hard to know where to look sometimes. Then there's all the errors that seem to happen that aren't really problems AFAICT.
Anyway, I just ran through an attempted deploy again and I don't see ovirtmgmt mentioned anywhere in /var/log/vdsm/vdsm.log, nor anything else in that file looking majorly bad WRT networking. I do see a bunch of errors related to storage though. Any hints what I should be search for? Would it be best just to post the file somewhere -- it's a rather portly 12MB ATM. And just because I'm feeling stupid right now.... I don't need to do anything special WRT configuring networking prior to running hosted-engine --deploy do I? I mean its got working IP, DNS, etc. but IIRC, the installation handles all the setup of ovirtmgmt, right?
2015-11-09 19:12:40,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] START, GetHardwareInfoVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, vds=Host[hosted_engine_2,fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df]), log id: 3cf70a78 2015-11-09 19:12:40,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [a4459ff] FINISH, GetHardwareInfoVDSCommand, log id: 3cf70a78 2015-11-09 19:12:40,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] START, SetVdsStatusVDSCommand(HostName = hosted_engine_2, HostId = fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df, status=NonOperational, nonOperationalReason=NETWORK_UNREACHABLE, stopSpmFailureLogged=false), log id: 17fa8ac0 2015-11-09 19:12:40,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] FINISH, SetVdsStatusVDSCommand, log id: 17fa8ac0 2015-11-09 19:12:40,413 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [3031cc4e] Host hosted_engine_2 is set to Non-Operational, it is missing the following networks: ovirtmgmt 2015-11-09 19:12:40,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fab55ebe-cc0f-4f95-87aa-fc3a5e08a5df Type: VDS 2015-11-09 19:12:40,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Status of host hosted_engine_2 was set to NonOperational. 2015-11-09 19:12:40,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [5e83f7aa] ResourceManager::refreshVdsRunTimeInfo: Error: DataIntegrityViolationException: CallableStatementCallback; SQL [{call updatevdsdynamic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: value too long for type character varying(255) That's ugly too, not sure why it happened and how reproducible, but I think you should first take care of the ovirtmgmt network.
Adding Eli anyway.
Where: SQL statement "UPDATE vds_dynamic SET cpu_cores = v_cpu_cores,cpu_threads = v_cpu_threads, cpu_model = v_cpu_model,cpu_speed_mh = v_cpu_speed_mh, if_total_speed = v_if_total_speed,kvm_enabled = v_kvm_enabled, mem_commited = v_mem_commited,physical_mem_mb = v_physical_mem_mb, status = v_status,vm_active = v_vm_active,vm_count = v_vm_count, vm_migrating = v_vm_migrating,reserved_mem = v_reserved_mem, guest_overhead = v_guest_overhead,rpm_version = v_rpm_version, software_version = v_software_version, version_name = v_version_name,build_name = v_build_name,previous_status = v_previous_status, cpu_flags = v_cpu_flags, vms_cores_count = v_vms_cores_count,pending_vcpus_count = v_pending_vcpus_count, pending_vmem_size = v_pending_vmem_size, cpu_sockets = v_cpu_sockets,net_config_dirty = v_net_config_dirty, supported_cluster_levels = v_supported_cluster_levels, supported_engines = v_supported_engines,host_os = v_host_os, kvm_version = v_kvm_version,libvirt_version = v_libvirt_version,spice_version = v_spice_version, gluster_version = v_gluster_version, kernel_version = v_kernel_version,iscsi_initiator_name = v_iscsi_initiator_name, transparent_hugepages_state = v_transparent_hugepages_state, hooks = v_hooks, _update_date = LOCALTIMESTAMP,non_operational_reason = v_non_operational_reason, hw_manufacturer = v_hw_manufacturer, hw_product_name = v_hw_product_name, hw_version = v_hw_version, hw_serial_number = v_hw_serial_number, hw_uuid = v_hw_uuid, hw_family = v_hw_family, hbas = v_hbas, supported_emulated_machines = v_supported_emulated_machines, kdump_status = v_kdump_status, selinux_enforce_mode = v_selinux_enforce_mode, auto_numa_balancing = v_auto_numa_balancing, is_numa_supported = v_is_numa_supported, supported_rng_sources = v_supported_rng_sources, is_live_snapshot_supported = v_is_live_snapshot_supported, is_live_merge_supported = v_is_live_merge_supported, online_cpus = v_online_cpus WHERE vds_id = v_vds_id"
The SQL error then seems to repeat several more times.
-- John Florian Thanks for the report.
Best,
-- John Florian
On Thu, Nov 12, 2015 at 2:37 AM, John Florian <jflorian@doubledog.org> wrote: [snip]
2015-11-09 19:12:40,335 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [5569d8a6] Correlation ID: 5569d8a6, Job ID: 98718a7b-4f64-4a3d-bb72-e6d46100edc5, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_2 does not comply with the cluster Default networks, the following networks are missing on host: 'ovirtmgmt' Didn't you see this one anywhere? There was some problem creating the bridge, or something else making the engine think so.
If it seems ok to you, perhaps check also vdsm logs. I might have missed that; I don't recall for certain as there is a lot of logging going on in various places, it's hard to know where to look sometimes. Then there's all the errors that seem to happen that aren't really problems AFAICT.
Anyway, I just ran through an attempted deploy again and I don't see ovirtmgmt mentioned anywhere in /var/log/vdsm/vdsm.log, nor anything else in that file looking majorly bad WRT networking.
But you still have problems? Do you have on the host an interface 'ovirtmgmt'? What do you see in the web admin? mark the host -> Network Interfaces -> Setup host networks.
I do see a bunch of errors related to storage though.
If the storage relies on the network (likely), perhaps first fix network issues, then storage.
Any hints what I should be search for?
If you do not find there 'ovirtmgmt', not sure. You can check also /var/log/ovirt-hosted-engine-setup on the host, as well as normal system log (/var/log/syslog or messages or whatever).
Would it be best just to post the file somewhere -- it's a rather portly 12MB ATM.
Perhaps. Some file sharing site or something.
And just because I'm feeling stupid right now.... I don't need to do anything special WRT configuring networking prior to running hosted-engine --deploy do I? I mean its got working IP, DNS, etc. but IIRC, the installation handles all the setup of ovirtmgmt, right?
If you start from a clean machine, it should do all. If you start from a machine which already has an interface 'ovirtmgmt' but has problems, iiuc the script will not fix them. Best, -- Didi
This is a multi-part message in MIME format. --------------030401080100050500050006 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit I'm getting better results now but still no success. My attempts at rebuilding my host "ophelia" had a number of issues related to networking which was no surprise given that ovirtmgmt wasn't present. I believe this all stemmed from the initial conditions of the host and especially the ifcfg files for the two NICs. I had started with a clean install and did minimal changes but it was seeing too many weird issues to try and sort out at once. Plus once vdsm starts doing its thing managing these it's hard to say from where exactly I was starting. So..... time for a fresh slate. Here are those results: First off, the mail Subject is wholly wrong anymore since I ran into really difficult sounding problems on that path. There's no CentOS involved right now, just Fedora. I don't know which is worse, changing the subject or leaving it wrong or prolonging a mail thread. Anyway, my current situation is: GOOD: oVirt 3.5.4 hosted engine on enceladus-f20 atop F20 GOOD: oVirt 3.5.4 host on oberon (also F20) PROBLEMS: attempting oVirt 3.6 on ophelia atop F22 The battle plan for oVirt 3.6 on ophelia: * installed F22 using minimal install option * initial networking setup immediately after installing F22: [root@ophelia ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:1f:d0:dc:27:af brd ff:ff:ff:ff:ff:ff 3: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 68:05:ca:2f:f4:7c brd ff:ff:ff:ff:ff:ff inet 172.16.7.7/24 brd 172.16.7.255 scope global enp4s0 valid_lft forever preferred_lft forever inet6 fe80::6a05:caff:fe2f:f47c/64 scope link valid_lft forever preferred_lft forever 4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default link/ether 3a:67:d6:86:11:71 brd ff:ff:ff:ff:ff:ff [root@ophelia ~]# cat /etc/resolv.conf # Generated by NetworkManager search doubledog.org nameserver 172.16.7.6 nameserver 172.16.7.154 * dnf update * edited /etc/ssh/sshd_config to add necessary KexAlgorithms * systemctl restart sshd * setenforce permissive so that vdsm can start o The Release Notes mention this is necessary for gluster. I'm only doing ISCSI but I must use permissive otherwise vdsm will not start later. This lesson already learned so applied here to keep the logs void of any confusion from this aspect. * dnf install http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm * dnf install ovirt-hosted-engine-setup * hosted-engine --deploy I'm still seeing the SQL error (in the engine log) that I mentioned in prior mails, though none of the networking issues I had on earlier attempts. Thus I believe I've made some progress but need advice on what to dig into next for y'all. Rather than pasting parts of the logs here, I've just dropped them in their entirety where they should be easily retrievable here: http://www.doubledog.org/john/ovirt-3.6-install-logs/attempt-10/ If additional logs are needed, please let me know. Thanks for all the help with this!!! -- John Florian --------------030401080100050500050006 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> I'm getting better results now but still no success. My attempts at rebuilding my host "ophelia" had a number of issues related to networking which was no surprise given that ovirtmgmt wasn't present. I believe this all stemmed from the initial conditions of the host and especially the ifcfg files for the two NICs. I had started with a clean install and did minimal changes but it was seeing too many weird issues to try and sort out at once. Plus once vdsm starts doing its thing managing these it's hard to say from where exactly I was starting. So..... time for a fresh slate. Here are those results:<br> <br> First off, the mail Subject is wholly wrong anymore since I ran into really difficult sounding problems on that path. There's no CentOS involved right now, just Fedora. I don't know which is worse, changing the subject or leaving it wrong or prolonging a mail thread. Anyway, my current situation is:<br> <br> GOOD: oVirt 3.5.4 hosted engine on enceladus-f20 atop F20 <br> GOOD: oVirt 3.5.4 host on oberon (also F20)<br> PROBLEMS: attempting oVirt 3.6 on ophelia atop F22<br> <br> The battle plan for oVirt 3.6 on ophelia:<br> <ul> <li>installed F22 using minimal install option</li> <li>initial networking setup immediately after installing F22:</li> </ul> [root@ophelia ~]# ip a<br> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default <br> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br> inet 127.0.0.1/8 scope host lo<br> valid_lft forever preferred_lft forever<br> inet6 ::1/128 scope host <br> valid_lft forever preferred_lft forever<br> 2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000<br> link/ether 00:1f:d0:dc:27:af brd ff:ff:ff:ff:ff:ff<br> 3: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000<br> link/ether 68:05:ca:2f:f4:7c brd ff:ff:ff:ff:ff:ff<br> inet 172.16.7.7/24 brd 172.16.7.255 scope global enp4s0<br> valid_lft forever preferred_lft forever<br> inet6 fe80::6a05:caff:fe2f:f47c/64 scope link <br> valid_lft forever preferred_lft forever<br> 4: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default <br> link/ether 3a:67:d6:86:11:71 brd ff:ff:ff:ff:ff:ff<br> <br> [root@ophelia ~]# cat /etc/resolv.conf <br> # Generated by NetworkManager<br> search doubledog.org<br> nameserver 172.16.7.6<br> nameserver 172.16.7.154<br> <br> <ul> <li>dnf update</li> <li>edited /etc/ssh/sshd_config to add necessary KexAlgorithms</li> <li>systemctl restart sshd</li> <li>setenforce permissive so that vdsm can start</li> <ul> <li>The Release Notes mention this is necessary for gluster. I'm only doing ISCSI but I must use permissive otherwise vdsm will not start later. This lesson already learned so applied here to keep the logs void of any confusion from this aspect.</li> </ul> <li>dnf install <a class="moz-txt-link-freetext" href="http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm">http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm</a></li> <li>dnf install ovirt-hosted-engine-setup</li> <li>hosted-engine --deploy</li> </ul> I'm still seeing the SQL error (in the engine log) that I mentioned in prior mails, though none of the networking issues I had on earlier attempts. Thus I believe I've made some progress but need advice on what to dig into next for y'all.<br> <br> Rather than pasting parts of the logs here, I've just dropped them in their entirety where they should be easily retrievable here:<br> <a class="moz-txt-link-freetext" href="http://www.doubledog.org/john/ovirt-3.6-install-logs/attempt-10/">http://www.doubledog.org/john/ovirt-3.6-install-logs/attempt-10/</a><br> <br> If additional logs are needed, please let me know.<br> <br> Thanks for all the help with this!!!<br> <pre class="moz-signature" cols="72">-- John Florian </pre> </body> </html> --------------030401080100050500050006--
participants (4)
-
Alon Bar-Lev -
John Florian -
Roy Golan -
Yedidyah Bar David