[ovirt-users] Error during hosted-engine-setup for 3.5.1 on F20 (Cannot add the host to cluster ... SSH has failed)

Mon Mar 9 16:53:04 UTC 2015

----- Original Message -----
> From: "Bob Doolittle" <bob at doolittle.us.com>
> To: "Simone Tiraboschi" <stirabos at redhat.com>
> Cc: "users-ovirt" <users at ovirt.org>
> Sent: Monday, March 9, 2015 12:48:37 PM
> Subject: Re: [ovirt-users] Error during hosted-engine-setup for 3.5.1 on F20 (Cannot add the host to cluster ... SSH
> has failed)
> 
> 
> On 03/09/2015 07:12 AM, Simone Tiraboschi wrote:
> >
> > ----- Original Message -----
> >> From: "Bob Doolittle" <bob at doolittle.us.com>
> >> To: "Simone Tiraboschi" <stirabos at redhat.com>
> >> Sent: Monday, March 9, 2015 12:02:49 PM
> >> Subject: Re: [ovirt-users] Error during hosted-engine-setup for 3.5.1 on
> >> F20 (Cannot add the host to cluster ... SSH
> >> has failed)
> >>
> >> On Mar 9, 2015 5:23 AM, "Simone Tiraboschi" <stirabos at redhat.com> wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Bob Doolittle" <bob at doolittle.us.com>
> >>>> To: "users-ovirt" <users at ovirt.org>
> >>>> Sent: Friday, March 6, 2015 9:21:20 PM
> >>>> Subject: [ovirt-users] Error during hosted-engine-setup for 3.5.1 on
> >> F20 (Cannot add the host to cluster ... SSH has
> >>>> failed)
> >>>>
> >>>> Hi,
> >>>>
> >>>> I'm following the instructions here:
> >> http://www.ovirt.org/Hosted_Engine_Howto
> >>>> My self-hosted install failed near the end:
> >>>>
> >>>> To continue make a selection from the options below:
> >>>>           (1) Continue setup - engine installation is complete
> >>>>           (2) Power off and restart the VM
> >>>>           (3) Abort setup
> >>>>           (4) Destroy VM and abort setup
> >>>>
> >>>>           (1, 2, 3, 4)[1]: 1
> >>>> [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
> >>>>           Enter the name of the cluster to which you want to add the
> >> host
> >>>>           (Default) [Default]:
> >>>> [ ERROR ] Cannot automatically add the host to cluster Default: Cannot
> >> add
> >>>> Host. Connecting to host via SSH has failed, verify that the host is
> >>>> reachable (IP address, routable address etc.) You may refer to the
> >>>> engine.log file for further details.
> >>>> [ ERROR ] Failed to execute stage 'Closing up': Cannot add the host to
> >>>> cluster Default
> >>>> [ INFO  ] Stage: Clean up
> >>>> [ INFO  ] Generating answer file
> >>>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20150306135624.conf'
> >>>> [ INFO  ] Stage: Pre-termination
> >>>> [ INFO  ] Stage: Termination
> >>>>
> >>>> I can ssh into the engine VM both locally and remotely. There is no
> >>>> /root/.ssh directory, however. Did I need to set that up somehow?
> >>> It's the engine that needs to open an SSH connection to the host calling
> >> it by its hostname.
> >>> So please be sure that you can SSH to the host from the engine using its
> >> hostname and not its IP address.
> >>
> >> I'm assuming this should be a password-less login (key-based
> >> authentication?).
> > Yes, it is.
> >
> >> As what user?
> > root
> 
> OK, I see a couple of problems.
> First off, I didn't have my deploying-host hostname in the hosts map for my
> engine.

This is enough by itself to make the deploy procedure failing. If possible we recommend to rely a DNS infrastructure especially if you are deploying more than one host. 

> After adding it to /etc/hosts (both hostname and FQDN), when I try to ssh
> from root at engine to root at host it is prompting me for a password.
> 
> On my engine, ~root/.ssh does not contain any keys.
> On my host, ~root/.ssh has authorized_keys, and in it there is a key with the
> comment "ovirt-engine".
> 
> It's possible that I inadvertently removed ~root/.ssh on engine while I was
> preparing the engine (I started to set up my own no-password logins and then
> thought better and cleaned up, not realizing that some prior setup affecting
> that directory had occurred). That would explain the second issue.

No, it's OK: the private key is contained in /etc/pki/ovirt-engine/keys/engine.p12

> How/when does the key for root at engine get populated to the host's
> ~root/.ssh/authenticated_keys during setup?

It's part of hosted-engine deploy procedure: when the engine setup on the VM it's completed, it gathers the engine SSH public key from http://{enginefqdn}/engine.ssh.key.txt and it stores it under ~root/.ssh/authenticated_keys to make the engine able to add the host without knowing the host root password.
Than hosted-engine setup contacts the engine via REST APIs to trigger the host setup procedure.

If the engine wasn't able to contact the host due to bad hostname resolution as we pointed out, you missed some steps to have a safe deployment.

> -Bob
> 
> >
> >> -Bob
> >>
> >>> Till hosted-engine hosts were simply identified by their IP address but
> >> than we had some bug report on side effects of that.
> >>> So now we generate and sign certs using host hostnames and so the engine
> >> should be able to correctly resolve them.
> >>>> When I log into the Administration portal, the engine VM does not appear
> >>>> under the Virtual machine view (it's empty).
> >>> It's cause the setup didn't complete.
> >>>
> >>>> I've attached what I think are the relevant logs.
> >>>>
> >>>> Also, when my host reboots, the ovirt-ha-broker and ovirt-ha-agent
> >> services
> >>>> do not come up automatically. I have to use systemctl to start them
> >>>> manually.
> >>> It's cause the setup didn't complete.
> >>>
> >>>> This is a fresh Fedora 20 machine installing a fresh copy of Ovirt
> >> 3.5.1.
> >>>> What's the cleanest approach to restore/complete sanity of my setup
> >> please?
> >>> First step is to clarify what went wrong in order to avoid it in the
> >> future.
> >>> Than, if you want a really sanity environment for production use I'd
> >> suggest to redeploy.
> >>> So
> >>>  hosted-engine --vm-poweroff
> >>> empty the storage domain share and deploy again
> >>>
> >>>> Thanks,
> >>>> Bob
> >>>>
> >>>>
> >>>> I've linked 3 files to this email:
> >>>> server.log (12.4 MB) Dropbox https://db.tt/g5p09AaD
> >>>> vdsm.log (3.2 MB) Dropbox https://db.tt/P4572SUm
> >>>> ovirt-hosted-engine-setup-20150306123622-tad1fy.log (413 KB) Dropbox
> >>>> https://db.tt/XAM9ffhi
> >>>> Mozilla Thunderbird makes it easy to share large files over email.
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Users mailing list
> >>>> Users at ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>>
> 
> 