On Sat, 10 Oct 2020, 01:24 Gianluca Cecchi, <gianluca.cecchi@gmail.com> wrote:
On Fri, Oct 9, 2020 at 7:12 PM Martin Perina <mperina@redhat.com> wrote:


Could you please share with us all logs from engine gathered by logcollector? We will try to find out any clue what's wrong in your env ...

Thanks,
Martin


I will try to collect.
In the mean time I've found that SSH could be in some way involved

When I add the host and get the immediate failure and apparently nothing happens at all,  I see these two lines in /var/log/ovirt-engine/server.log

2020-10-09 18:15:09,369+02 WARN  [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[7cb54873]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200/10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported
2020-10-09 18:15:09,699+02 WARN  [org.apache.sshd.client.session.ClientConnectionService] (sshd-SshClient[2cbceeab]-nio2-thread-1) globalRequest(ClientConnectionService[ClientSessionImpl[root@ov200/10.4.192.32:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported

This harmless, AFAIK EdDSA is not supported by default in OpenJDK 11 and engine uses only ssh-rsa and ssh-rsa2 anyway


could it be that the ssh client embedded is not able to connect to the CentOS 8.2 for some reason?

If that's the case we should see an error either in engine.log or ansible-runner-service.log


On host at the moment when I try to add it I see again two sessions opened and immediately closed (tried several times), eg in the timeframe above I have:

Oct  9 18:15:09 ov200 systemd-logind[1237]: New session 41 of user root.
Oct  9 18:15:09 ov200 systemd[1]: Started Session 41 of user root.
Oct  9 18:15:09 ov200 systemd-logind[1237]: Session 41 logged out. Waiting for processes to exit.
Oct  9 18:15:09 ov200 systemd-logind[1237]: Removed session 41.
Oct  9 18:15:09 ov200 systemd-logind[1237]: New session 42 of user root.
Oct  9 18:15:09 ov200 systemd[1]: Started Session 42 of user root.
Oct  9 18:15:09 ov200 systemd-logind[1237]: Session 42 logged out. Waiting for processes to exit.
Oct  9 18:15:09 ov200 systemd-logind[1237]: Removed session 42.

anyway at sshd service level it seems it is ok om the host:

journalctl -u sshd.service has

Oct 09 18:15:09 ov200 sshd[13379]: Accepted password for root from 10.4.192.43 port 46008 ssh2
Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 09 18:15:09 ov200 sshd[13379]: pam_unix(sshd:session): session closed for user root
Oct 09 18:15:09 ov200 sshd[13398]: Accepted password for root from 10.4.192.43 port 46014 ssh2
Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 09 18:15:09 ov200 sshd[13398]: pam_unix(sshd:session): session closed for user root

On the host I have not customized anything ssh related:

[root@ov200 ssh]# ps -ef|grep sshd
root        1274       1  0 Oct08 ?        00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=rsa-sha2-256,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,rsa-sha2-512,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

and in sshd_config

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

That looks good 


Can I replicate the command that the engine would run on host through ssh?

I don't think so there is an easy way to do it 
Let's see what else we can get from the logs... 

Martin 


Gianluca