Unfortunately, upgrading to 4.0.1RC didn't solve the problem. Actually, the error changed to 'General SSLEngine problem', but the result was the same, like this:
2016-07-13 09:52:22,010 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /10.X.X.X 2016-07-13 09:52:22,018 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem
It's worth mentioning that we're using our own SSL certificates (not self-signed), and I imported the combined certificate into the /etc/pki/ovirt-engine/.truststore key file. Not sure if related, but just in case.
I had to downgrade to 3.6.7. I'm attaching requested logs, if you need anything else don't hesitate to ask.
FYI I migrated my 3.6 env (engine + 1 host) to 4.0 and the host is up and running fine on datacenter/cluster 4.0 compat level.
FYA there's a BZ about engine certs https://bugzilla.redhat.com/show_bug.cgi?id=1336838
Ah, I forgot to mention that I used my own CA and thus custom certificate for Apache httpd. j.