7:35 p.m.
Hello!
We're started the upgrade one of our oVirt clusters to the recent 4.4
minor version (4.4.8.5-1.el8) and we found a bug in the recent change of
the aaa-ldap plugin.
This bug came along after the IPv4/IPv6 selection introduced in
ovirt-engine-extension-aaa-ldap 1.4.4
We've dig down in the rabbit hole, and it looks like, our DNS solution
is just not compatible with the plugin after this commit:
https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/commit/4c0f2e72d...
I don't know, how the industry actually use CNAME records, but we have a
rule, that we use independet DNS names for services, and machines
itself. This become handy, when you migrate/upgrade services, like LDAP
to another machine, and I think, this should be supported.
Our LDAP server service domain is actually ldap-master.niif.hu and
ldap.niif.hu, but in real, there are two servers, which actually have
their own FQDNs.
The actual DNS structure:
host -t CNAME ldap-master.niif.hu
ldap-master.niif.hu is an alias for elm.niif.hu.
host -t CNAME ldap.niif.hu
ldap.niif.hu is an alias for holly.ldap.einfra.hu.
host elm.niif.hu
elm.niif.hu has address 193.225.14.175
elm.niif.hu has IPv6 address 2001:738:0:701::3
host holly.ldap.einfra.hu
holly.ldap.einfra.hu has address 193.224.92.6
holly.ldap.einfra.hu has IPv6 address 2001:738:0:431::6
The commit above (as far as I understand) only tries to resolve A and
AAAA records in DNS, and drop the connection if it not found. Of course,
the certificate only have ldap-master and ldap.niif.hu in it, so using
holly end elm does not solve the problem (also, if the service will be
migrated, the service domain will be kept, but not the machine's FQDN,
since we cannot afford to shut down one of our LDAP server for a
migration windows.
We've tried to downgrade the package to 1.4.3, which is works fine.
The actual error looks like this (engine.log)
2021-09-07 15:53:09,833+02 WARN
[org.ovirt.engine.extension.aaa.ldap.AuthnExtension] (default task-1) []
[ovirt-engine-extension-aaa-ldap.authn::NIIFLdap-authn] Cannot
initialize LDAP framework, deferring initialization. Error: An error
occurred while attempting to connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error),
errorMessage='An error occurred while attempting to establish a
connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default
task-1) [] Internal Server Error: An error occurred while attempting to
connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error),
errorMessage='An error occurred while attempting to establish a
connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.service.SsoService] (default task-1) [] An
error occurred while attempting to connect to server
ldap-master.niif.hu:636: IOException(LDAPException(resultCode=91
(connect error), errorMessage='An error occurred while attempting to
establish a connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,854+02 ERROR
[org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-1)
[] server_error: An error occurred while attempting to connect to server
ldap-master.niif.hu:636: IOException(LDAPException(resultCode=91
(connect error), errorMessage='An error occurred while attempting to
establish a connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
As we tried to run the setup tool, that is also looks broken (after
that, we've copied the required files to /etc/ovirt-engine/aaa and
/etc/ovirt-engine/extensions.d/ from other, working hosted engine) so
we've tested the plugin itself, and the setup too, but no luck.
I think, this (CNAME in DNS) should be working with the plugin.
Could you please investigate this issue? (we're here to help test the
repaired version/patch, if needed, but not feel the knowledge to create
the patch ourself)
Regards:
Peter
--
*Erdősi Péter
* /Informatikus, IKT Fejlesztési Főosztály /
*Kormányzati Informatikai Fejlesztési Ügynökség
* cím: 1134 Budapest, Váci út 35.
tel: +36 1 450 3080 e-mail: erdosi.peter(a)kifu.gov.hu
<mailto:erdosi.peter@kifu.gov.hu>
KIFÜ - www.kifu.gov.hu <http://kifu.gov.hu/kifu/>
8:33 a.m.
New subject: ovirt-engine-extension-aaa-ldap broken, if using CNAME for LDAP server DNS
On Wed, Sep 8, 2021 at 2:36 AM Erdősi Péter <erdosi.peter(a)kifu.gov.hu>
wrote:
Hello!
We're started the upgrade one of our oVirt clusters to the recent 4.4
minor version (4.4.8.5-1.el8) and we found a bug in the recent change of
the aaa-ldap plugin.
This bug came along after the IPv4/IPv6 selection introduced in
ovirt-engine-extension-aaa-ldap 1.4.4
We've dig down in the rabbit hole, and it looks like, our DNS solution is
just not compatible with the plugin after this commit:
https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/commit/4c0f2e72d...
I don't know, how the industry actually use CNAME records, but we have a
rule, that we use independet DNS names for services, and machines itself.
This become handy, when you migrate/upgrade services, like LDAP to another
machine, and I think, this should be supported.
Our LDAP server service domain is actually ldap-master.niif.hu and
ldap.niif.hu, but in real, there are two servers, which actually have
their own FQDNs.
The actual DNS structure:
host -t CNAME ldap-master.niif.hu
ldap-master.niif.hu is an alias for elm.niif.hu.
host -t CNAME ldap.niif.hu
ldap.niif.hu is an alias for holly.ldap.einfra.hu.
host elm.niif.hu
elm.niif.hu has address 193.225.14.175
elm.niif.hu has IPv6 address 2001:738:0:701::3
host holly.ldap.einfra.hu
holly.ldap.einfra.hu has address 193.224.92.6
holly.ldap.einfra.hu has IPv6 address 2001:738:0:431::6
The commit above (as far as I understand) only tries to resolve A and AAAA
records in DNS, and drop the connection if it not found. Of course, the
certificate only have ldap-master and ldap.niif.hu in it, so using holly
end elm does not solve the problem (also, if the service will be migrated,
the service domain will be kept, but not the machine's FQDN, since we
cannot afford to shut down one of our LDAP server for a migration windows.
We've tried to downgrade the package to 1.4.3, which is works fine.
The actual error looks like this (engine.log)
2021-09-07 15:53:09,833+02 WARN
[org.ovirt.engine.extension.aaa.ldap.AuthnExtension] (default task-1) []
[ovirt-engine-extension-aaa-ldap.authn::NIIFLdap-authn] Cannot initialize
LDAP framework, deferring initialization. Error: An error occurred while
attempting to connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default
task-1) [] Internal Server Error: An error occurred while attempting to
connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.service.SsoService] (default task-1) [] An error
occurred while attempting to connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,854+02 ERROR
[org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-1) []
server_error: An error occurred while attempting to connect to server
ldap-master.niif.hu:636: IOException(LDAPException(resultCode=91
(connect error), errorMessage='An error occurred while attempting to
establish a connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
As we tried to run the setup tool, that is also looks broken (after that,
we've copied the required files to /etc/ovirt-engine/aaa and
/etc/ovirt-engine/extensions.d/ from other, working hosted engine) so we've
tested the plugin itself, and the setup too, but no luck.
I think, this (CNAME in DNS) should be working with the plugin.
Could you please investigate this issue? (we're here to help test the
repaired version/patch, if needed, but not feel the knowledge to create the
patch ourself)
From very casual skimming of the patch, it seems to me like you should be
able to restore the previous behavior by setting in your conf file
'pool.default.socketfactory.resolver.detectIPVersion = false'. Can you
please try that?
Adding Martin.
Best regards,
--
Didi
6:32 a.m.
New subject: ovirt-engine-extension-aaa-ldap broken, if using CNAME for LDAP server DNS
Hy!
2021. 09. 09. 14:33 keltezéssel, Yedidyah Bar David írta:
From very casual skimming of the patch, it seems to me like you
should
be able to restore the previous behavior by setting in your conf file
'pool.default.socketfactory.resolver.detectIPVersion = false'. Can you
please try that?
We've tried it, but no luck...
[root@virt ~]# cat /etc/ovirt-engine/aaa/NIIFLdap.properties include =
<rfc2307-generic.properties> vars.user = ... vars.password = ...
pool.default.auth.simple.bindDN = ${global:vars.user}
pool.default.auth.simple.password = ${global:vars.password}
pool.default.serverset.type= failover
pool.default.serverset.failover.00.server = ldap.niif.hu
pool.default.serverset.failover.00.port =
636pool.default.serverset.failover.01.server = ldap-master.niif.hu
pool.default.serverset.failover.01.port = 636pool.default.ssl.enable =
truepool.default.socketfactory.resolver.detectIPVersion = false
After that, the engine was restarted, and we still got this error on GUI:
server_error: An error occurred whileattempting toconnect toserver
ldap-master.niif.hu:636: IOException(LDAPException(resultCode=91(connect
error), errorMessage='An error occurred while attempting to establish a
connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
Could you please validate this CNAME issue with this new behaviour of
IPv4/6 detection?
Thanks:
Peter
--
*Erdősi Péter
* /Informatikus, IKT Fejlesztési Főosztály /
*Kormányzati Informatikai Fejlesztési Ügynökség
* cím: 1134 Budapest, Váci út 35.
tel: +36 1 450 3080 e-mail: erdosi.peter(a)kifu.gov.hu
<mailto:erdosi.peter@kifu.gov.hu>
KIFÜ - www.kifu.gov.hu <http://kifu.gov.hu/kifu/>
8:54 a.m.
New subject: ovirt-engine-extension-aaa-ldap broken, if using CNAME for LDAP server DNS
Hi,
On Wed, Sep 8, 2021 at 1:36 AM Erdősi Péter <erdosi.peter(a)kifu.gov.hu>
wrote:
Hello!
We're started the upgrade one of our oVirt clusters to the recent 4.4
minor version (4.4.8.5-1.el8) and we found a bug in the recent change of
the aaa-ldap plugin.
This bug came along after the IPv4/IPv6 selection introduced in
ovirt-engine-extension-aaa-ldap 1.4.4
We've dig down in the rabbit hole, and it looks like, our DNS solution is
just not compatible with the plugin after this commit:
https://github.com/oVirt/ovirt-engine-extension-aaa-ldap/commit/4c0f2e72d...
I don't know, how the industry actually use CNAME records, but we have a
rule, that we use independet DNS names for services, and machines itself.
This become handy, when you migrate/upgrade services, like LDAP to another
machine, and I think, this should be supported.
Actually we haven't changed the way we resolve DNS records, we have just
changed a way to detect if we should resolve to IPv4 or IPv6 or both. In
previous version we decided to detect relevant IP version if an active
interface with IP version existed on engine machine. But this was not
reliable enough, so now we get an addresses of default gateway and we ask
for A and/or AAAA record upon results we get.
Our LDAP server service domain is actually ldap-master.niif.hu and
ldap.niif.hu, but in real, there are two servers, which actually have
their own FQDNs.
The actual DNS structure:
host -t CNAME ldap-master.niif.hu
ldap-master.niif.hu is an alias for elm.niif.hu.
host -t CNAME ldap.niif.hu
ldap.niif.hu is an alias for holly.ldap.einfra.hu.
host elm.niif.hu
elm.niif.hu has address 193.225.14.175
elm.niif.hu has IPv6 address 2001:738:0:701::3
host holly.ldap.einfra.hu
holly.ldap.einfra.hu has address 193.224.92.6
holly.ldap.einfra.hu has IPv6 address 2001:738:0:431::6
The commit above (as far as I understand) only tries to resolve A and AAAA
records in DNS, and drop the connection if it not found. Of course, the
certificate only have ldap-master and ldap.niif.hu in it, so using holly
end elm does not solve the problem (also, if the service will be migrated,
the service domain will be kept, but not the machine's FQDN, since we
cannot afford to shut down one of our LDAP server for a migration windows.
We've tried to downgrade the package to 1.4.3, which is works fine.
The actual error looks like this (engine.log)
2021-09-07 15:53:09,833+02 WARN
[org.ovirt.engine.extension.aaa.ldap.AuthnExtension] (default task-1) []
[ovirt-engine-extension-aaa-ldap.authn::NIIFLdap-authn] Cannot initialize
LDAP framework, deferring initialization. Error: An error occurred while
attempting to connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default
task-1) [] Internal Server Error: An error occurred while attempting to
connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,833+02 ERROR
[org.ovirt.engine.core.sso.service.SsoService] (default task-1) [] An error
occurred while attempting to connect to server ldap-master.niif.hu:636:
IOException(LDAPException(resultCode=91 (connect error), errorMessage='An
error occurred while attempting to establish a connection to server
ldap-master.niif.hu/193.225.14.175:636: UnknownHostException(
ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
2021-09-07 15:53:09,854+02 ERROR
[org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-1) []
server_error: An error occurred while attempting to connect to server
ldap-master.niif.hu:636: IOException(LDAPException(resultCode=91
(connect error), errorMessage='An error occurred while attempting to
establish a connection to server ldap-master.niif.hu/193.225.14.175:636:
UnknownHostException(ldap-master.niif.hu), ldapSDKVersion=4.0.14,
revision=c0fb784eebf9d36a67c736d0428fb3577f2e25bb'))
So this UnknownHost exception is a bit confusing, but I guess you have hit
the same issue as we found out with some of our DNS servers. If you try to
fetch both A and AAAA records from the DNS server, it will just timeout in
a while and unboundid-ldap-skd takes that as an UnknownHost exception. If
you ask the same DNS server to fetch A and AAA records in separate
requests, then both records are returned immediately. Unfortunately we
haven't been able to find out why this happens on some of our testing
servers while it works perfectly fine on others. But you can perform a test
using attached Java test tool:
javac Test.java
java -cp . Test <DNS_SERVER_IP> <LDAP FQDN>
The test tool just asks for A address, then AAAA address and then both at
once, so if you hit the above issue you will see an exception when asking
for both records at once.
Anyway if you want to fix this issue in production you will either need to
fix your DNS server (but again no idea what the issue is) or disable either
IPv4 or IPv6 address usage in aaa ldap configuration file. For example if
you would like to use only IPv4, you need to edit your
/etc/ovirt-engine/aaa/<PROFILE>.properties and add following options:
pool.default.socketfactory.resolver.detectIPVersion = false
pool.default.socketfactory.resolver.supportIPv6 = false
Regards,
Martin
As we tried to run the setup tool, that is also looks broken (after
that,
we've copied the required files to /etc/ovirt-engine/aaa and
/etc/ovirt-engine/extensions.d/ from other, working hosted engine) so we've
tested the plugin itself, and the setup too, but no luck.
I think, this (CNAME in DNS) should be working with the plugin.
Could you please investigate this issue? (we're here to help test the
repaired version/patch, if needed, but not feel the knowledge to create the
patch ourself)
Regards:
Peter
--
* Erdősi Péter * * Informatikus, IKT Fejlesztési Főosztály *
* Kormányzati Informatikai Fejlesztési Ügynökség * cím: 1134 Budapest,
Váci út 35.
tel: +36 1 450 3080 e-mail: erdosi.peter(a)kifu.gov.hu
[image: KIFÜ - www.kifu.gov.hu] <http://kifu.gov.hu/kifu/>
_______________________________________________
Devel mailing list -- devel(a)ovirt.org
To unsubscribe send an email to devel-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/VUWRCEQWLJX...
--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.
1199
days inactive
1207
days old
3 comments
3 participants
participants (3)
-
Erdősi Péter -
Martin Perina -
Yedidyah Bar David