Hi Anton,
Just to let you know. I investigated this issue. If you want to use keycloak in version >=10  you would need to define all additional scopes as 'optional client scopes' in your client configuration.
In my case, on my test environment, I only had to add  'ovirt-ext=auth:sequence-priority=~' but in your case you may need all listed in error_description:
{"error_description":"Cannot authenticate user Invalid scopes: ovirt-app-api ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access.","error":"access_denied"}
This configuration change is required because it has been changed/fixed how 'unknown' scopes are handled in keycloak. Now keycloak must always be aware of all scopes and previously unknown ones were simply ignored.

Here is BZ with details:
https://bugzilla.redhat.com/show_bug.cgi?id=1849569

best,
Artur


On Tue, Jun 23, 2020 at 5:03 PM Artur Socha <asocha@redhat.com> wrote:
On Tue, 2020-06-23 at 14:41 +0000, Anton Louw wrote:


Hi Artur,

 

Apologies for the late response. So we have downgraded the version of KeyCloak, and all seems to be working 100% again, I can obtain a token, and do API calls.

Hi Anton,
I'm glad it works now. This keycloak version (9.0.x) will stay for some time the recommended & supported choice for oVirt because it is part of 'Red Hat SSO' just like oVirt is part of 'Red Hat Virtualization'.
Artur

 

Thank you very much for all the help

 

From: Artur Socha <asocha@redhat.com>
Sent: 22 June 2020 16:52
To: Anton Louw <Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Cc: Stephen Hutchinson <Stephen.Hutchinson@voxtelecom.co.za>
Subject: Re: [ovirt-users] KeyCloak Integration

 

On Mon, 2020-06-22 at 15:14 +0200, Artur Socha wrote:

Anton,

I managed to re-create the issue on my local environment. 

Previously I tested it against Keycloak 8.0.1 with users loaded from LDAP. Currently I have users/groups created via Keycloak management panel. I need to investigate it further which of the two changes is the root cause (it works fine with the old setup)

 

One more update: it seems the issue is keycloak version related. Trying to figure out what was changed and how it affected engine sso integration.

 

Latest keycloak version I tested and verified that works is 9.0.3. Perhaps it could be possible for you to use it until we fully support 10.0.x ?

Artur

 


Anton Louw
Cloud Engineer: Storage and Virtualization at Vox

T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

F
 
T
 
I
 
L
 
Y
 

Artur

 

On Mon, 2020-06-22 at 11:05 +0000, Anton Louw wrote:

 

Hi Artur,

 

Great, thanks a lot! 😊

 

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

From: Artur Socha <asocha@redhat.com>
Sent: 22 June 2020 11:23
To: Anton Louw <Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Cc: Stephen Hutchinson <Stephen.Hutchinson@voxtelecom.co.za>
Subject: Re: [ovirt-users] KeyCloak Integration

 

Hi Anton,

Thanks for the specs. I have create BZ issue for tracking:

Feel free to add comments/change it when needed.

 

Artur

 

On Fri, 2020-06-19 at 10:57 +0000, Anton Louw wrote:

 

Hi Artur,

 

Please see below:

 

ovirt-engine.noarch                     4.3.10.4-1.el7    @ovirt-4.3

ovirt-engine-extension-aaa-misc.noarch  1.0.4-1.el7       @ovirt-4.3

mod_auth_openidc.x86_64                 1.8.8-5.el7       @base

 

[root@virt ~]# cat /etc/*elease

CentOS Linux release 7.7.1908 (Core)

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"

 

CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"

 

CentOS Linux release 7.7.1908 (Core)

CentOS Linux release 7.7.1908 (Core)

 

KeyCloak –

 

Server Version

10.0.1

 

Thanks a lot for your help Artur. Please let me know if you need anything else.

 

From: Artur Socha <asocha@redhat.com>
Sent: 19 June 2020 12:39
To: Anton Louw <Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Cc: Stephen Hutchinson <Stephen.Hutchinson@voxtelecom.co.za>
Subject: Re: [ovirt-users] KeyCloak Integration

 

On Fri, 2020-06-19 at 10:21 +0000, Anton Louw wrote:

 

Yes I didn’t get to the OVN part yet, as I first wanted to test the if the token can be obtained.

 

This is the first time we are testing KeyCloak in any environment, so we have never been able to obtain a token for API access.

 

Please post the exact versions of:

- ovirt-engine* :   

yum list --installed | grep ovirt-engine 

yum list --intalled | grep ovirt-engine-extension-aaa-misc

yum list --installed | grep mod_auth_openidc

- keycloak

- OS

cat /etc/*elease    

 

I'll submit a bug ... which, most likely, I will assign to myself anyway :)

 

Artur

 

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

Thanks

 

From: Artur Socha <asocha@redhat.com>
Sent: 19 June 2020 12:16
To: Anton Louw <Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Cc: Stephen Hutchinson <Stephen.Hutchinson@voxtelecom.co.za>
Subject: Re: [ovirt-users] KeyCloak Integration

 

On Fri, 2020-06-19 at 10:03 +0000, Anton Louw wrote:

 

Hi Artur,

 

Sure, please see below output:

 

[root@virt ~]# curl -vvv -H "Accept:application/json" 'https://virt.example.co.za/ovirt-engine/sso/oauth/token?grant_type=password&username=myuser&password=mypass&scope=ovirt-app-api'

* About to connect() to virt.example.co.za port 443 (#0)

*   Trying 127.0.0.1...

* Connected to virt.example.co.za (127.0.0.1) port 443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

* Server certificate:

*       subject: CN=*.example.co.za,OU=Domain Control Validated

*       start date: Sep 25 07:46:12 2019 GMT

*       expire date: Oct 02 07:39:01 2020 GMT

*       common name: *example.co.za

*       issuer: CN=Starfield Secure Certificate Authority - G2,OU=http://certs.starfieldtech.com/repository/,O="Starfield Technologies, Inc.",L=Scottsdale,ST=Arizona,C=US

> GET /ovirt-engine/sso/oauth/token?grant_type=password&username=myuser&password=mypass&scope=ovirt-app-api HTTP/1.1

> User-Agent: curl/7.29.0

> Host: virt.example.co.za

> Accept:application/json

< HTTP/1.1 400 Bad Request

< Date: Fri, 19 Jun 2020 09:52:11 GMT

< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips

< Set-Cookie: locale=en_US; path=/; secure; HttpOnly; Max-Age=2147483647; Expires=Wed, 07-Jul-2088 13:06:18 GMT

< X-XSS-PROTECTION: 1; MODE=BLOCK

< X-CONTENT-TYPE-OPTIONS: NOSNIFF

< X-FRAME-OPTIONS: SAMEORIGIN

< Content-Type: application/json

< Content-Length: 233

< Connection: close

* Closing connection 0

{"error_code":"access_denied","error":"Cannot authenticate user Invalid scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access."}

 

1) Test connection using python script (from the blog post ) using sdk. I suspect it will not work either.

Testing from Python gives me the same error as well.

 

2) I saw some errors in the log on revoking token. Please go to keycloak admin panel, and under users kill all its active sessions. Then, please without logging in to engine admin UI, use that curl to obtain token.

Tested this again, but still getting the below:

{"error_code":"access_denied","error":"Cannot authenticate user Invalid scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access."}

 

Thanks for these test ... unfortunately nothing helped

 

 

3) Does it work without OVN integration enabled?

Can you explain a bit more? How can I disable OVN integration to test this?

 

I had in mind reverting OVN vs Keycloak integration done according to "Configuring OVN" chapter in https://blogs.ovirt.org/2019/01/federate-ovirt-engine-authentication-to-openid-connect-infrastructure/

Unless, of course, you skipped it.

 

Most likely you found a bug. Have you ever been able to obtain token for api access with keycloak integration (even with you previous environments)? 

I am now trying to understand what happened and how to reproduce it before submitting the bug into http://bugzilla.redhat.com

 

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

 

Thanks

 

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

From: Artur Socha <asocha@redhat.com>
Sent: 19 June 2020 11:40
To: Anton Louw <Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Cc: Stephen Hutchinson <Stephen.Hutchinson@voxtelecom.co.za>
Subject: Re: [ovirt-users] KeyCloak Integration

 

On Fri, 2020-06-19 at 08:34 +0000, Anton Louw wrote:

 

Hi Artur,

 

Thank you for the quick response.

 

I have actually tried creating another user, but I still get the same error. I have attached the output of curl -vvv as well as the logs the engine and keycloak logs.

 

This `curl -vvv ...` is actually is incorrect because it is missing -H before 'Accept' header. However, previous attempts that led to this error seemed to be fine. Could you just re-send output of the correct curl?

 

There are few things we can test to try to narrow down the root cause:

 

1) Test connection using python script (from the blog post ) using sdk. I suspect it will not work either.

 

2) I saw some errors in the log on revoking token. Please go to keycloak admin panel, and under users kill all its active sessions. Then, please without logging in to engine admin UI, use that curl to obtain token.

 

3) Does it work without OVN integration enabled?

 

Artur

 

 

 

Thank you

 

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

From: Artur Socha <asocha@redhat.com>
Sent: 19 June 2020 10:23
To: Anton Louw <
Anton.Louw@voxtelecom.co.za>; users@ovirt.org
Subject: Re: [ovirt-users] KeyCloak Integration

 

O

n Fri, 2020-06-19 at 07:35 +0000, Anton Louw via Users wrote:

 

Hi Everybody,

 

Hi Anton,

 

So I have implemented KeyCloak into our oVirt environment, which works, up until a point. So WebUI access works, but when calling the API, using:

curl -k -H "Accept: application/json" 'https://virt.example.co.za/ovirt-engine/sso/oauth/token?grant_type=password&username=admin@openidchttp&password=mypass&scope=ovirt-app-api'

 

I get the below error:

 

{"error_description":"Cannot authenticate user Invalid scopes: ovirt-app-api ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access.","error":"access_denied"}

 

If my configs are removed, and I use “admin@internal” for my username, then it works.

 

I followed the below article step by step, and I double checked that all the scopes are added into KeyCloak (ovirt-app-api and ovirt-app-admin)

 

https://blogs.ovirt.org/2019/01/federate-ovirt-engine-authentication-to-openid-connect-infrastructure/

 

Anybody have any ideas?

 

It is my blind shot but could create & check another user?

 

One more thing to check please use curl -vvv to check if there are any redirects along the way.

I will check keycloak settings on my setup - perhaps there is something non-obvious that could have been missed.

 

Any chance to get a bit more logs from engine.log and even from keycloak? Perhaps there is something there that could help.

 

Artur

 

 

Thank you

 

Anton Louw

Cloud Engineer: Storage and Virtualization at Vox


T:  087 805 0000 | D: 087 805 1572
M: N/A
E: anton.louw@voxtelecom.co.za
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
www.vox.co.za

 

F

 

T

 

I

 

L

 

Y

 

 

#VoxBrand


Disclaimer

The contents of this email are confidential to the sender and the intended recipient. Unless the contents are clearly and entirely of a personal nature, they are subject to copyright in favour of the holding company of the Vox group of companies. Any recipient who receives this email in error should immediately report the error to the sender and permanently delete this email from all storage devices.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more
Click Here.

 

_______________________________________________
Users mailing list -- 

users@ovirt.org

 

 

 

 

 

 

 
To unsubscribe send an email to 

users-leave@ovirt.org

 

 

 

 

 

 

 
Privacy Statement: 

https://www.ovirt.org/privacy-policy.html

 

 

 

 

 

 

 
oVirt Code of Conduct: 

https://www.ovirt.org/community/about/community-guidelines/

 

 

 

 

 

 

 
List Archives: 

https://lists.ovirt.org/archives/list/users@ovirt.org/message/CC54IPZLYJYE2B3NP4LT4TN4CJX4C7BU/

 

 

 

 

 

 

 

 

 

 

 

 




--
Artur Socha
Senior Software Engineer, RHV
Red Hat