I'm trying to get rhevm 3.1 (which seems to be pretty much ovirt 3.1 from what I can tell) authenticating against our active directory infrastructure bu am having some difficulty that I don't quite understand and was hoping someone may know what is happening. The server where rhevm/ovirt is running is a RHEL6 based server that has NIS configured (with user home directories mounted via nfs/automounter). The userids in nis match the userids in our ActiveDirectory server (in fact the passwords should match too since there is a sync between the two). I added the Activedirectory server into ovirt (through rhevm-manage-domains) and it is added/validated successfully. As the local admin user I can go in and search agains the active directory, add permissions, etc. But... If I try to log into the webadmin/user portals with one of the active directory accounts it seems to hang... and I noticed that it seems to be trying to mount the home directory of a bunch of users via the automounter (perhaps its trying to mount everyones home directory... can't tell). This takes a super long time since the home directories are all across the world and nfs access to some of these filesystems is really slow... i'm not sure it will ever complete... certainly not before the user gives up. Anyone know what would cause this? I wouldn't think this should happen. I was thinking it should just authenticate the password and then look at the permissions granted inside overt/rhevm.

On 3/2/13 2:51 PM, Itamar Heim wrote: > On 01/03/2013 18:54, Keith Mitchell wrote: >> >> I'm trying to get rhevm 3.1 (which seems to be pretty much ovirt 3.1 >> from what I can tell) authenticating against our active directory >> infrastructure bu am having some difficulty that I don't quite >> understand and was hoping someone may know what is happening. >> >> The server where rhevm/ovirt is running is a RHEL6 based server that has >> NIS configured (with user home directories mounted via >> nfs/automounter). The userids in nis match the userids in our >> ActiveDirectory server (in fact the passwords should match too since >> there is a sync between the two). >> >> I added the Activedirectory server into ovirt (through >> rhevm-manage-domains) and it is added/validated successfully. As the >> local admin user I can go in and search agains the active directory, add >> permissions, etc. >> >> But... If I try to log into the webadmin/user portals with one of the >> active directory accounts it seems to hang... and I noticed that it >> seems to be trying to mount the home directory of a bunch of users via >> the automounter (perhaps its trying to mount everyones home directory... >> can't tell). This takes a super long time since the home directories >> are all across the world and nfs access to some of these filesystems is >> really slow... i'm not sure it will ever complete... certainly not >> before the user gives up. >> >> Anyone know what would cause this? I wouldn't think this should >> happen. I was thinking it should just authenticate the password and >> then look at the permissions granted inside overt/rhevm. > > there is no need for the engine (rhev) machine to be part of the AD > domain for AD authentication to work, and i don't see why this should > happen. > yair/juan - thoughts? > Turns out the home directory mounting thing had nothing to do with my login issues or ovirt... The home directory issue was due to an issue with mod_dnssd (part of apache) in RHEL6. But even after fixing that, I still have login issues. Whenever I try to authenticate against active directory the webadmin/user gui seems to hang. I've looked at the network trace and it looks like the active directory authentication succeeded without issue, but the login screen just hangs. I can log in with the local admin user fine and I don't see anything in the engine.log files. Perhaps there may be some debug I can turn on to help identify what it is doing?

From: "Keith Mitchell" <kamitch(a)cisco.com&gt; To: "Itamar Heim" <iheim(a)redhat.com&gt; Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt;, "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; Sent: Sunday, March 3, 2013 7:15:16 AM Subject: Re: [Users] webadmin login issues with AD On 3/2/13 11:57 PM, Itamar Heim wrote: > On 03/03/2013 06:41, Keith Mitchell wrote: >> On 3/2/13 2:51 PM, Itamar Heim wrote: >>> On 01/03/2013 18:54, Keith Mitchell wrote: >>>> >>>> I'm trying to get rhevm 3.1 (which seems to be pretty much ovirt >>>> 3.1 >>>> from what I can tell) authenticating against our active >>>> directory >>>> infrastructure bu am having some difficulty that I don't quite >>>> understand and was hoping someone may know what is happening. >>>> >>>> The server where rhevm/ovirt is running is a RHEL6 based server >>>> that has >>>> NIS configured (with user home directories mounted via >>>> nfs/automounter). The userids in nis match the userids in our >>>> ActiveDirectory server (in fact the passwords should match too >>>> since >>>> there is a sync between the two). >>>> >>>> I added the Activedirectory server into ovirt (through >>>> rhevm-manage-domains) and it is added/validated successfully. As >>>> the >>>> local admin user I can go in and search agains the active >>>> directory, add >>>> permissions, etc. >>>> >>>> But... If I try to log into the webadmin/user portals with one >>>> of the >>>> active directory accounts it seems to hang... and I noticed that >>>> it >>>> seems to be trying to mount the home directory of a bunch of >>>> users via >>>> the automounter (perhaps its trying to mount everyones home >>>> directory... >>>> can't tell). This takes a super long time since the home >>>> directories >>>> are all across the world and nfs access to some of these >>>> filesystems is >>>> really slow... i'm not sure it will ever complete... certainly >>>> not >>>> before the user gives up.

>>>> >>>> Anyone know what would cause this? I wouldn't think this should >>>> happen. I was thinking it should just authenticate the password >>>> and >>>> then look at the permissions granted inside overt/rhevm. >>> >>> there is no need for the engine (rhev) machine to be part of the >>> AD >>> domain for AD authentication to work, and i don't see why this >>> should >>> happen. >>> yair/juan - thoughts? >>> >> Turns out the home directory mounting thing had nothing to do with >> my >> login issues or ovirt... The home directory issue was due to an >> issue >> with mod_dnssd (part of apache) in RHEL6. >> >> But even after fixing that, I still have login issues. Whenever I >> try >> to authenticate against active directory the webadmin/user gui >> seems to >> hang. I've looked at the network trace and it looks like the >> active >> directory authentication succeeded without issue, but the login >> screen >> just hangs. >> >> I can log in with the local admin user fine and I don't see >> anything in >> the engine.log files. Perhaps there may be some debug I can turn >> on to >> help identify what it is doing? >> >> >> > > does the rest api works for an AD user? > (user@domain is the user name format. url is http://xxx/api) That seems to hang too.

From: "Keith Mitchell" <kamitch(a)cisco.com&gt; To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt;, "Itamar Heim" <iheim(a)redhat.com&gt; Sent: Sunday, March 3, 2013 1:48:27 PM Subject: Re: [Users] webadmin login issues with AD On 3/3/13 1:45 AM, Yair Zaslavsky wrote: > > ----- Original Message ----- >> From: "Keith Mitchell" <kamitch(a)cisco.com&gt; >> To: "Itamar Heim" <iheim(a)redhat.com&gt; >> Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" >> <jhernand(a)redhat.com&gt;, "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >> Sent: Sunday, March 3, 2013 7:15:16 AM >> Subject: Re: [Users] webadmin login issues with AD >> >> On 3/2/13 11:57 PM, Itamar Heim wrote: >>> On 03/03/2013 06:41, Keith Mitchell wrote: >>>> On 3/2/13 2:51 PM, Itamar Heim wrote: >>>>> On 01/03/2013 18:54, Keith Mitchell wrote: >>>>>> I'm trying to get rhevm 3.1 (which seems to be pretty much >>>>>> ovirt >>>>>> 3.1 >>>>>> from what I can tell) authenticating against our active >>>>>> directory >>>>>> infrastructure bu am having some difficulty that I don't quite >>>>>> understand and was hoping someone may know what is happening. >>>>>> >>>>>> The server where rhevm/ovirt is running is a RHEL6 based >>>>>> server >>>>>> that has >>>>>> NIS configured (with user home directories mounted via >>>>>> nfs/automounter). The userids in nis match the userids in our >>>>>> ActiveDirectory server (in fact the passwords should match too >>>>>> since >>>>>> there is a sync between the two). >>>>>> >>>>>> I added the Activedirectory server into ovirt (through >>>>>> rhevm-manage-domains) and it is added/validated successfully. >>>>>> As >>>>>> the >>>>>> local admin user I can go in and search agains the active >>>>>> directory, add >>>>>> permissions, etc. >>>>>> >>>>>> But... If I try to log into the webadmin/user portals with one >>>>>> of the >>>>>> active directory accounts it seems to hang... and I noticed >>>>>> that >>>>>> it >>>>>> seems to be trying to mount the home directory of a bunch of >>>>>> users via >>>>>> the automounter (perhaps its trying to mount everyones home >>>>>> directory... >>>>>> can't tell). This takes a super long time since the home >>>>>> directories >>>>>> are all across the world and nfs access to some of these >>>>>> filesystems is >>>>>> really slow... i'm not sure it will ever complete... certainly >>>>>> not >>>>>> before the user gives up. > Hi, > Currently, both search of users in specific domain + login perform > both authentication + authorization check + running ldap queries ( > authorization is a part of the login). > It seems really odd to me that login takes you quite some time, and > search of users/groups does not. > What other info can you provide about the user you try to login to? > Did you give permissions to many entities? At the moment there is just one AD account in the permissions and that is my AD account. At first I added "Domain Users" to the permissions, but I took that out and just stuck in my user account to see if that helped. In ovirt, my account is part of the System (i.e. top-level) and is give then SuperUser privilege, just like the local admin account. My account is just a user account (no admin rights in the AD domain). I am a member of quite a few groups on the AD domain but I wouldn't think ovirt would care about that or need to query each group I am a member of.

Ultimately I was hoping to add the domain users group into the permissions to let anyone in the domain have access :) I used wireshark to sniff for the LDAP packets instead of just the kerberos packets and during the "hang" it is sending constant ldap packets back and forth. Looks like its doing bind request, then it succeeds and then there is a SASL-GSSAPI exchange followed by a connection close (i.e. FIN packet) and then it starts all over again. Everything is encrypted so its difficult to see anything in the packets. On this particular sniff, the packets went back and forth for 10 minutes and then they stopped and when I looked it had logged me into the GUI. I don't usually wait that long. I have on occasion just left the window up and sometimes it would eventually log me in and sometimes it never logged me in... in the never cases the login window just stays there spinning until I reload the web page... perhaps something timed out and it gave up before the exchange finished. Are there any debugs I can turn on in ovirt to have it spit out what its doing?

From: "Keith Mitchell" <kamitch(a)cisco.com&gt; To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt;, "Itamar Heim" <iheim(a)redhat.com&gt; Sent: Sunday, March 3, 2013 2:28:38 PM Subject: Re: [Users] webadmin login issues with AD On 3/3/13 6:57 AM, Yair Zaslavsky wrote: > Please elaborate on "quite a few groups" - actually this is a well > known issue. > I was afraid you might have permissions on "too many objects" or > that the account is a member of too many groups. > However, being a member of too many groups should have caused the > search to be slow/hang as well. I don't have an exact count, but I think its along the order of magnitude of 300-400.

I didn't notice the searches (when trying to add the account to the ovirt permissions) was unbearable slow like the logins.

But why does ovirt even care about the groups? I thought it was only using AD for authentication and that the authorization was all done internally through the permissions granted. Or is that just a standard "library" that ovirt is using that is doing this? I don't suppose there is a work around? > Hi, you can look at the following link - > > http://docs.oracle.com/javase/jndi/tutorial/ldap/security/sasl.html > > we support changing sasl_qop. You can use engine-config to do that. > engine-config -s sasl_qop=auth will change Quality of Propetction > to be only at authentication. > Please let us know if using that you will be able to see the ldap > queries (i.e - have them plain and not encrypted) Ok, yeah that allows me to see the ldap requests... Looks like its going through all of the groups I am a member of and doing a search on each one. And in a not so terribly efficient way (connect/bind/search/close... repeat).

On 3/3/13 7:42 AM, Yair Zaslavsky wrote: > > ----- Original Message ----- >> From: "Keith Mitchell" <kamitch(a)cisco.com&gt; >> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >> Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" >> <jhernand(a)redhat.com&gt;, "Itamar Heim" <iheim(a)redhat.com&gt; >> Sent: Sunday, March 3, 2013 2:28:38 PM >> Subject: Re: [Users] webadmin login issues with AD >> >> On 3/3/13 6:57 AM, Yair Zaslavsky wrote: >>> Please elaborate on "quite a few groups" - actually this is a well >>> known issue. >>> I was afraid you might have permissions on "too many objects" or >>> that the account is a member of too many groups. >>> However, being a member of too many groups should have caused the >>> search to be slow/hang as well. >> I don't have an exact count, but I think its along the order of >> magnitude of 300-400. > Hi, > I gave an incorrect explanation before (I thought about it and > understood where my error lies ). > If I add a user using engine-manage-domains and do not provide > -addPermissions, I will still be able to login to the system using > admin@internal, and perform search for users & groups. > This means I do not need to have permissions for the user I added for > that domain to perform search so the "permissions" check is of course > not performed at search! > > The number of groups is important in login - oVirt will try to > calculate all the permissions of the users, and this is based on the > permission the user have directly on an object, or that its group has. > If the user is a member of 300 groups, oVirt tries to get information > for all that groups. > THis is why login hands, but search does not hang. I guess I don't understand why ovirt needs to do that. You should be able to get the list of groups a user is a member which I thought was sufficient for most apps to determine authorization. I know we use AD authentication for a lot of things and i've never hit this before. Changing the AD config isn't something I can do so it sounds like there is no workaround and i'll just have to live with the local authentication. Or pehaps I can stick some ldap server in front of AD that

From: "Keith Mitchell" <kamitch(a)cisco.com&gt; To: "Itamar Heim" <iheim(a)redhat.com&gt; Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt; Sent: Sunday, March 3, 2013 5:02:59 PM Subject: Re: [Users] webadmin login issues with AD On 3/3/13 8:37 AM, Itamar Heim wrote: > On 03/03/2013 15:26, Keith Mitchell wrote: >> On 3/3/13 7:42 AM, Yair Zaslavsky wrote: >>> >>> ----- Original Message ----- >>>> From: "Keith Mitchell" <kamitch(a)cisco.com&gt; >>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>> Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" >>>> <jhernand(a)redhat.com&gt;, "Itamar Heim" <iheim(a)redhat.com&gt; >>>> Sent: Sunday, March 3, 2013 2:28:38 PM >>>> Subject: Re: [Users] webadmin login issues with AD >>>> >>>> On 3/3/13 6:57 AM, Yair Zaslavsky wrote: >>>>> Please elaborate on "quite a few groups" - actually this is a >>>>> well >>>>> known issue. >>>>> I was afraid you might have permissions on "too many objects" >>>>> or >>>>> that the account is a member of too many groups. >>>>> However, being a member of too many groups should have caused >>>>> the >>>>> search to be slow/hang as well. >>>> I don't have an exact count, but I think its along the order of >>>> magnitude of 300-400. >>> Hi, >>> I gave an incorrect explanation before (I thought about it and >>> understood where my error lies ). >>> If I add a user using engine-manage-domains and do not provide >>> -addPermissions, I will still be able to login to the system >>> using >>> admin@internal, and perform search for users & groups. >>> This means I do not need to have permissions for the user I added >>> for >>> that domain to perform search so the "permissions" check is of >>> course >>> not performed at search! >>> >>> The number of groups is important in login - oVirt will try to >>> calculate all the permissions of the users, and this is based on >>> the >>> permission the user have directly on an object, or that its group >>> has. >>> If the user is a member of 300 groups, oVirt tries to get >>> information >>> for all that groups. >>> THis is why login hands, but search does not hang. >> I guess I don't understand why ovirt needs to do that. You should >> be >> able to get the list of groups a user is a member which I thought >> was >> sufficient for most apps to determine authorization. >> >> I know we use AD authentication for a lot of things and i've never >> hit >> this before. >> >> Changing the AD config isn't something I can do so it sounds like >> there >> is no workaround and i'll just have to live with the local >> authentication. Or pehaps I can stick some ldap server in front >> of >> AD that >> > > actually the issue is not getting the list of groups, rather than > ovirt is is checking which other groups these groups are part of, > to > make sure user gets the right permissions from nested groups as > well. > we didn't find an easy way to do this which doesn't involve looping > on > all the groups. > is this common for most users in your AD to have 300-400 groups? > > Thanks, > Itamar Yes, my case is fairly typical of our AD setup. Not sure what other apps are doing here, but I do know that it doesn't take this long to get logged in :) Maybe they only look at direct group membership? Or they do things in reverse... i.e. look up the groups in the access list to determine if the user that is authenticating can be found rather than traversing all the groups a user is a member of and trying to match all those groups to a group (or username) on the access list. That might run into the same issue if the group had lots of group members as opposed to user members.

Changing the way the searches are made may speed up things too (if thats possible with the framework) to not reconnect for each search and to do multiple searches on the same connection. From my network capture each search request was taking about 1.5 seconds (from the bind request to the unbind request). Just some thoughts...

From: "Keith Mitchell" <kamitch(a)cisco.com&gt; To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; Cc: "Juan Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt;, users(a)ovirt.org Sent: Sunday, March 3, 2013 5:48:48 PM Subject: Re: [Users] webadmin login issues with AD On 3/3/13 10:12 AM, Yair Zaslavsky wrote: > > ----- Original Message ----- >> From: "Keith Mitchell" <kamitch(a)cisco.com&gt; >> To: "Itamar Heim" <iheim(a)redhat.com&gt; >> Cc: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt;, users(a)ovirt.org, "Juan >> Antonio Hernandez Fernandez" <jhernand(a)redhat.com&gt; >> Sent: Sunday, March 3, 2013 5:02:59 PM >> Subject: Re: [Users] webadmin login issues with AD >> >> On 3/3/13 8:37 AM, Itamar Heim wrote: >>> On 03/03/2013 15:26, Keith Mitchell wrote: >>>> On 3/3/13 7:42 AM, Yair Zaslavsky wrote: >>>>> ----- Original Message ----- >>>>>> From: "Keith Mitchell" <kamitch(a)cisco.com&gt; >>>>>> To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; >>>>>> Cc: users(a)ovirt.org, "Juan Antonio Hernandez Fernandez" >>>>>> <jhernand(a)redhat.com&gt;, "Itamar Heim" <iheim(a)redhat.com&gt; >>>>>> Sent: Sunday, March 3, 2013 2:28:38 PM >>>>>> Subject: Re: [Users] webadmin login issues with AD >>>>>> >>>>>> On 3/3/13 6:57 AM, Yair Zaslavsky wrote: >>>>>>> Please elaborate on "quite a few groups" - actually this is a >>>>>>> well >>>>>>> known issue. >>>>>>> I was afraid you might have permissions on "too many objects" >>>>>>> or >>>>>>> that the account is a member of too many groups. >>>>>>> However, being a member of too many groups should have caused >>>>>>> the >>>>>>> search to be slow/hang as well. >>>>>> I don't have an exact count, but I think its along the order >>>>>> of >>>>>> magnitude of 300-400. >>>>> Hi, >>>>> I gave an incorrect explanation before (I thought about it and >>>>> understood where my error lies ). >>>>> If I add a user using engine-manage-domains and do not provide >>>>> -addPermissions, I will still be able to login to the system >>>>> using >>>>> admin@internal, and perform search for users & groups. >>>>> This means I do not need to have permissions for the user I >>>>> added >>>>> for >>>>> that domain to perform search so the "permissions" check is of >>>>> course >>>>> not performed at search! >>>>> >>>>> The number of groups is important in login - oVirt will try to >>>>> calculate all the permissions of the users, and this is based >>>>> on >>>>> the >>>>> permission the user have directly on an object, or that its >>>>> group >>>>> has. >>>>> If the user is a member of 300 groups, oVirt tries to get >>>>> information >>>>> for all that groups. >>>>> THis is why login hands, but search does not hang. >>>> I guess I don't understand why ovirt needs to do that. You >>>> should >>>> be >>>> able to get the list of groups a user is a member which I >>>> thought >>>> was >>>> sufficient for most apps to determine authorization. >>>> >>>> I know we use AD authentication for a lot of things and i've >>>> never >>>> hit >>>> this before. >>>> >>>> Changing the AD config isn't something I can do so it sounds >>>> like >>>> there >>>> is no workaround and i'll just have to live with the local >>>> authentication. Or pehaps I can stick some ldap server in front >>>> of >>>> AD that >>>> >>> actually the issue is not getting the list of groups, rather than >>> ovirt is is checking which other groups these groups are part of, >>> to >>> make sure user gets the right permissions from nested groups as >>> well. >>> we didn't find an easy way to do this which doesn't involve >>> looping >>> on >>> all the groups. >>> is this common for most users in your AD to have 300-400 groups? >>> >>> Thanks, >>> Itamar >> Yes, my case is fairly typical of our AD setup. >> >> Not sure what other apps are doing here, but I do know that it >> doesn't >> take this long to get logged in :) Maybe they only look at direct >> group >> membership? Or they do things in reverse... >> >> i.e. look up the groups in the access list to determine if the >> user >> that >> is authenticating can be found rather than traversing all the >> groups >> a >> user is a member of and trying to match all those groups to a >> group >> (or >> username) on the access list. That might run into the same issue >> if >> the >> group had lots of group members as opposed to user members. > Keith, > Not sure I understood this, can you please elaborate? Say you configure ovirt so that 'groupa' has some permissions. Rather than enumerating every group the user belongs to and comparing the group name to one of the groups in the ovirt permissions list, we would enumerate the groups in the ovirt permissions list and then search for the authenticated user in those. Then you only have enumerate groups that are actually used for authorization. I bet that there are typically just a few groups used in the permissions list in a typical deployment. The current method also seems to want to query the server to get all possible information and then make the decisions. Ideally you want to do the fewest number of searches possible... and short circuit the authorization algorithm as soon as sufficient data is gathered. One common optimization is to compare all of the members of a group first before you start recursing to optimize for the direct member case. In my current case I just have my user listed in the permissions. It should be able to fully authorize me without having to look at a single one of the group memberships... here are no groups listed in ovirt so all it really needs to do is match the user.

It was just a thought... I don't know much about the implementation of ovirt... just what I saw in the network trace. For my initial deployment, it looks like i'm going to have to just use local authentication as this is just way to slow with our AD infrastructure currently. >> Changing the way the searches are made may speed up things too (if >> thats >> possible with the framework) to not reconnect for each search and >> to >> do >> multiple searches on the same connection. From my network capture >> each >> search request was taking about 1.5 seconds (from the bind request >> to >> the unbind request). >> >> Just some thoughts... > One of the things we tested is introducing ldap connection pooling. > Unfortunately, the current JDK implementation automatically turns > off ldap connection pooling if more than one domain is used. > Ravi, care to elaborate a bit on your findings? >> >> >> _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users