Re: [Kimchi-devel] Raised #739, CURL GET list of resources for 2500+ resource in list, timed-out.

Hello Everyone, Problem Statement: We notice that get list of resources for approx 2500+ resources it was timing out when the command executed on system, gives list of resources greater than 2500+. Considering that to get the list of resource's identifiers(in get_list()) and to get info for each resource(lookup()), there is just one single command(which return lines where each line has info for one resource), and 2500+ resources are present. For e.g. lscss in system z gives output like Device Subchan. DevType CU Type Use PIM PAM POM CHPIDs ---------------------------------------------------------------------- 0.0.0200 0.0.0000 3390/0a 3990/e9 yes e0 e0 ff b0b10d00 00000000 0.0.0201 0.0.0001 3390/0a 3990/e9 yes e0 e0 ff b0b10d00 00000000 0.0.0202 0.0.0002 3390/0c 3990/e9 e0 e0 ff b0b10d00 00000000 ... ... With the current up streamed code, 1) it executes corresponding command and loop through the output to get the list of resource identifiers.(in Collection's get_list()) 2) another loop for looping through the list(output of get_list()) to get the identifier and then foreach get its info details again by running the same command and looping through command output for getting info for that identifier. So, O(n*2) in worse case. n--> is number of lines command output has, each line correspond to one resource. Options to reduce the Big-O complexity for such scenario: In case where a single command gives full resource info, this can be avoided, with below suggested ways: While looping through command output in get_list() just compose list of dict where each dict will have info for each resource and return the list of dict which has all info of the resource(instead of list of resource identifiers). And while doing looping through the get_list output(), each entry will give dict of resource, so just assign it to self.info of each resource. This avoids another looping mention in #2*. And makes it O(n), n--> is number of lines command output has, each line correspond to one resource.. Just FYI: With Current up streamed code for 2500+ resource, Curl API call for GET list of all resource was timing out, and with this improvement we made it in approx 1.3sec approx. One way is to overridden the _get_resources(..) method in corresponding control class. And another way I can think of is to incorporate this option by adding another Boolean parameter in base.py's _get_resource(..). And depending upon the parameter either call lookup() for each resource or we do the one suggested above. But as this is being called from get(), we need to figure out the way to pass this parameter from control/model of the corresponding resource class. Code changes I am suggesting here will look similar to below. def _get_resources(self, flag_filter, bulk_list): """ In case of bulk_list, get_list of model is suppose to return, dict. """ try: get_list = getattr(self.model, model_fn(self, 'get_list')) idents = get_list(self.model_args, *flag_filter) res_list = [] for ident in idents: # internal text, get_list changes ident to unicode for sorted args = self.resource_args + [ident] res = self.resource(self.model, *args) if(bulk_list): res.info = ident else: res.lookup() res_list.append(res) return res_list except AttributeError: return [] Please provide your inputs. Thanks, Archana Singh On 10/13/2015 6:43 PM, Aline Manera wrote: