[PATCH] plugin: fix dynamic import mechanism
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path. This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3]. The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample". This patch provides the full package context to __import__() to mimic the standard "import" statement behavior. [1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path): def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), ['']) -- 1.7.11.7
On 12/31/2013 08:21 PM, Zhou Zheng Sheng wrote:
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path.
This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3].
The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample".
This patch provides the full package context to __import__() to mimic the standard "import" statement behavior.
[1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import
Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path):
def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), ['']) Reviewed-by: Mark Wu<wudxw@linux.vnet.ibm.com>
Thanks for the fix!
Reviewed-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> Tested-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> I have find this problem, when I run test. $ sudo ./run_tests.sh test_rest Failed to import plugin plugins.sample.Drawings this patch has fixed this problem. On 12/31/2013 08:21 PM, Zhou Zheng Sheng wrote:
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path.
This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3].
The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample".
This patch provides the full package context to __import__() to mimic the standard "import" statement behavior.
[1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import
Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path):
def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), [''])
-- Thanks and best regards! Sheldon Feng(冯少合)<shaohef@linux.vnet.ibm.com> IBM Linux Technology Center
On 01/02/2014 08:41 AM, Sheldon wrote:
Reviewed-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> Tested-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com>
I have find this problem, when I run test. $ sudo ./run_tests.sh test_rest Failed to import plugin plugins.sample.Drawings
this patch has fixed this problem.
I am still getting the import error while running mockmodel tests: alinefm@alinefm:~/kimchi/tests$ sudo ./run_tests.sh test_mockmodel Failed to import plugin plugins.sample.Drawings Is it another issue or this patch should fix it too?
On 12/31/2013 08:21 PM, Zhou Zheng Sheng wrote:
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path.
This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3].
The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample".
This patch provides the full package context to __import__() to mimic the standard "import" statement behavior.
[1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import
Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path):
def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), [''])
on 2014/01/02 20:30, Aline Manera wrote:
On 01/02/2014 08:41 AM, Sheldon wrote:
Reviewed-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> Tested-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com>
I have find this problem, when I run test. $ sudo ./run_tests.sh test_rest Failed to import plugin plugins.sample.Drawings
this patch has fixed this problem.
I am still getting the import error while running mockmodel tests:
alinefm@alinefm:~/kimchi/tests$ sudo ./run_tests.sh test_mockmodel Failed to import plugin plugins.sample.Drawings
Is it another issue or this patch should fix it too?
Thanks Aline and Sheldon. The problem you found is a different one. That is because the plugins sub-directory is not in the import path when we run the test. When installed , the plugins sub-directory is in python_site_packages/kimchi/plugins, so after this patch, importing "kimchi.plugins.sample" and "plugins.sample" can both succeed. However when running test from the tests/ directory, the kimchi source is in ../src/kimchi while plugins is in ../plugins, so neither "kimchi.plugins.sample" nor "plugins.sample" would succeed. This patch only to fix the use of the __import__() function for the installed kimchi directory structure. If we want to fix the import problem in unit test, I can submit another patch. Agree?
On 12/31/2013 08:21 PM, Zhou Zheng Sheng wrote:
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path.
This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3].
The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample".
This patch provides the full package context to __import__() to mimic the standard "import" statement behavior.
[1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import
Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path):
def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), [''])
-- Thanks and best regards! Zhou Zheng Sheng / 周征晟 E-mail: zhshzhou@linux.vnet.ibm.com Telephone: 86-10-82454397
On 01/03/2014 12:41 AM, Zhou Zheng Sheng wrote:
on 2014/01/02 20:30, Aline Manera wrote:
On 01/02/2014 08:41 AM, Sheldon wrote:
Reviewed-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com> Tested-by: ShaoHe Feng <shaohef@linux.vnet.ibm.com>
I have find this problem, when I run test. $ sudo ./run_tests.sh test_rest Failed to import plugin plugins.sample.Drawings
this patch has fixed this problem.
I am still getting the import error while running mockmodel tests:
alinefm@alinefm:~/kimchi/tests$ sudo ./run_tests.sh test_mockmodel Failed to import plugin plugins.sample.Drawings
Is it another issue or this patch should fix it too?
Thanks Aline and Sheldon. The problem you found is a different one. That is because the plugins sub-directory is not in the import path when we run the test.
When installed , the plugins sub-directory is in python_site_packages/kimchi/plugins, so after this patch, importing "kimchi.plugins.sample" and "plugins.sample" can both succeed. However when running test from the tests/ directory, the kimchi source is in ../src/kimchi while plugins is in ../plugins, so neither "kimchi.plugins.sample" nor "plugins.sample" would succeed.
This patch only to fix the use of the __import__() function for the installed kimchi directory structure. If we want to fix the import problem in unit test, I can submit another patch. Agree?
Yes.
On 12/31/2013 08:21 PM, Zhou Zheng Sheng wrote:
The current kimchi plugin subsystem implement a dynamic discovery and import mechanism. It uses the Python builtin function __import__() to actually import a plugin module by name. However it turns out it only works when we start kimchi from the project source directory, not work when it is started from the installed path.
This is because the __import__() accepts a "global" arugment and it needs "__name__", "__path__" and "__package__" in this "global" dict to determine the package context [1][2]. When in kimchi/utils.py it runs "import plugins.XXX", it actually fetch the "__package__" from utils.py, which is "kimchi", then it imports "kimchi.plugins.XXX" under the hood [3].
The current import_module() function calls __import__() without giving a global dict. When kimchid is started from /usr/bin, without the package context, __import__() can not find "plugins.sample".
This patch provides the full package context to __import__() to mimic the standard "import" statement behavior.
[1] http://docs.python.org/2/library/functions.html#__import__ [2] http://hg.python.org/cpython/file/990d7647ea51/Python/import.c [3] http://docs.python.org/2/reference/simple_stmts.html#import
Signed-off-by: Zhou Zheng Sheng <zhshzhou@linux.vnet.ibm.com> --- src/kimchi/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/kimchi/utils.py b/src/kimchi/utils.py index f7eda93..b6d84fd 100644 --- a/src/kimchi/utils.py +++ b/src/kimchi/utils.py @@ -83,4 +83,4 @@ def import_class(class_path):
def import_module(module_name): - return __import__(module_name, fromlist=['']) + return __import__(module_name, globals(), locals(), [''])
participants (4)
-
Aline Manera -
Mark Wu -
Sheldon -
Zhou Zheng Sheng