Python packages and plugins as namespace packages
A user of
wrote a Crystal handler for their own use-case.
They asked on the Gitter channel
if we could allow to load external handlers, so they don't have to fork
the project and install the fork, but rather just install their
lightweight package containing just the handler.
We both saw Python namespace packages as good candidates, so I experimented a bit, and here are my conclusions.
Based on the documentation, there are 3 ways of creating a Python namespace package:
- native namespace packages
- pkgutil-style namespace packages
- pkg-resources-style namespace packages
I only considered the first two options (I'll let you read the docs to see why).
Native namespace packages¤
Native namespace packages are Python 3 compatible only. Good, I don't support Python 2 in my projects.
To write a native namespace package, you just have to do one thing:
my_namespace_package/ my_module_a.py my_namespace_package/ my_module_b.py
Install both packages and you will be able to import
If you want to provide deeper modules, you will have to drop
__init__.py files along the way:
# main package package/ level1/ level2/ builtin_module.py subpackage/ __init__.py sub.py a.py # plugin package/ level1/ level2/ plugin_module.py
Notice how we don't have any
This is required for the plugin package to add modules there.
In this example,
subpackage has an
it means that our plugin package could not add a module into it.
pkgutil-style namespace packages¤
pkgutil way is compatible with Python 2 and Python 3.
It allows to keep
# main package package/ __init__.py level1/ __init__.py level2/ __init__.py builtin_module.py # plugin package/ __init__.py level1/ __init__.py level2/ __init__.py plugin_module.py
__init__.py must contain this code:
from pkgutil import extend_path __path__ = extend_path(__path__, __name__)
Basically, it's a bit like the native way,
except that you keep your
but they must be identical across all packages!
__init__.py modules of the plugin will overwrite the
__init__.py modules of the main package. If they are not
identical, things will break (mostly imports).
So, with the native way, you cannot write code in
because those modules must not exist,
and with the
pkgutil way you must not write code in
because you don't want to have users duplicate it in their own.
At this state, I prefer the native way, as it's less files to write.
What I would have liked¤
Maybe namespace packages are not the best option for plugins.
But I would have loved being able to write my main package normally,
__init__.py, and as much code in those as I want,
and write the plugin package without any
__init__.py, so as to
"plug" (i.e. merge) additional modules into the main package:
# main package package/ __init__.py level1/ __init__.py level2/ __init__.py builtin_module.py # plugin package/ level1/ level2/ plugin_module.py
The order of installation would not matter of course, as merging the main package into the plugin one, or the opposite, would result in the same final package on the disk:
# main package package/ __init__.py level1/ __init__.py level2/ __init__.py builtin_module.py plugin_module.py
The main package author doesn't even have to think about plugins
and prepare the field by removing
__init__.py where it is needed.
Instead, they just write their package normally.
Then users can write a namespace package, mimicking the main package
structure, but without
__init__.py files, and BOOM, they
successfully wrote a plugin!
Native namespace packages look like they are trying to be implicit, but in my taste they are still not implicit enough.
Also, merging namespace packages into packages of the same name would allow easy patching of projects! Something not quite working like you want to? Quickly create a namespace package with just the patched module, and list it as a dependency.
What do you use to allow plugins in your Python projects ?