If you do not know what __path__ is,    this talk is   NOT for you.                   Sorry.
import this,                  that, and the                   other thing                     Custom importers in Python  ...
Thanks ...• Python Software Foundation • PyCon Financial Aid committee• Nasuni • Jesse Noller
What the heck is an             importer?Relevant since Python 2.3
importer       =finder + loader
A finder finds   modules.
A loader loads   modules.
“Why do I want one?”Customization/control, easier to work w/ than __import__
How are custom             importers used by                  import?Simplified view; ignoring implicit importers
Meta path sys.meta_path
Start                          for finder in sys.meta_path:        ...      False                     loader = finder.find_mo...
Pathsys.path or __path__, sys.path_hooks,      & sys.path_importer_cache
...   Parent module      has __path__                             False       search = sys.path                           ...
Search              for entry in search:          raise ImportError        finder = sys.path_importer_cache               p...
path                       hook                    for hook in                                               sys.path_impo...
how do I write   my own  importer? Only masochists need apply.
Option 1: Painfully from     scratchRead PEP 302 for the gory details.
Option 2:Use importlib     Available since Python 3.1.I have suffered so you don’t have to.
Option 3:                         importers               http://packages.python.org/importers/           File path abstra...
Using a                   zipfile importer                    as an exampleAssuming use of importlib.Talking from perspect...
we need a hook   For sys.path_hooks.
Refresher:              Hooks look for a              finder for a pathPath either from sys.path or __path__
Hooks can get                funky paths            E.g. /path/to/file/code.zip/some/pkgSearch backwards looking for a file...
Consider caching         archive file objectsNo need to keep 3 connection objects open for the same sqlite3 file
Pass your finder            the “location”:          1)the path/object &          2) the package pathImport assumes you ar...
Raise ImportErrorif you got nuthin’
Have finder, will look for code
Don’t treat modules        as code but as filesJust trust me. Too many people/code make this assumption already for stuff ...
You did remember               where in the              package you are             looking, RIGHT?!?Needed because of __...
fullname.rpartition(‘.’)[-1]
Need to care        about packages &            modules                      some/pkg/name/__init__.py                    ...
Avoid cachingwithin a finder Blame sys.path_importer_cache
Tell the loader  if package &  path to codeDon’t Repeat Yourself ... within reason.
Nuthin’?Give back None
Now it gets  tricky  Writing a loader.
Are you stillthinking in terms of    file paths?
importlib.abc.PyLoader           • source_path()              • Might be changing...           • is_package()           • ...
importlib.abc.PyPycLoader           • source_path()           • is_package()           • get_data()           • source_mti...
Reasons to ignore .pyc• Jython, IronPython couldn’t care less. • Safe to support, though.• Another thing to code up. • Byt...
What to do when     usingimportlib ABCs
Require anchor            point for paths                 somewhere/mod.py is too ambiguousToo hazy as to where a relative...
Consider caching            stat calls                Only for stand-alone loaders!            Also consider caching if pa...
Don’t overdoerror checking   EAFP is your friend.
Perk of importers is  the abstraction
Lazy loader mix-in    written in     19 lines
class Module(types.ModuleType):    passclass Mixin:    def load_module(self, name):        if name in sys.modules:        ...
... or you could usethe importers package http://packages.python.org/importers/
Fin
Upcoming SlideShare
Loading in...5
×

Import this, that, and the other thing: custom importers

1,212

Published on

Mr. Brett Cannon
in PyCon2010-USA-Atlanta
45min ◊◊◊ Advanced
Friday 11:45am, Centennial I
categories: core
Since Python 2.3, the ability has existed to customize the import process so that one can support code stored in alternative formats, e.g. zipimport and its support of importing code from a zip file. The aim of this talk is to make sure you understand how the custom import mechanism works and how to write your own custom importers using importlib from Python 3.1.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,212
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
20
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Import this, that, and the other thing: custom importers

  1. 1. If you do not know what __path__ is, this talk is NOT for you. Sorry.
  2. 2. import this, that, and the other thing Custom importers in Python Brett Cannon www.DrBrett.ca brett@python.orgSlides are sparse, so do listen to what I say.
  3. 3. Thanks ...• Python Software Foundation • PyCon Financial Aid committee• Nasuni • Jesse Noller
  4. 4. What the heck is an importer?Relevant since Python 2.3
  5. 5. importer =finder + loader
  6. 6. A finder finds modules.
  7. 7. A loader loads modules.
  8. 8. “Why do I want one?”Customization/control, easier to work w/ than __import__
  9. 9. How are custom importers used by import?Simplified view; ignoring implicit importers
  10. 10. Meta path sys.meta_path
  11. 11. Start for finder in sys.meta_path: ... False loader = finder.find_module(name, path) True return loader.load_module(name)What ‘path’ arg is
  12. 12. Pathsys.path or __path__, sys.path_hooks, & sys.path_importer_cache
  13. 13. ... Parent module has __path__ False search = sys.path search True search = parents __path__
  14. 14. Search for entry in search: raise ImportError finder = sys.path_importer_cache path [entry] False hook finderFalse True loader = finder.find_module(name) True return loader.load_module(name)
  15. 15. path hook for hook in sys.path_importer_cache[entry] = dummy sys.path_hooks: False finder = hook(entry) True sys.path_importer_cache[entry] = finder finderTrue/False = ImportError (not) raised
  16. 16. how do I write my own importer? Only masochists need apply.
  17. 17. Option 1: Painfully from scratchRead PEP 302 for the gory details.
  18. 18. Option 2:Use importlib Available since Python 3.1.I have suffered so you don’t have to.
  19. 19. Option 3: importers http://packages.python.org/importers/ File path abstraction on top of importlib. Treating as purgatory for importlib inclusion.If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
  20. 20. Using a zipfile importer as an exampleAssuming use of importlib.Talking from perspective of using an archive.
  21. 21. we need a hook For sys.path_hooks.
  22. 22. Refresher: Hooks look for a finder for a pathPath either from sys.path or __path__
  23. 23. Hooks can get funky paths E.g. /path/to/file/code.zip/some/pkgSearch backwards looking for a file; find a directory then you have gone too far.
  24. 24. Consider caching archive file objectsNo need to keep 3 connection objects open for the same sqlite3 file
  25. 25. Pass your finder the “location”: 1)the path/object & 2) the package pathImport assumes you are looking in a part of a package.
  26. 26. Raise ImportErrorif you got nuthin’
  27. 27. Have finder, will look for code
  28. 28. Don’t treat modules as code but as filesJust trust me. Too many people/code make this assumption already for stuff like __file__,__path__, etc.
  29. 29. You did remember where in the package you are looking, RIGHT?!?Needed because of __path__ manipulation by user code.
  30. 30. fullname.rpartition(‘.’)[-1]
  31. 31. Need to care about packages & modules some/pkg/name/__init__.py and some/pkg/name.pyCare about bytecode if you want.Notice how many stat calls this takes?
  32. 32. Avoid cachingwithin a finder Blame sys.path_importer_cache
  33. 33. Tell the loader if package & path to codeDon’t Repeat Yourself ... within reason.
  34. 34. Nuthin’?Give back None
  35. 35. Now it gets tricky Writing a loader.
  36. 36. Are you stillthinking in terms of file paths?
  37. 37. importlib.abc.PyLoader • source_path() • Might be changing... • is_package() • get_data()Everything in terms of exactly what it takes to import source
  38. 38. importlib.abc.PyPycLoader • source_path() • is_package() • get_data() • source_mtime() • bytecode_path() • Might be changing...This is what is needed to get source w/ bytecode right
  39. 39. Reasons to ignore .pyc• Jython, IronPython couldn’t care less. • Safe to support, though.• Another thing to code up. • Bytecode is just an optimization. • If you only ship .pyc for code protection, stop it.
  40. 40. What to do when usingimportlib ABCs
  41. 41. Require anchor point for paths somewhere/mod.py is too ambiguousToo hazy as to where a relative path is anchored; archive? Package location?
  42. 42. Consider caching stat calls Only for stand-alone loaders! Also consider caching if package or not.Consider whether storage is read-only, append-only, or read-write.
  43. 43. Don’t overdoerror checking EAFP is your friend.
  44. 44. Perk of importers is the abstraction
  45. 45. Lazy loader mix-in written in 19 lines
  46. 46. class Module(types.ModuleType): passclass Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return moduleclass LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
  47. 47. ... or you could usethe importers package http://packages.python.org/importers/
  48. 48. Fin
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×