Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Import this, that, and the other thing: custom importers


Published on

Mr. Brett Cannon
in PyCon2010-USA-Atlanta
45min ◊◊◊ Advanced
Friday 11:45am, Centennial I
categories: core
Since Python 2.3, the ability has existed to customize the import process so that one can support code stored in alternative formats, e.g. zipimport and its support of importing code from a zip file. The aim of this talk is to make sure you understand how the custom import mechanism works and how to write your own custom importers using importlib from Python 3.1.

Published in: Technology
  • Be the first to comment

Import this, that, and the other thing: custom importers

  1. 1. If you do not know what __path__ is, this talk is NOT for you. Sorry.
  2. 2. import this, that, and the other thing Custom importers in Python Brett Cannon brett@python.orgSlides are sparse, so do listen to what I say.
  3. 3. Thanks ...• Python Software Foundation • PyCon Financial Aid committee• Nasuni • Jesse Noller
  4. 4. What the heck is an importer?Relevant since Python 2.3
  5. 5. importer =finder + loader
  6. 6. A finder finds modules.
  7. 7. A loader loads modules.
  8. 8. “Why do I want one?”Customization/control, easier to work w/ than __import__
  9. 9. How are custom importers used by import?Simplified view; ignoring implicit importers
  10. 10. Meta path sys.meta_path
  11. 11. Start for finder in sys.meta_path: ... False loader = finder.find_module(name, path) True return loader.load_module(name)What ‘path’ arg is
  12. 12. Pathsys.path or __path__, sys.path_hooks, & sys.path_importer_cache
  13. 13. ... Parent module has __path__ False search = sys.path search True search = parents __path__
  14. 14. Search for entry in search: raise ImportError finder = sys.path_importer_cache path [entry] False hook finderFalse True loader = finder.find_module(name) True return loader.load_module(name)
  15. 15. path hook for hook in sys.path_importer_cache[entry] = dummy sys.path_hooks: False finder = hook(entry) True sys.path_importer_cache[entry] = finder finderTrue/False = ImportError (not) raised
  16. 16. how do I write my own importer? Only masochists need apply.
  17. 17. Option 1: Painfully from scratchRead PEP 302 for the gory details.
  18. 18. Option 2:Use importlib Available since Python 3.1.I have suffered so you don’t have to.
  19. 19. Option 3: importers File path abstraction on top of importlib. Treating as purgatory for importlib inclusion.If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
  20. 20. Using a zipfile importer as an exampleAssuming use of importlib.Talking from perspective of using an archive.
  21. 21. we need a hook For sys.path_hooks.
  22. 22. Refresher: Hooks look for a finder for a pathPath either from sys.path or __path__
  23. 23. Hooks can get funky paths E.g. /path/to/file/ backwards looking for a file; find a directory then you have gone too far.
  24. 24. Consider caching archive file objectsNo need to keep 3 connection objects open for the same sqlite3 file
  25. 25. Pass your finder the “location”: 1)the path/object & 2) the package pathImport assumes you are looking in a part of a package.
  26. 26. Raise ImportErrorif you got nuthin’
  27. 27. Have finder, will look for code
  28. 28. Don’t treat modules as code but as filesJust trust me. Too many people/code make this assumption already for stuff like __file__,__path__, etc.
  29. 29. You did remember where in the package you are looking, RIGHT?!?Needed because of __path__ manipulation by user code.
  30. 30. fullname.rpartition(‘.’)[-1]
  31. 31. Need to care about packages & modules some/pkg/name/ and some/pkg/name.pyCare about bytecode if you want.Notice how many stat calls this takes?
  32. 32. Avoid cachingwithin a finder Blame sys.path_importer_cache
  33. 33. Tell the loader if package & path to codeDon’t Repeat Yourself ... within reason.
  34. 34. Nuthin’?Give back None
  35. 35. Now it gets tricky Writing a loader.
  36. 36. Are you stillthinking in terms of file paths?
  37. 37. • source_path() • Might be changing... • is_package() • get_data()Everything in terms of exactly what it takes to import source
  38. 38. • source_path() • is_package() • get_data() • source_mtime() • bytecode_path() • Might be changing...This is what is needed to get source w/ bytecode right
  39. 39. Reasons to ignore .pyc• Jython, IronPython couldn’t care less. • Safe to support, though.• Another thing to code up. • Bytecode is just an optimization. • If you only ship .pyc for code protection, stop it.
  40. 40. What to do when usingimportlib ABCs
  41. 41. Require anchor point for paths somewhere/ is too ambiguousToo hazy as to where a relative path is anchored; archive? Package location?
  42. 42. Consider caching stat calls Only for stand-alone loaders! Also consider caching if package or not.Consider whether storage is read-only, append-only, or read-write.
  43. 43. Don’t overdoerror checking EAFP is your friend.
  44. 44. Perk of importers is the abstraction
  45. 45. Lazy loader mix-in written in 19 lines
  46. 46. class Module(types.ModuleType): passclass Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return moduleclass LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
  47. 47. ... or you could usethe importers package
  48. 48. Fin