• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Import this, that, and the other thing: custom importers
 

Import this, that, and the other thing: custom importers

on

  • 1,234 views

Mr. Brett Cannon...

Mr. Brett Cannon
in PyCon2010-USA-Atlanta
45min ◊◊◊ Advanced
Friday 11:45am, Centennial I
categories: core
Since Python 2.3, the ability has existed to customize the import process so that one can support code stored in alternative formats, e.g. zipimport and its support of importing code from a zip file. The aim of this talk is to make sure you understand how the custom import mechanism works and how to write your own custom importers using importlib from Python 3.1.

Statistics

Views

Total Views
1,234
Views on SlideShare
1,234
Embed Views
0

Actions

Likes
5
Downloads
17
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Import this, that, and the other thing: custom importers Import this, that, and the other thing: custom importers Presentation Transcript

    • If you do not know what __path__ is, this talk is NOT for you. Sorry.
    • import this, that, and the other thing Custom importers in Python Brett Cannon www.DrBrett.ca brett@python.orgSlides are sparse, so do listen to what I say.
    • Thanks ...• Python Software Foundation • PyCon Financial Aid committee• Nasuni • Jesse Noller
    • What the heck is an importer?Relevant since Python 2.3
    • importer =finder + loader
    • A finder finds modules.
    • A loader loads modules.
    • “Why do I want one?”Customization/control, easier to work w/ than __import__
    • How are custom importers used by import?Simplified view; ignoring implicit importers
    • Meta path sys.meta_path
    • Start for finder in sys.meta_path: ... False loader = finder.find_module(name, path) True return loader.load_module(name)What ‘path’ arg is
    • Pathsys.path or __path__, sys.path_hooks, & sys.path_importer_cache
    • ... Parent module has __path__ False search = sys.path search True search = parents __path__
    • Search for entry in search: raise ImportError finder = sys.path_importer_cache path [entry] False hook finderFalse True loader = finder.find_module(name) True return loader.load_module(name)
    • path hook for hook in sys.path_importer_cache[entry] = dummy sys.path_hooks: False finder = hook(entry) True sys.path_importer_cache[entry] = finder finderTrue/False = ImportError (not) raised
    • how do I write my own importer? Only masochists need apply.
    • Option 1: Painfully from scratchRead PEP 302 for the gory details.
    • Option 2:Use importlib Available since Python 3.1.I have suffered so you don’t have to.
    • Option 3: importers http://packages.python.org/importers/ File path abstraction on top of importlib. Treating as purgatory for importlib inclusion.If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
    • Using a zipfile importer as an exampleAssuming use of importlib.Talking from perspective of using an archive.
    • we need a hook For sys.path_hooks.
    • Refresher: Hooks look for a finder for a pathPath either from sys.path or __path__
    • Hooks can get funky paths E.g. /path/to/file/code.zip/some/pkgSearch backwards looking for a file; find a directory then you have gone too far.
    • Consider caching archive file objectsNo need to keep 3 connection objects open for the same sqlite3 file
    • Pass your finder the “location”: 1)the path/object & 2) the package pathImport assumes you are looking in a part of a package.
    • Raise ImportErrorif you got nuthin’
    • Have finder, will look for code
    • Don’t treat modules as code but as filesJust trust me. Too many people/code make this assumption already for stuff like __file__,__path__, etc.
    • You did remember where in the package you are looking, RIGHT?!?Needed because of __path__ manipulation by user code.
    • fullname.rpartition(‘.’)[-1]
    • Need to care about packages & modules some/pkg/name/__init__.py and some/pkg/name.pyCare about bytecode if you want.Notice how many stat calls this takes?
    • Avoid cachingwithin a finder Blame sys.path_importer_cache
    • Tell the loader if package & path to codeDon’t Repeat Yourself ... within reason.
    • Nuthin’?Give back None
    • Now it gets tricky Writing a loader.
    • Are you stillthinking in terms of file paths?
    • importlib.abc.PyLoader • source_path() • Might be changing... • is_package() • get_data()Everything in terms of exactly what it takes to import source
    • importlib.abc.PyPycLoader • source_path() • is_package() • get_data() • source_mtime() • bytecode_path() • Might be changing...This is what is needed to get source w/ bytecode right
    • Reasons to ignore .pyc• Jython, IronPython couldn’t care less. • Safe to support, though.• Another thing to code up. • Bytecode is just an optimization. • If you only ship .pyc for code protection, stop it.
    • What to do when usingimportlib ABCs
    • Require anchor point for paths somewhere/mod.py is too ambiguousToo hazy as to where a relative path is anchored; archive? Package location?
    • Consider caching stat calls Only for stand-alone loaders! Also consider caching if package or not.Consider whether storage is read-only, append-only, or read-write.
    • Don’t overdoerror checking EAFP is your friend.
    • Perk of importers is the abstraction
    • Lazy loader mix-in written in 19 lines
    • class Module(types.ModuleType): passclass Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return moduleclass LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
    • ... or you could usethe importers package http://packages.python.org/importers/
    • Fin