Successfully reported this slideshow.
Your SlideShare is downloading. ×

pyjamas22_ generic composite in python.pdf

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 23 Ad

More Related Content

Recently uploaded (20)

Advertisement

pyjamas22_ generic composite in python.pdf

  1. 1. © BST LABS 2022 ALL RIGHTS RESERVED Generic Composite in Python Rationale behind the pycomposite Open Source Library Asher Sterkin SVP Engineering, GM Pyjamas Conf 2022
  2. 2. © BST LABS 2022 ALL RIGHTS RESERVED ● A 40-year industry veteran specializing in software architecture and technology ● SVP Engineering @ BST LABS, BlackSwan Technologies ● Former VP Technology @ NDS and Distinguished Engineer @ Cisco ● Initiator and chief technology lead of the Cloud AI Operating System project ● Areas of interest: ○ Serverless Cloud Architecture ○ (Strategic) Domain-Driven Design ○ Promise Theory and Semantic SpaceTimes ○ Complex Adaptive Systems ● Contacts: ○ Linkedin: https://www.linkedin.com/in/asher-sterkin-10a1063 ○ Twitter: https://www.twitter.com/asterkin ○ Email: asher.sterkin@gmail.com ○ Slideshare: https://www.slideshare.net/AsherSterkin ○ Medium: https://asher-sterkin.medium.com/ Who am I?
  3. 3. © BST LABS 2022 ALL RIGHTS RESERVED ● Design Patterns ● The “Composite” Design Pattern ● Need for a Generic One ● Design Patterns Density ● Implementation ● More Advanced Case Study ● Things to Remember Table of Contents
  4. 4. © BST LABS 2022 ALL RIGHTS RESERVED The Wikipedia definition: “In software engineering, a software design pattern is a general, reusable solution to a commonly occurring problem within a given context in software design.” Quality Without a Name (QWAN): “To seek the timeless way we must first know the quality without a name. There is a central quality which is the root criterion of life and spirit in a man, a town, a building, or a wilderness. This quality is objective and precise, but it cannot be named.” Design Patterns A good collection of Design Patterns in Python: https://github.com/faif/python-patterns Fundamental, though a bit outdated, with examples in C++ and Java Origin of the idea of Design Patterns
  5. 5. © BST LABS 2022 ALL RIGHTS RESERVED Whenever we have a tree-like data structure, be it a file system, cloud stack resources, code, or anything composed from smaller parts, the Composite Design Pattern is the first candidate to consider. “Composite” Design Pattern
  6. 6. © BST LABS 2022 ALL RIGHTS RESERVED Composite Design Pattern (from Wikipedia) Traditional OO Implementation
  7. 7. © BST LABS 2022 ALL RIGHTS RESERVED ● The Composite class has to overload all the abstract interface methods. ● The implementation of Composite methods would most likely be trivial: ○ Go over all child components and invoke the corresponding methods ○ Could be Depth-First or Breadth-First traversal. ○ In most of cases nobody cares which one. ● Outcome of the Composite operation is an aggregation of the child operations. ● Generally, not a big deal if the number of methods is small Limitations of the Traditional OO Implementation
  8. 8. © BST LABS 2022 ALL RIGHTS RESERVED I F C services vendors APIs location acquire configure consume operate My Context: Cloud Infrastructure From Python Code service.py service_config.py
  9. 9. © BST LABS 2022 ALL RIGHTS RESERVED build_service_parameters() build_service_template_variables() build_service_conditions() build_service_outputs() build_service_resources() build_function_permissions() build_function_resources() build_function_resource_properties() build_function_environment() Core Mechanism: Service Template Builder Composite ResourceTemplateBuilder CompositeResourceTemplateBuilder * ● Every high-level feature (e.g. MySQL Database) has multiple sub-features: ○ Data on rest encryption: yes/no ○ Private network protection: yes/no ○ Interface: (e.g. db_connection vs SQLAlchemy) ○ Allocation: internal/external ○ Access: read-only/read-write ○ User authentication: yes/no ● Each sub-feature might have multiple flavours (e.g. type of encryption) ● Every feature or sub-feature requires one or more cloud resources ● Every service function has to be properly configured to get an access to eachy feature resources ● Every service function implements a particular API, which might bring its own resource feature(s) Service Template Builder Composite API Template Builder Composite API Resource Template Builder Feature Template Builder Composite Feature Resource Template Builder
  10. 10. © BST LABS 2022 ALL RIGHTS RESERVED ● The measurement of the amount of design that can be represented as instances of design patterns ● When applied correctly, higher design pattern density implies a higher maturity of the design. ● When applied incorrectly, leads to disastrous over-engineering. Design Pattern Density
  11. 11. © BST LABS 2022 ALL RIGHTS RESERVED build_service_parameters() build_service_template_variables() build_service_conditions() build_service_outputs() build_service_resources() build_function_permissions() build_function_resources() build_function_resource_properties() build_function_environment() <<null object>> <<builder>> ResourceTemplateBuilder Putting All Patterns Together * <<composite>> <<iterator>> CompositeResourceTemplateBuilder <<decorator>> composite(cls) <<factory method>> get_builder(config) XyzResourceTemplateBuilder <<specification>> XyzResourceSpec * For each method, define a default implementation that returns an empty structure <<decorator>> ResourceTemplateBuilderDecorator
  12. 12. © BST LABS 2022 ALL RIGHTS RESERVED Let’s Go to the Code @composite class CompositeResourceTemplateBuilder(ResourceTemplateBuilder): """ Implements a Composite Design Pattern to support combining multiple ResourceTemplateBuilders. By default, works as a PassThroughGate for underlying builder(s). """ pass class ResourceTemplateBuilderDecorator(ResourceTemplateBuilder): """ Base class for generic (e.g. Storage) ResourceTemplateBuilders selectively delegating specific tasks to platform-dependent builders. By default, works as a StopGate for the underlying builder. :param builder: reference to external platform-specific builder """ def __init__(self, builder: ResourceTemplateBuilder): self._builder = builder
  13. 13. © BST LABS 2022 ALL RIGHTS RESERVED composite class decorator def composite(cls: type) -> type: """ Generic class decorator to create a Composite from the original class. Notes: 1. the constructor does not make copy, so do not pass generators, if you plan to invoke more than one operation. 2. it will return always flattened results of any operation. :param cls: original class :return: Composite version of original class """ attrs = { n: _make_method(n, f) for n, f in getmembers(cls, predicate=isfunction) if not n.startswith("_") } attrs["__init__"] = _constructor composite_cls = type(cls.__name__, cls.__bases__, attrs) composite_cls.__iter__ = _make_iterator(composite_cls) return composite_cls
  14. 14. © BST LABS 2022 ALL RIGHTS RESERVED _constructor is simple def _constructor(self, *parts) -> None: self._parts = parts
  15. 15. © BST LABS 2022 ALL RIGHTS RESERVED _make_method is where the real stuff is done def _make_method(name: str, func: callable) -> callable: > def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha > def _make_foreach(m) -> callable: <-- Python Closure Gotcha rt_ = signature(func).return_annotation rt = get_origin(rt_) or rt_ # strip type annotation parameters like tuple[int, ...] if present return _make_foreach(name) if rt is None else _make_reduce(name, rt) # Top-down decomposition in action
  16. 16. © BST LABS 2022 ALL RIGHTS RESERVED _make_foreach is boring def _make_foreach(m) -> callable: def _foreach_parts(self, *args, **kwargs) -> callable: <-- Python Closure Gotcha # this is a member function, hence self # self is iterable, concrete functions invoked depth first for obj in self: getattr(obj, m)(*args, **kwargs) return _foreach_parts
  17. 17. © BST LABS 2022 ALL RIGHTS RESERVED _make_reduce is more interesting def _make_method(name: str, func: callable) -> callable: def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha init_value = rt() combine = add if rt in (int, str, tuple) else always_merger.merge def _reduce_parts(self, *args, **kwargs): # this is a member function, hence self # self is iterable, results come out flattened return reduce( lambda acc, obj: combine(acc, getattr(obj, m)(*args, **kwargs)), self, init_value ) return _reduce_parts
  18. 18. © BST LABS 2022 ALL RIGHTS RESERVED _make_iterator is the most tricky def _make_iterator(cls): def _iterator(self): # Simple depth-first composite Iterator # Recursive version did not work for some mysterious reason # This one proved to be more reliable # Credit: https://stackoverflow.com/questions/26145678/implementing-a-depth-first-tree-iterator-in-python stack = deque(self._parts) while stack: # Pop out the first element in the stack part = stack.popleft() if cls == type(part): # The same composite exactly stack.extendleft(reversed(part._parts)) elif isinstance(part, cls) or not isinstance(part, Iterable): yield part # derived classes presumably have overloads else: # Iterable stack.extendleft(reversed(part)) return _iterator
  19. 19. © BST LABS 2022 ALL RIGHTS RESERVED <<builder>> K8SManifestBuilder __call__(...) <<builder>> <<template method>> K8SManifestPartBuilder build_manifest_part(...) _build_manifest_part_data(...) __subclasses__() More Advanced Use Case: K8S Manifest Builder * <<composite>> <<iterator>> K8SManifestPartCompositeBuilder <<decorator>> composite(cls) <<factory method>> get_builders() K8SXYZManifestBuilder Even with one function, this approach proved to lead to better modularity and testability
  20. 20. © BST LABS 2022 ALL RIGHTS RESERVED I need ‘em all @composite class K8SManifestPartCompositeBuilder(K8SManifestPartBuilder): pass def get_builders(): for m in iter_modules(__path__): # ensure all builders are registered import_module(f'{__package__}.{m.name}') return K8SManifestPartCompositeBuilder( *(sb() for sb in K8SManifestPartBuilder.__subclasses__() if 'K8SManifestPartCompositeBuilder' not in sb.__name__) )
  21. 21. © BST LABS 2022 ALL RIGHTS RESERVED ● In this solution I did not implement the Visitor Design Pattern. While it is not hard to implement, I just did not have any real use case for it. If you do, drop me a line ● The current set of aggregation functions is a bit limited reflecting what I needed at the moment. Support for a larger set might come in the future. ● This presentation used a more advanced version of code, currently at the separate branch Concluding Remarks
  22. 22. © BST LABS 2022 ALL RIGHTS RESERVED ● Design Patterns are extremely powerful tools. ● Design Patterns work best in concert (high density). ● Composite Design Pattern is suitable for implementing the on/off logic a complex feature ● Also, for implementing a variable list of components you do not want to enumerate explicitly ● Python metaprogramming could make miracles. ● With more power comes more responsibility. ● An unjustified chase after advanced techniques is a sure road to hell. Things to Remember
  23. 23. © BST LABS 2022 ALL RIGHTS RESERVED Questions?

×