Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

When Should You Consider Meta Architectures


Published on

There are times when a system must adapt to new requirements within months or weeks rather than years. These new requirements can include complicated rules and new products or services the organization needs to scale to support. As the system scales up and becomes more complicated it can become very hard to adapt quickly to these changing requirements. In fact, the system can even become harder to change and slower to accept new requirements. This can lead to a desired architecture that is designed to scale allowing the system to more easily adapt to changing requirements.
Do you have the goal of building a system that can be extended and adapted without programmer intervention? Do you have the itch to explore meta-programming, not just because it is cool or complicated, but because you want your system's behavior or domain representations to scale? If so, consider learning about the design of systems that represent user-defined behavior specifications as metadata.
Architects create Adaptive Object-Model systems when they want to enable end-user programmers to adapt their system's behavior and they don't want developers to become the bottleneck. But what does it take to build a system that can be changed and adapted without programming? How do Adaptive Object-Models differ from little languages or DSLs and when is it appropriate to consider stepping into the meta world to build such an extensible system? This talk presents the basic ideas about meta-architecture such as an Adaptive Object-Model architecture and shares experiences that the presenters have had building and tuning various production meta-architecture systems that have enabled organizations to scale them more easily and without programming wizardry.

Published in: Technology
  • I have seen some practical application and need for this meta model. Great article. I guess only folks who have delivered or in the midst of such changing requirements and dynamics will be able to understand the need of this meta model. However, with Big Data becoming more and more important, this will be become a framework that will grow rapidly.
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

When Should You Consider Meta Architectures

  1. 1. Joseph W. Yoder & Rebecca Wirfs-Brock When Should You Consider Meta-Architectures? Joseph W. Yoder Copyright 2011, Joseph W. Yoder & The Refactory, Inc. and Wirfs-Brock Associates Agenda Ways to evolve systems What meta really allows The difference between meta- architectures and DSLs Adaptive Object-Model (AOM) basics Case studies of systems that scale Evolved from a talk given with Rebecca Wirfs-Brock at QCon 2010 Slide - 2Using Meta to Scale Page - 1
  2. 2. Joseph W. Yoder & Rebecca Wirfs-Brock Contrast: Two Ways to Evolve a Working System Traditional SW Meta-Data Based Architecture Architecture Who implements a A programmer. A domain expert might, if it is requirements change? a domain object or business rule change. How are changes verified? Programmer writes tests, QA Still have to run all unit and writes and runs acceptance acceptance tests. But can also tests, end-user approves. build into end-user tool checks that model changes won’t ―break‖ the existing working system. How often can the system be At the end of an iteration. Whenever changes are updated in production? verified. Not tied to dev cycle. What’s the big deal? Still have to go through some Significant changes can be release cycle. Programming made by end-users. Releasing and deployment can become can be simply updating bottlenecks. production metadata. Slide - 3 Stuart Brand’s Shearing Layers Buildings are a set of components that evolve in different timescales. Layers: site, structure, skin, services, space plan, stuff. Each layer has its own value, and speed of change (pace). Buildings adapt because faster layers (services) are not obstructed by slower ones (structure). —Stuart Brand, How Buildings Learn Slide - 4Using Meta to Scale Page - 2
  3. 3. Joseph W. Yoder & Rebecca Wirfs-Brock Yoder and Foote’s Software Shearing Layers ―Factor your system so that artifacts that change at similar rates are together.‖—Foote & Yoder, Ball of Mud, PLoPD4 Layers Slower  The platform  Infrastructure  Data schema  Standard frameworks, components, and modules  Abstract classes and interfaces  Classes  Code  Data Faster Slide - 5 Our Design Values Respect your system’s shearing layers.  Understand the rates of what changes. Determine who should be able to make changes, when, and at what cost.  Support them! Make what is too difficult, time consuming, or tedious easier.  Create tools, leverage design patterns, use data to drive behavior… Don’t overdesign!!! Slide - 6Using Meta to Scale Page - 3
  4. 4. Joseph W. Yoder & Rebecca Wirfs-Brock A Disclaimer This talk is not about  Using metadata to generate code  Building a ―stack‖ of meta-models and/or transforming between various models This talk focuses on practical techniques and cases studies for using metadata to scale solutions Slide - 7 What do we want to scale? Increased ✗   performance  throughput  adaptability/flexibility ✔  rate of supporting new requirements  time to deploy  support for configuration options  Reduced code bulk  Higher-leverage code (frameworks, tools…) Slide - 8Using Meta to Scale Page - 4
  5. 5. Joseph W. Yoder & Rebecca Wirfs-Brock Alternative Computation Styles Imperative Declarative Traditional Executable programming specifications Define a series of Many options: steps with  DSLs (Special language) conditionals and loops  Decision tables State machines to vary sequence   Production rule systems  Adaptive Model Systems Domain Specific Languages, Martin Fowler and Rebecca Parsons Slide - 9 The Power of Metadata Code is data, data is code. Everything is data. And data can drive behavior. Meta data simply describes other data. ―If something is going to vary in a predictable way, store the description of the variation in a database so that it is easy to change‖—Ralph Johnson Slide - 10Using Meta to Scale Page - 5
  6. 6. Joseph W. Yoder & Rebecca Wirfs-Brock Architecture of DSL Processing optional ✗ __________ __________ __________ __________ __________ __________ ______ __________ __________ ______ __________ _________ __________ __________ DSL Script parse generate DSL Script semantic target code model We focus on a virtual machine to interpret the semantic model! Domain Specific Languages, Martin Fowler and Rebecca Parsons Slide - 11 What’s a Semantic Model? Syntax describes ―legal‖ expressions. Semantics is what it means; what it does when it executes. Could be:  A representation, such as an in-memory object model or an adaptive object model  A data structure with behavior coming from functions acting on the data  Production rules and a rules ―engine‖  A decision table Slide - 12Using Meta to Scale Page - 6
  7. 7. Joseph W. Yoder & Rebecca Wirfs-Brock Domain Specific Languages Simple DSLs might be implemented via scripting languages and parameterization. They may not be interpreted. Sometimes they can be compiled directly to executable code. AOMs can be a Domain Specific Language but don’t have to be and vice-versa Slide - 13 Motivation: Need to Quickly Adapt to Change Business Rules or Domain Elements are changing quickly:  New calculations for insurance policies and new types of policies offered  Online store catalog with new products and services and rules applying to them  New cell phone product and services… Need quick ways to develop and adapt to these changing requirements. Slide - 14Using Meta to Scale Page - 7
  8. 8. Joseph W. Yoder & Rebecca Wirfs-Brock Adaptive Object-Model Mechanics  Meta information about the domain model and business rules is interpreted at runtime.  Domain classes, attributes, behaviors and relationships described by metadata  Instance-based  Object-Model stored in a database or in files and interpreted (can be XML/XMI).  When you change the metadata, system behavior changes.  Domain experts typically make changes, not programmers. Slide - 15 AOM ―Virtual Machine‖ Application Persistence Mechanism A O Database M A Metadata r Domain Repository/Namespace c Objects h I t e c t Interpreter/ u XML Parser Builder r e XML Slide - 16Using Meta to Scale Page - 8
  9. 9. Joseph W. Yoder & Rebecca Wirfs-Brock Architecture Elements of Adaptive Object Models • Metadata • Strategy/RuleObjects • TypeObject • Entity-Relationship • Properties • Interpreters/Builders • Type Square • Editors/GUIs If you want something to change quickly, push it into data and build tools geared towards changemakers’ needs. Slide - 17 AOM and TypeSquare Entity EntityType Constrains 0..n type possible properties for an entity properties 0..n Constrains 0..n properties type of a property PropertyType Property 0..n type -name : String -type : Type Slide - 18Using Meta to Scale Page - 9
  10. 10. Joseph W. Yoder & Rebecca Wirfs-Brock Type-Object PLoPD3 - Johnson and Woolf Before After Solution: Factor Symptom: Explosion of classes based on common attributes minor attribute differences into ―type‖ classes Slide - 19 Type Square (instance diagram) aPropertyType anEntityType aPropertyType <RecSpeed (int)> <vinylAudioRec> <AudMedia(Str)> aProperty anAccountabilityType anEntity aProperty name <45> <PARENT> Joe’s Garage name <SuperFid> Instance-Based!!! Slide - 20Using Meta to Scale Page - 10
  11. 11. Joseph W. Yoder & Rebecca Wirfs-Brock AOM and TypeSquare (code example) public class EntityType { private string _typeName; private Type _type; public EntityType (string name, Type type) { _typeName = name; _type = type; } public string Name { get{return _typeName;} set{_typeName = value;} }…} Christopher G. Lasater Slide - 21 AOM Support for Different Types of Behavior If behavior varies, specify rules in meta- data and interpret. Typical behaviors:  Constraints on values, relationships, state changes…(discrete values, ranges, permissible associations)  Functional  Workflow  Event based Slide - 22Using Meta to Scale Page - 11
  12. 12. Joseph W. Yoder & Rebecca Wirfs-Brock Adaptive Object Model ―Core Domain Architecture‖ An AOM Semantic Model is Instance- -children * * -children 1 Classes with Based -type Attributes and Accountability AccountabilityType 1 1 1 Relationships 1 -accountabilities 1 -accountabilitieTypes Behavior -children ** -children * * -rules * Entity -type EntityType Rule 1 1 1 1 1 +valueUsing:() 1 1 -attributeTypes -attributes * * * 1 * Attribute AttributeType Constant CompositeRule +name +value -type +type 1 * 1 +valueUsing:() * Operational Knowledge (meta) TableLookup BinaryOperation * Slide - 23 Adaptive Object Model ―Describing Your Entities‖ <entities> <entity name="President" entityType="TopLevelExecutiveType" actualAttributes="EmployeeId" /> <entity name="VicePresident" entityType="ExecutiveType" actualAttributes="EmployeeId" /> …. </entities> Slide - 24Using Meta to Scale Page - 12
  13. 13. Joseph W. Yoder & Rebecca Wirfs-Brock Adaptive Object Model ―Describing Your Relationships‖ <accountibilities> <!-- Hiring //--> <accountability name="PresidentHiresVicePresident" accountabilityType="TopLevelExecutiveHiresExecutive" actualParent="President" actualChildren="VicePresident" /> <accountability name="VicePresidentHiresSalesManager" accountabilityType="ExecutiveHiresManager" actualParent="VicePresident" actualChildren="SalesManager" /> … </accountibilities> Slide - 25 Scaling Examples Industrial (Manufacturing) Medical (IDPH) Insurance Telephony Billing System Pontis Telephony Marketing Portugal System Slide - 26Using Meta to Scale Page - 13
  14. 14. Joseph W. Yoder & Rebecca Wirfs-Brock Manufacturing - Shared Corporate Organization Data ~1,000,000 lines of COBOL code for editing organization data. Organization data shared by many systems. Had a common UI editor for adding, and updating this data. Editor was cloned and modified many times…grew to 100+ copies. Documentation and maintenance cumbersome. Slide - 27 Manufacturing - Shared Corporate Organization Data Rewrote in Java code ~ 40k LOC that used XML to describe the mappings to the organization data and generated the GUI and SQL queries. Documentation could be generated easily from XML mappings. Easy to change and release new versions for new tables through updates to the XML schema. Slide - 28Using Meta to Scale Page - 14
  15. 15. Joseph W. Yoder & Rebecca Wirfs-Brock User Defined Product (insurance example) Wirfs-Brock’s * Component Policy New types of Policies require new classes and attributes…and changes to rules require updates to methods Slide - 29 User Defined Product (insurance example) -children * * -children 1 -rules * Component -type ComponentType Rule 1 1 1 1 +valueUsing:() 1 1 -attributes -attributes * * * 1 * Attribute AttributeType Constant CompositeRule +name +value -type +type 1 * 1 +valueUsing:() * TableLookup BinaryOperation * Core Architecture is TypeSquare Could quickly adapt to new rules within weeks or less Slide - 30Using Meta to Scale Page - 15
  16. 16. Joseph W. Yoder & Rebecca Wirfs-Brock Objectiva Business Entities (telephony billing system example) Originally took a few person years to build. AOM implementation reduced this to few person months Slide - 31 Objectiva Attributes (telephony example) Slide - 32Using Meta to Scale Page - 16
  17. 17. Joseph W. Yoder & Rebecca Wirfs-Brock Objectiva Entity Relationships (telephony example) Slide - 33 How Objectiva Scaled Initial AOM implementation could handle ~1 million transactions per day Not enough for larger installations…  So, performance was tackled by adding caching and other point optimizations….increasing throughput to over 10 million transactions  Lesson: Meta Architectures can be tweaked for performance…nothing magic here Slide - 34Using Meta to Scale Page - 17
  18. 18. Joseph W. Yoder & Rebecca Wirfs-Brock Refugee and Newborn Screening Example Doctor, HealthProf. Hospital, Lab Blood Specimen Mother, Infant Slide - 35 Screening Refugees Slide - 36Using Meta to Scale Page - 18
  19. 19. Joseph W. Yoder & Rebecca Wirfs-Brock Medical Observation – Basic OO Design Model Person * Observation Measurement Trait Quantity traitValue unit convertTo: amount expressOnUnit: expressOnUnits: PhysicalMeasure Blood EyeColor HairColor Gender … Height Weight FeedingType Hearing Vision … What happens when a new observation is required? Had well over 100 observations! Slide - 37 Medical Observations Design 1..* type contDescr 1 Entities guard 1 Validator validatorName ObservationType 1..* varType PartyType isValid: obsValue phenomenonType 1 descriptor isValid: obsValue 1 descriptor 1..* 1..* elements Party DiscreteValidator RangedValidator NullValidator descriptorSet intervalSet PrimitiveObservation CompositeObservation validUnit Type Type properties 1..* 1..* Observation Behavior instance recordedDate comments elements 1..* isValid Quantity unit value 1..* quantity CompositeObservation Primitive Observation observationValue convertTo: Knowledge level Operational level dvalue 1..* DiscreteValues Attributes Modelled100s of Entities with this core architecture. Can add new entities or make changes to entities very quickly. Slide - 38Using Meta to Scale Page - 19
  20. 20. Joseph W. Yoder & Rebecca Wirfs-Brock PONTIS AOM ENGINEERS Telephony Marketing System Highly Flexible Slide - 39 PONTIS WORKFLOW Developer AOM Engineer ―Adaptive Object-Model Metadata Evolver‖, PLoP 2010, Atzmon Hen-Tov, David H. Lorenz, Lena Nikolaev, Lior Schachter, Rebecca Wirfs-Brock, Joseph Yoder Slide - 40Using Meta to Scale Page - 20
  21. 21. Joseph W. Yoder & Rebecca Wirfs-Brock How the Pontis System Scales Reduced deployment from 1 year to 1 month Updates every couple of weeks Deployed to over 30 clients Code base much smaller 500,000 LOC vs 3,000,000 (previous implementation) Fastest growing business award  5,500 percent within five years Slide - 41 Portuguese Historical Artifacts Catalogue Catalogue of historical artifacts Mapping of site artifacts, relationships over time, historical site evolution 2 years churning on requirements Technical team finally made the decision to support easy ways to change the domain…metadata-driven software 1. Hugo Sereno Ferreira, Filipe Figueiredo Correia, Ademar Aguiar, and João Pascoal Faria. "Adaptive Object-Models: a Research Roadmap". International Journal On Advances in Software, volume 3, numbers 1 and 2, 2010. ISSN: 1942-2628. 2. Hugo Sereno Ferreia, Filipe Figueiredo Correia, Ademar Aguiar. "Design for an Adaptive Object-Model Framework: An Overview". Proceedings of the 4th International Workshop on Models@run.time. Co-located with the 12th International Conference on Model Driven Engineering Languages and Systems. Denver, Colorado. USA. 3. Hugo Sereno Ferreira, Ademar Aguiar, João Pascoal Faria. "Adaptive Object Modelling: Patterns, Tools and Applications". 3rd Symposium Slide - 42 on Doctoral Students of Software Engineering. Proceedings of the 4th International Conference on Software Engineering Advances. Porto. Portugal.Using Meta to Scale Page - 21
  22. 22. Joseph W. Yoder & Rebecca Wirfs-Brock Project Timeline: Growth of the System metadata 4 months to first deployment features code added rapidly at project end Slide - 43 Project Timeline: Release Frequency Once customers realized they could make model changes, they made lots of changes. The rate of deployment of new releases increased, too. Slide - 44Using Meta to Scale Page - 22
  23. 23. Joseph W. Yoder & Rebecca Wirfs-Brock Where Metadata Techniques Have been Successfully Used: Representing Insurance Policies Telephone Billing and Back Office Systems Workflow Systems Medical Observations Banking and Trading Validate Equipment Configuration Documents Management System Gauge Calibration Systems Simulation Software Invoicing Systems (some can be found in papers) Manufacturing Systems … Slide - 45 Some Disadvantages of Metadata Based Architectures Requires infrastructure for storing, building, and interpreting metadata. Requires strong design skills to create. Interpreting metadata may result in lower performance (this can be remedied by well-known techniques). Slide - 46Using Meta to Scale Page - 23
  24. 24. Joseph W. Yoder & Rebecca Wirfs-Brock Agile Best Practices for Creating Metadata Solutions Separate what changes quickly from what changes slowly (hot-spots). Develop iteratively and incrementally. Get feedback early and often. Add flexibility using meta-data techniques only when and where needed. Write tests for expected system behaviors and the meta tools/interpreters. Slide - 47 Summary Meta-architectures enable rapid changes to domain objects and rules Core application code less than with conventional programming Scale in two dimensions:  support for rapid requirements changes  design leverage (team size ~10^1) Slide - 48Using Meta to Scale Page - 24
  25. 25. Joseph W. Yoder & Rebecca Wirfs-Brock Meta Collaborators Ademar Aguiar Atzmon Hen-Tov Francis Anderson Ralph Johnson Ali Arsanjani David H. Lorenz Jean Bezivin Lena Nikolaev Paulo Borba Jeff Oaks Filipe Correia Reza Razavi Krzysztof Czarnecki Nicolas Revault Ayla Dantas Dirk Riehle Martine Devos Lior Schachter Hugo Ferreira Dave Thomas Brian Foote Michel Tilman Martin Fowler Leon Welicki Richard Gabriel Rebecca Wirfs-Brock…Slide - 49 Resources Adaptive Object Models  Agile Software  Refactoring  Agile Alliance:  The Agile Manifesto  12 Principles of Agile Development Slide - 50Using Meta to Scale Page - 25
  26. 26. Joseph W. Yoder & Rebecca Wirfs-Brock Obrigado! That’s All Twitter: @metayoda Twitter: @rebeccawb Slide - 51Using Meta to Scale Page - 26