There are times when a system must adapt to new requirements within months or weeks rather than years. These new requirements can include complicated rules and new products or services the organization needs to scale to support. As the system scales up and becomes more complicated it can become very hard to adapt quickly to these changing requirements. In fact, the system can even become harder to change and slower to accept new requirements. This can lead to a desired architecture that is designed to scale allowing the system to more easily adapt to changing requirements. Do you have the goal of building a system that can be extended and adapted without programmer intervention? Do you have the itch to explore meta-programming, not just because it is cool or complicated, but because you want your system's behavior or domain representations to scale? If so, consider learning about the design of systems that represent user-defined behavior specifications as metadata. Architects create Adaptive Object-Model systems when they want to enable end-user programmers to adapt their system's behavior and they don't want developers to become the bottleneck. But what does it take to build a system that can be changed and adapted without programming? How do Adaptive Object-Models differ from little languages or DSLs and when is it appropriate to consider stepping into the meta world to build such an extensible system? This talk presents the basic ideas about meta-architecture such as an Adaptive Object-Model architecture and shares experiences that the presenters have had building and tuning various production meta-architecture systems that have enabled organizations to scale them more easily and without programming wizardry.

1. Joseph W. Yoder & Rebecca Wirfs-Brock
When Should You Consider
Meta-Architectures?
Joseph W. Yoder
joe@refactory.com
Copyright 2011, Joseph W. Yoder & The Refactory, Inc. and Wirfs-Brock Associates
Agenda
Ways to evolve systems
What meta really allows
The difference between meta-
architectures and DSLs
Adaptive Object-Model (AOM) basics
Case studies of systems that scale
Evolved from a talk given with
Rebecca Wirfs-Brock at QCon 2010 Slide - 2
Using Meta to Scale Page - 1

2. Joseph W. Yoder & Rebecca Wirfs-Brock
Contrast: Two Ways to Evolve a
Working System
Traditional SW Meta-Data Based
Architecture Architecture
Who implements a A programmer. A domain expert might, if it is
requirements change? a domain object or business
rule change.
How are changes verified? Programmer writes tests, QA Still have to run all unit and
writes and runs acceptance acceptance tests. But can also
tests, end-user approves. build into end-user tool
checks that model changes
won’t ―break‖ the existing
working system.
How often can the system be At the end of an iteration. Whenever changes are
updated in production? verified. Not tied to dev cycle.
What’s the big deal? Still have to go through some Significant changes can be
release cycle. Programming made by end-users. Releasing
and deployment can become can be simply updating
bottlenecks. production metadata.
Slide - 3
Stuart Brand’s Shearing Layers
Buildings are a set of components that
evolve in different timescales.
Layers: site, structure, skin, services,
space plan, stuff. Each layer has its own
value, and speed of change (pace).
Buildings adapt because
faster layers (services) are
not obstructed by slower
ones (structure).
—Stuart Brand, How Buildings Learn Slide - 4
Using Meta to Scale Page - 2

3. Joseph W. Yoder & Rebecca Wirfs-Brock
Yoder and Foote’s
Software Shearing Layers
―Factor your system so that artifacts that change at
similar rates are together.‖—Foote & Yoder, Ball of
Mud, PLoPD4
Layers
Slower  The platform
 Infrastructure
 Data schema
 Standard frameworks, components, and modules
 Abstract classes and interfaces
 Classes
 Code
 Data
Faster Slide - 5
Our Design Values
Respect your system’s shearing layers.
 Understand the rates of what changes.
Determine who should be able to make
changes, when, and at what cost.
 Support them!
Make what is too difficult, time
consuming, or tedious easier.
 Create tools, leverage design patterns,
use data to drive behavior…
Don’t overdesign!!! Slide - 6
Using Meta to Scale Page - 3

4. Joseph W. Yoder & Rebecca Wirfs-Brock
A Disclaimer
This talk is not about
 Using metadata to generate code
 Building a ―stack‖ of meta-models and/or
transforming between various models
This talk focuses on practical techniques
and cases studies for using metadata to
scale solutions
Slide - 7
What do we want to scale?
Increased
✗

 performance
 throughput
 adaptability/flexibility
✔
 rate of supporting new requirements
 time to deploy
 support for configuration options
 Reduced code bulk
 Higher-leverage code (frameworks, tools…)
Slide - 8
Using Meta to Scale Page - 4

5. Joseph W. Yoder & Rebecca Wirfs-Brock
Alternative Computation Styles
Imperative Declarative
Traditional Executable
programming specifications
Define a series of Many options:
steps with  DSLs (Special language)
conditionals and loops  Decision tables
State machines
to vary sequence 
 Production rule systems
 Adaptive Model Systems
Domain Specific Languages, Martin Fowler and Rebecca Parsons
Slide - 9
The Power of Metadata
Code is data, data is code. Everything is
data. And data can drive behavior.
Meta data simply describes other data.
―If something is going to vary in a
predictable way, store the description of
the variation in a database so that it is
easy to change‖—Ralph Johnson
Slide - 10
Using Meta to Scale Page - 5

6. Joseph W. Yoder & Rebecca Wirfs-Brock
Architecture of DSL Processing
optional
✗
__________
__________ __________
__________
__________ __________
______
__________ __________
______
__________ _________
__________
__________
DSL Script
parse generate
DSL Script semantic target code
model
We focus on a virtual machine to interpret the semantic model!
Domain Specific Languages, Martin Fowler and Rebecca Parsons
Slide - 11
What’s a Semantic Model?
Syntax describes ―legal‖ expressions.
Semantics is what it means; what it
does when it executes. Could be:
 A representation, such as an in-memory
object model or an adaptive object model
 A data structure with behavior coming from
functions acting on the data
 Production rules and a rules ―engine‖
 A decision table
Slide - 12
Using Meta to Scale Page - 6

7. Joseph W. Yoder & Rebecca Wirfs-Brock
Domain Specific Languages
Simple DSLs might be implemented via
scripting languages and parameterization.
They may not be interpreted.
Sometimes they can be compiled directly to
executable code.
AOMs can be a Domain Specific Language
but don’t have to be and vice-versa
Slide - 13
Motivation:
Need to Quickly Adapt to Change
Business Rules or Domain Elements are
changing quickly:
 New calculations for insurance policies and
new types of policies offered
 Online store catalog with new products and
services and rules applying to them
 New cell phone product and services…
Need quick ways to develop and
adapt to these changing requirements.
Slide - 14
Using Meta to Scale Page - 7

8. Joseph W. Yoder & Rebecca Wirfs-Brock
Adaptive Object-Model
Mechanics
 Meta information about the domain model
and business rules is interpreted at runtime.
 Domain classes, attributes, behaviors and relationships
described by metadata
 Instance-based
 Object-Model stored in a database or in
files and interpreted (can be XML/XMI).
 When you change the metadata, system behavior
changes.
 Domain experts typically make changes,
not programmers. Slide - 15
AOM
―Virtual Machine‖
Application
Persistence
Mechanism
A
O
Database M
A
Metadata r Domain
Repository/Namespace c Objects
h
I
t
e
c
t
Interpreter/ u
XML Parser
Builder r
e
XML
Slide - 16
Using Meta to Scale Page - 8

9. Joseph W. Yoder & Rebecca Wirfs-Brock
Architecture Elements of
Adaptive Object Models
• Metadata • Strategy/RuleObjects
• TypeObject • Entity-Relationship
• Properties • Interpreters/Builders
• Type Square • Editors/GUIs
If you want something to change quickly,
push it into data and build tools geared towards
changemakers’ needs. Slide - 17
AOM and TypeSquare
Entity EntityType Constrains
0..n type
possible
properties for
an entity
properties 0..n Constrains 0..n properties
type of a
property PropertyType
Property
0..n type -name : String
-type : Type
Slide - 18
Using Meta to Scale Page - 9

10. Joseph W. Yoder & Rebecca Wirfs-Brock
Type-Object
PLoPD3 - Johnson and Woolf
Before After
Solution: Factor
Symptom: Explosion of classes based on common attributes
minor attribute differences into ―type‖ classes Slide - 19
Type Square
(instance diagram)
aPropertyType anEntityType aPropertyType
<RecSpeed (int)> <vinylAudioRec> <AudMedia(Str)>
aProperty anAccountabilityType
anEntity aProperty
name <45> <PARENT>
Joe’s Garage name <SuperFid>
Instance-Based!!!
Slide - 20
Using Meta to Scale Page - 10

11. Joseph W. Yoder & Rebecca Wirfs-Brock
AOM and TypeSquare
(code example)
public class EntityType {
private string _typeName;
private Type _type;
public EntityType (string name, Type type) {
_typeName = name;
_type = type; }
public string Name {
get{return _typeName;}
set{_typeName = value;} }…}
http://www.codeproject.com/
Christopher G. Lasater Slide - 21
AOM Support for Different
Types of Behavior
If behavior varies, specify rules in meta-
data and interpret. Typical behaviors:
 Constraints on values, relationships, state
changes…(discrete values, ranges,
permissible associations)
 Functional
 Workflow
 Event based
Slide - 22
Using Meta to Scale Page - 11

12. Joseph W. Yoder & Rebecca Wirfs-Brock
Adaptive Object Model ―Core
Domain Architecture‖ An AOM Semantic
Model is Instance-
-children * * -children
1 Classes with
Based
-type
Attributes and
Accountability AccountabilityType
1
1 1 Relationships
1 -accountabilities 1 -accountabilitieTypes Behavior
-children ** -children * *
-rules *
Entity -type EntityType Rule
1 1 1
1 1 +valueUsing:()
1 1 -attributeTypes
-attributes * *
* 1
*
Attribute AttributeType Constant CompositeRule
+name +value
-type +type 1 * 1
+valueUsing:()
*
Operational Knowledge (meta) TableLookup BinaryOperation
*
Slide - 23
Adaptive Object Model
―Describing Your Entities‖
<entities>
<entity name="President"
entityType="TopLevelExecutiveType"
actualAttributes="EmployeeId"
/>
<entity name="VicePresident"
entityType="ExecutiveType"
actualAttributes="EmployeeId"
/>
….
</entities>
Slide - 24
Using Meta to Scale Page - 12

13. Joseph W. Yoder & Rebecca Wirfs-Brock
Adaptive Object Model
―Describing Your Relationships‖
<accountibilities>
<!-- Hiring //-->
<accountability name="PresidentHiresVicePresident"
accountabilityType="TopLevelExecutiveHiresExecutive"
actualParent="President"
actualChildren="VicePresident" />
<accountability name="VicePresidentHiresSalesManager"
accountabilityType="ExecutiveHiresManager"
actualParent="VicePresident"
actualChildren="SalesManager" />
…
</accountibilities>
Slide - 25
Scaling Examples
Industrial (Manufacturing)
Medical (IDPH)
Insurance
Telephony Billing System
Pontis Telephony Marketing
Portugal System
Slide - 26
Using Meta to Scale Page - 13

14. Joseph W. Yoder & Rebecca Wirfs-Brock
Manufacturing - Shared
Corporate Organization Data
~1,000,000 lines of COBOL code for editing
organization data.
Organization data shared by many systems.
Had a common UI editor for adding, and
updating this data. Editor was cloned and
modified many times…grew to 100+ copies.
Documentation and maintenance
cumbersome.
Slide - 27
Manufacturing - Shared
Corporate Organization Data
Rewrote in Java code ~ 40k LOC that
used XML to describe the mappings to
the organization data and generated
the GUI and SQL queries.
Documentation could be generated
easily from XML mappings.
Easy to change and release new
versions for new tables through
updates to the XML schema.
Slide - 28
Using Meta to Scale Page - 14

15. Joseph W. Yoder & Rebecca Wirfs-Brock
User Defined Product
(insurance example) Wirfs-Brock’s
*
Component Policy
New types of Policies require new
classes and attributes…and changes
to rules require updates to methods
Slide - 29
User Defined Product
(insurance example)
-children * * -children
1 -rules *
Component -type ComponentType Rule
1 1
1 1 +valueUsing:()
1 1 -attributes
-attributes * *
* 1
*
Attribute AttributeType Constant CompositeRule
+name +value
-type +type 1 * 1
+valueUsing:()
*
TableLookup BinaryOperation
*
Core Architecture is TypeSquare
Could quickly adapt to new rules within weeks or less
http://st-www.cs.illinois.edu/users/johnson/papers/udp/ Slide - 30
Using Meta to Scale Page - 15

16. Joseph W. Yoder & Rebecca Wirfs-Brock
Objectiva Business Entities
(telephony billing system example)
Originally took a few person years to build.
AOM implementation reduced this to few person months
Slide - 31
Objectiva Attributes
(telephony example)
Slide - 32
Using Meta to Scale Page - 16

17. Joseph W. Yoder & Rebecca Wirfs-Brock
Objectiva Entity Relationships
(telephony example)
Slide - 33
How Objectiva Scaled
Initial AOM implementation could
handle ~1 million transactions per day
Not enough for larger installations…
 So, performance was tackled by adding
caching and other point
optimizations….increasing throughput
to over 10 million transactions
 Lesson: Meta Architectures can be tweaked
for performance…nothing magic here
Slide - 34
Using Meta to Scale Page - 17

18. Joseph W. Yoder & Rebecca Wirfs-Brock
Refugee and Newborn
Screening Example
Doctor, HealthProf.
Hospital, Lab
Blood Specimen
Mother, Infant
Slide - 35
Screening Refugees
Slide - 36
Using Meta to Scale Page - 18

19. Joseph W. Yoder & Rebecca Wirfs-Brock
Medical Observation –
Basic OO Design Model
Person * Observation
Measurement Trait
Quantity
traitValue
unit convertTo:
amount
expressOnUnit:
expressOnUnits:
PhysicalMeasure Blood
EyeColor HairColor Gender …
Height Weight FeedingType Hearing Vision
…
What happens when a new observation is required?
Had well over 100 observations! Slide - 37
Medical Observations Design
1..* type contDescr 1
Entities guard 1
Validator
validatorName
ObservationType 1..* varType PartyType isValid: obsValue
phenomenonType 1 descriptor
isValid: obsValue 1 descriptor 1..*
1..* elements
Party DiscreteValidator RangedValidator NullValidator
descriptorSet intervalSet
PrimitiveObservation CompositeObservation validUnit
Type Type properties
1..*
1..* Observation
Behavior
instance recordedDate
comments
elements 1..* isValid
Quantity
unit
value 1..* quantity
CompositeObservation Primitive Observation
observationValue convertTo:
Knowledge level
Operational level dvalue 1..* DiscreteValues
Attributes
Modelled100s of Entities with this core architecture.
Can add new entities or make changes to entities very quickly. Slide - 38
Using Meta to Scale Page - 19

20. Joseph W. Yoder & Rebecca Wirfs-Brock
PONTIS AOM ENGINEERS
Telephony Marketing System
Highly Flexible
Slide - 39
PONTIS WORKFLOW
Developer AOM Engineer
―Adaptive Object-Model Metadata Evolver‖, PLoP 2010, Atzmon Hen-Tov, David
H. Lorenz, Lena Nikolaev, Lior Schachter, Rebecca Wirfs-Brock, Joseph Yoder Slide - 40
Using Meta to Scale Page - 20

21. Joseph W. Yoder & Rebecca Wirfs-Brock
How the Pontis System Scales
Reduced deployment from 1 year
to 1 month
Updates every couple of weeks
Deployed to over 30 clients
Code base much smaller 500,000 LOC
vs 3,000,000 (previous implementation)
Fastest growing business award
 5,500 percent within five years
Slide - 41
Portuguese Historical
Artifacts Catalogue
Catalogue of historical artifacts
Mapping of site artifacts, relationships over
time, historical site evolution
2 years churning on requirements
Technical team finally made the decision to
support easy ways to change the
domain…metadata-driven software
1. Hugo Sereno Ferreira, Filipe Figueiredo Correia, Ademar Aguiar, and João Pascoal Faria. "Adaptive Object-Models: a Research Roadmap".
International Journal On Advances in Software, volume 3, numbers 1 and 2, 2010. ISSN: 1942-2628.
2. Hugo Sereno Ferreia, Filipe Figueiredo Correia, Ademar Aguiar. "Design for an Adaptive Object-Model Framework: An Overview". Proceedings
of the 4th International Workshop on Models@run.time. Co-located with the 12th International Conference on Model Driven Engineering Languages
and Systems. Denver, Colorado. USA.
3. Hugo Sereno Ferreira, Ademar Aguiar, João Pascoal Faria. "Adaptive Object Modelling: Patterns, Tools and Applications". 3rd Symposium
Slide - 42
on Doctoral Students of Software Engineering. Proceedings of the 4th International Conference on Software Engineering Advances. Porto. Portugal.
Using Meta to Scale Page - 21

22. Joseph W. Yoder & Rebecca Wirfs-Brock
Project Timeline: Growth of
the System
metadata
4 months
to first
deployment
features
code added
rapidly at
project end
Slide - 43
Project Timeline: Release
Frequency
Once customers realized they could make
model changes, they made lots of changes.
The rate of deployment of new releases
increased, too.
Slide - 44
Using Meta to Scale Page - 22

23. Joseph W. Yoder & Rebecca Wirfs-Brock
Where Metadata Techniques
Have been Successfully Used:
Representing Insurance Policies
Telephone Billing and Back Office Systems
Workflow Systems
Medical Observations
Banking and Trading
Validate Equipment Configuration
Documents Management System
Gauge Calibration Systems
Simulation Software
Invoicing Systems (some can be found in papers)
Manufacturing Systems www.adaptiveobjectmodel.com
…
Slide - 45
Some Disadvantages of
Metadata Based Architectures
Requires infrastructure for storing,
building, and interpreting metadata.
Requires strong design skills to create.
Interpreting metadata may result in
lower performance (this can be
remedied by well-known techniques).
Slide - 46
Using Meta to Scale Page - 23

24. Joseph W. Yoder & Rebecca Wirfs-Brock
Agile Best Practices for
Creating Metadata Solutions
Separate what changes quickly from what
changes slowly (hot-spots).
Develop iteratively and incrementally. Get
feedback early and often.
Add flexibility using meta-data techniques
only when and where needed.
Write tests for expected system behaviors
and the meta tools/interpreters.
Slide - 47
Summary
Meta-architectures enable rapid
changes to domain objects and rules
Core application code less than with
conventional programming
Scale in two dimensions:
 support for rapid requirements changes
 design leverage (team size ~10^1)
Slide - 48
Using Meta to Scale Page - 24

25. Joseph W. Yoder & Rebecca Wirfs-Brock
Meta Collaborators
Ademar Aguiar Atzmon Hen-Tov
Francis Anderson Ralph Johnson
Ali Arsanjani David H. Lorenz
Jean Bezivin Lena Nikolaev
Paulo Borba Jeff Oaks
Filipe Correia Reza Razavi
Krzysztof Czarnecki Nicolas Revault
Ayla Dantas Dirk Riehle
Martine Devos Lior Schachter
Hugo Ferreira Dave Thomas
Brian Foote Michel Tilman
Martin Fowler Leon Welicki
Richard Gabriel Rebecca Wirfs-Brock…Slide - 49
Resources
Adaptive Object Models
 www.adaptiveobjectmodel.com
Agile Software
 Refactoring www.refactory.com
 Agile Alliance: www.agilealliance.org
 The Agile Manifesto
 12 Principles of Agile Development
Slide - 50
Using Meta to Scale Page - 25

26. Joseph W. Yoder & Rebecca Wirfs-Brock
Obrigado! That’s All
joe@refactory.com
Twitter: @metayoda
rebecca@wirfs-brock.com
Twitter: @rebeccawb Slide - 51
Using Meta to Scale Page - 26