SUSTAINABILITY SUPPORTING DATA VARIABILITY: KEEPING CORE COMPONENTS CLEAN WHILE DEALING WITH DATA VARIABILITY A big challenge in building complex, data-intensive systems is how to sustainably support data variation, schema, and feature evolution. This talk examines strategies, practices, and patterns drawn from real experiences that support new and evolving data-processing requirements while keeping the core architecture clean. As complex systems evolve to meet varying data formats, they can devolve into poorly architected Big Balls of Mud filled with special-case logic and one-off processing. Alternatively, you can isolate core components of your system and protect them from entanglements and unnecessary complexity by designing them to operate on common data formats while providing extension mechanisms that enable processing variations.

1. 10/12/2015
1
Sustainable Architecture
“Supporting Data Variability”
Copyright 2014 Joseph W. Yoder & The Refactory, Inc.
JDD – October 12th, 2015
Joseph W. Yoder -- www.refactory.com
Sustaining Your Architecture
Sustaining
Architecture
When dealing with data variability…
“What can be done to help sustain
our architecture in the long run?”

2. 10/12/2015
2
Sustaining Your Architecture
Motivation:
Need to Quickly
Adapt to Change
 Business Rules and Domain Elements
are varying a lot and changing quickly:
 New calculations for insurance policies
and new types of policies offered
 Online store catalog with new products
and services and rules applying to them
 New cell phone product and services…
 Need quick ways to develop and adapt
to these varying requirements
Sustaining Your Architecture
Core Ideas
Separate what changes
from what doesn’t change
Protect core components
Find the variability points and provide
ways to support this variability in data
 Description of changes (metadata)
 Hook Points for variability
 Dynamic Process Rules using Metadata
4

3. 10/12/2015
3
Sustaining Your Architecture
Stuart Brand’s Shearing Layers
 Buildings are a set of components that
evolve in different timescales
 Layers: site, structure, skin, services,
space plan, stuff. Each layer has its own
value, and speed of change (pace)
 Buildings adapt because
faster layers (services) are
not obstructed by slower
ones (structure)
—Stuart Brand, How Buildings Learn
Sustaining Your Architecture
Yoder and Foote’s
Software Shearing Layers
“Factor your system so that artifacts that change at
similar rates are together.”—Foote & Yoder, Ball of
Mud, PLoPD4.
• The platform
• Infrastructure
• Data schema
• Standard frameworks, components, services
• Abstract classes, Interfaces, APIs, Schemas
• Classes, Lambdas
• Code (functions, algorithms)
• Data (metadata)
LayersSlower
Faster

4. 10/12/2015
4
Sustaining Your Architecture
Sweep It Under the Rug
Cover it up to keep other areas clean
(Façade and other Wrapper Patterns)
Sustaining Your Architecture
Put a rug at the Front Door
Protect Important Components!
Keep other parts of the system clean
Sometimes Glue code (Mediators) helps
keep others parts of the system cleaner
(Anti-Corruption Layer -- Eric Evans)

5. 10/12/2015
5
Sustaining Your Architecture
Wipe your Feet
at the Front Door
ALIAS: ENCAPSULATE AND IGNORE
KEEPING THE INTERNALS CLEAN
Patterns for
Sustaining Architecture
PLoP 2012 Paper
Sustaining Your Architecture
Wipe your Feet
at the Front Door
Filtering and Cleansing Sequence to keep
Place Order Interface Clean
Protected
Components
S1
S2
S3
Sn
Adapter/
Proxy/
Façade
Frontdoor
Wrapper
Filtering,
Cleansing,
Security
Checks
...

6. 10/12/2015
6
Sustaining Your Architecture
THINGS ARE CHANGING:
SYSTEMS NEED TO QUICKLY
ADAPT TO VARIATIONS!!!
Sustaining Your Architecture
In my youth…two bad words
M and R words
Metadata and Reflection

7. 10/12/2015
7
Sustaining Your Architecture
Metadata
Sustaining Your Architecture
The Power of Metadata
Code is data, data is code. Everything
is data. And data can drive behavior
Meta data simply describes other data.
“If something is going to vary in a
predictable way, store the description
of the variation in a database so that it
is easy to change”—Ralph Johnson
"Anything you can do, I can do Meta"
”Meta is Beta"

8. 10/12/2015
8
Sustaining Your Architecture
What is Metadata
“Data about Data”
Document Type Book
Authors
Erich Gamma, John Vlissides,
Ralph Johnson, and Richard Helm
Year of
Publication
1994
Title
Design Patterns: Elements of
Reusable Object-Oriented Software
Page Count 395
Publisher Addison-Wesley
Description
Software engineering book describing
recurring solutions to common problems
in software design...
Sustaining Your Architecture
Metadata – the Three “M”s
Metaobjects
Markup Manifest
Brian Foote: Metadata, The Good, The Bad, The Ugly
Mini-SugarLoaf PLoP, São Paulo, Brazil - September 2011

9. 10/12/2015
9
Sustaining Your Architecture
Metadata – Markup
<xsl:stylesheet xmlns:xsl="http://www.w3.org/
1999/XSL/Transform" version="1.0">
<xsl:key match="/publisher/books/book" use="author-ref/@ref"
name="books-by-author"/>
<xsl:template match="/"><bestsellers-list> <xsl:apply-templates
select="/publisher/authors/author"> <xsl:sort select="sum(key('books-by-
author', @id)/sold)"/> <xsl:sort select="last_name"/> </xsl:apply-
templates> </bestsellers-list></xsl:template>
<xsl:template match="author"> <copy> <name> <xsl:value-of
select="last_name"/>, <xsl:value-of select="first_name"/> </name>
<total_publications> <xsl:value-of select="count(key('books-by-author',
@id))"/> </total_publications> <total_sold> <xsl:value-of
select="sum(key('books-by-author', @id)/sold)"/>
</total_sold> <rank> <xsl:value-of select="position()"/>
</rank> </copy> </xsl:template>
</xsl:stylesheet>
Sustaining Your Architecture
public class Car {
// Injectable constructor
@Inject public Car(Engine engine) { ... }
// Injectable field
@Inject private Provider<Seat> seatProvider;
// Injectable package-private method
@Inject void install(Windshield windshield, Trunk trunk) { ... }
}
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD) //can use in method only.
public @interface Test {…}
Metadata – Annotations

10. 10/12/2015
10
Sustaining Your Architecture
JUnit Annotations
Annotation Description
@Test public void method() Annotation @Test identifies that method is a test method.
@Before public void method()
Will perform the method() before each test. This method
can prepare the test environment, e.g. read input data,
initialize the class)
@After public void method() Test method must start with test
@BeforeClass
public void method()
Will perform the method before the start of all tests. This
can be used to perform time intensive activities such as
connecting to or loading data from a database.
@AfterClass
public void method()
Will perform the method after all tests have finished. This
can be used to perform clean-up activities for example be
used to disconnect to a database
@Ignore
Will ignore the test method. Useful if the underlying code
has changed and the test has not yet been adapted or if
running the test takes too long.
@Test(expected=IllegalArgume
ntException.class)
Tests if named exception is thrown.
@Test(timeout=100) Fails if the method takes longer then 100 milliseconds
Sustaining Your Architecture
Metadata – Most use 2*M
Markup and Manifest are “good”
enough for most successful
meta-architectures…
Metadata For Online Store Catalog
Sweaters (size=(S,M,L,XL),
color=(red, green, blue, yellow,…))
Canoes (length=float, width=float,
color=(red, brown, blue, silver,…))
Video (name=string, rating=(P,PG,R,…),
category=(Action, Drama,…))

11. 10/12/2015
11
Sustaining Your Architecture
Custom Metadata
Sustaining Your Architecture
AOM and TypeSquare
(code example)
public class EntityType {
private string _typeName;
private Type _type;
public EntityType
(string name, Type type) {
_typeName = name;
_type = type; }
public string Name {
get{return _typeName;}
set{_typeName = value;} }…}
http://www.codeproject.com/
Christopher G. Lasater
<entities>
<entity name="President"
entityType="TopLevelExecutiveType"
actualAttributes="EmployeeId"
/>
<entity name="VicePresident"
entityType="ExecutiveType"
actualAttributes="EmployeeId"
/>
….
</entities>

12. 10/12/2015
12
Sustaining Your Architecture
Dynamic Hook Points
Eli Acherkan, Atzmon Hen-Tov, David H. Lorenz,
Lior Schachter, Rebecca Wirfs-Brock, Joseph W. Yoder
Asian PLoP 2012
When building dynamic systems, it is often the case
that new behavior is needed which is not supported
by the core architecture. One way to vary the behavior
quickly is to provide well-defined variation points,
called hook-points, in predefined places in the
systems where new behavior can be dynamically
looked up and invoke at runtime when desired.
Can use metadata annotations to look up and invoke!
Sustaining Your Architecture
Dynamic Hook Points
Eli Acherkan, Atzmon Hen-Tov, David H. Lorenz,
Lior Schachter, Rebecca Wirfs-Brock, Joseph W. Yoder
Asian PLoP 2012

13. 10/12/2015
13
Sustaining Your Architecture
Paving over the
Wagon Trail
ALIAS: MAKE REPETITIVE TASKS EASIER
STREAMLINING REPETITIVE CODE TASKS
Create simple examples, templates, & scripts
Develop a tool that generates code
Identify and use existing tools or frameworks
Develop a framework &/or runtime environment
Develop a domain-specific language
Patterns for Sustaining Architecture
PLoP 2012 Paper
Sustaining Your Architecture
Paving over the
Wagon Trail
Patterns for Sustaining Architecture
PLoP 2012 Paper

14. 10/12/2015
14
Alternative Computation Styles
Imperative
Traditional programming
Define a series of steps
with conditionals and
loops to vary sequence
Declarative
Executable specifications
Many options:
DSLs (Special language)
Decision tables
State machines
Production rule systems
Adaptive Model Systems
Domain Specific Languages, Martin Fowler and Rebecca Parsons
Architecture of DSL Processing
__________
__________
__________
__________
__________ DSL Script
parse
semantic
model
generate
__________
__________
__________
__________
______
__________
______
_________
__________
DSL Script target code
optional
Domain Specific Languages, Martin Fowler and Rebecca Parsons
✗
DSLs can be a scripting language and work by means of parameterization,
They don’t need to have a meta-level but they often do have one!!!

15. 10/12/2015
15
Sustaining Your Architecture
IMPORT EXAMPLE:
HOW TO DEAL WITH
CLIENT VARIABILITY???
Sustaining Your Architecture
Import Orders
 Provided a standard
format for imports
 Lots of duplication
 If Clients needed to
vary from that format,
we had to write or update a complete,
self-contained custom import program
 Imports that need special processing, or
had client specific rules, required to write
a complete, self-contained import program

16. 10/12/2015
16
Sustaining Your Architecture
Motivation:
Need to Quickly
Adapt to Change
 Business Rules or Domain Elements
are changing quickly:
 New calculations for insurance policies
and new types of policies offered
 Online store catalog with new products
and services and rules applying to them
 New cell phone product and services…
 Need quick ways to develop and adapt
to these changing requirements.
Sustaining Your Architecture
Import Validation
[ValidationRule(“Joes Validation Rule")]
public class JoesValidationRule : ValidationRule {
public JoesValidationRule () : base() { }
public override void Validate(ImportContext context)
…}
Validate
Process
Core Rules
Load and Process
Cleint Specific Rules
• Dynamic
Hook Point

17. 10/12/2015
17
Sustaining Your Architecture
Architectural Practice:
Support Data Variability
 Separate what
changes from stable
part of system and
provide support for
adaptability/flexibility
 Integrate new
learning into
your architecture
 Refactoring
 Redesign
 Rework
 Code clean up
Sustaining Your Architecture
Sustaining an
Architecture
 Minimize architectural
debt: Support the ability
to change/adapt what
needs to change
 Make what is too
difficult, time consuming,
or tedious easier
 Decide at the most
responsible moment,
not the last possible
moment
 Learn and evolve
Keep the system “livable” for its users and developers

18. 10/12/2015
18
Sustaining Your Architecture
Sustainable
Architecture
Stewardship
 Follow through
 Ongoing attention
 Not ignoring the little
things that can undermine
our ability to grow, change
and adapt our systems
Sustaining Your Architecture
Indicators You’ve Paid Enough
Attention to Architecture
 Defects are localized
 Stable interfaces
 Consistency
 Developers can easily
add new functionality
 New functionality doesn’t
“break” existing architecture
 Few areas that developers avoid
because they are too difficult to work in
 Able to incrementally integrate new functionality

19. 10/12/2015
19
Sustaining Your Architecture
Continuous Inspection
Asian PLoP 2014 Paper
Sustaining Your Architecture
Continuous Inspection
Asian PLoP 2014 Paper
CODE SMELL DETECTION
METRICS (TEST COVERAGE, CYCLOMATIC COMPLEXITY,
TECHNICAL DEBT, SIZES, …)
APPLICATION SECURITY CHECKS
ARCHITECTURAL CONFORMANCE
AUTOMATE WHERE YOU CAN!!!

20. 10/12/2015
20
Sustaining Your Architecture
Design Values
Supporting Variability
 Respect your system’s shearing layers
 Understand the rates of what changes
 Determine & support who should be able
to make changes, when, and at what cost
 Make what is too difficult, time
consuming, or tedious easier
 Create tools, leverage design
patterns, use data to drive behavior…
 Don’t overdesign!!! (Only design
what you need when you need it)
Sustaining Your Architecture
Resources
Adaptive Object Models
 www.adaptiveobjectmodel.com
 www.metaplop.org
Teams That Innovate
 www.teamsthatinnovate.org
Refactoring www.refactory.com
Hillside Patterns www.hillside.net

21. 10/12/2015
21
Dziękuję!!!
joe@refactory.com
Twitter: @metayoda