Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
10/12/2015
1
Sustainable Architecture
“Supporting Data Variability”
Copyright 2014 Joseph W. Yoder & The Refactory, Inc.
J...
10/12/2015
2
Sustaining Your Architecture
Motivation:
Need to Quickly
Adapt to Change
 Business Rules and Domain Elements...
10/12/2015
3
Sustaining Your Architecture
Stuart Brand’s Shearing Layers
 Buildings are a set of components that
evolve i...
10/12/2015
4
Sustaining Your Architecture
Sweep It Under the Rug
Cover it up to keep other areas clean
(Façade and other W...
10/12/2015
5
Sustaining Your Architecture
Wipe your Feet
at the Front Door
ALIAS: ENCAPSULATE AND IGNORE
KEEPING THE INTER...
10/12/2015
6
Sustaining Your Architecture
THINGS ARE CHANGING:
SYSTEMS NEED TO QUICKLY
ADAPT TO VARIATIONS!!!
Sustaining Y...
10/12/2015
7
Sustaining Your Architecture
Metadata
Sustaining Your Architecture
The Power of Metadata
Code is data, data i...
10/12/2015
8
Sustaining Your Architecture
What is Metadata
“Data about Data”
Document Type Book
Authors
Erich Gamma, John ...
10/12/2015
9
Sustaining Your Architecture
Metadata – Markup
<xsl:stylesheet xmlns:xsl="http://www.w3.org/
1999/XSL/Transfo...
10/12/2015
10
Sustaining Your Architecture
JUnit Annotations
Annotation Description
@Test public void method() Annotation ...
10/12/2015
11
Sustaining Your Architecture
Custom Metadata
Sustaining Your Architecture
AOM and TypeSquare
(code example)
...
10/12/2015
12
Sustaining Your Architecture
Dynamic Hook Points
Eli Acherkan, Atzmon Hen-Tov, David H. Lorenz,
Lior Schacht...
10/12/2015
13
Sustaining Your Architecture
Paving over the
Wagon Trail
ALIAS: MAKE REPETITIVE TASKS EASIER
STREAMLINING RE...
10/12/2015
14
Alternative Computation Styles
Imperative
Traditional programming
Define a series of steps
with conditionals...
10/12/2015
15
Sustaining Your Architecture
IMPORT EXAMPLE:
HOW TO DEAL WITH
CLIENT VARIABILITY???
Sustaining Your Architec...
10/12/2015
16
Sustaining Your Architecture
Motivation:
Need to Quickly
Adapt to Change
 Business Rules or Domain Elements...
10/12/2015
17
Sustaining Your Architecture
Architectural Practice:
Support Data Variability
 Separate what
changes from s...
10/12/2015
18
Sustaining Your Architecture
Sustainable
Architecture
Stewardship
 Follow through
 Ongoing attention
 No...
10/12/2015
19
Sustaining Your Architecture
Continuous Inspection
Asian PLoP 2014 Paper
Sustaining Your Architecture
Contin...
10/12/2015
20
Sustaining Your Architecture
Design Values
Supporting Variability
 Respect your system’s shearing layers
 ...
10/12/2015
21
Dziękuję!!!
joe@refactory.com
Twitter: @metayoda
Upcoming SlideShare
Loading in …5
×

JDD2015: Sustainability Supporting Data Variability: Keeping Core Components Clean While Dealing with Data Variability - Joseph W. Yoder

120 views

Published on

SUSTAINABILITY SUPPORTING DATA VARIABILITY: KEEPING CORE COMPONENTS CLEAN WHILE DEALING WITH DATA VARIABILITY

A big challenge in building complex, data-intensive systems is how to sustainably support data variation, schema, and feature evolution. This talk examines strategies, practices, and patterns drawn from real experiences that support new and evolving data-processing requirements while keeping the core architecture clean. As complex systems evolve to meet varying data formats, they can devolve into poorly architected Big Balls of Mud filled with special-case logic and one-off processing. Alternatively, you can isolate core components of your system and protect them from entanglements and unnecessary complexity by designing them to operate on common data formats while providing extension mechanisms that enable processing variations.

Published in: Software
  • Be the first to comment

  • Be the first to like this

JDD2015: Sustainability Supporting Data Variability: Keeping Core Components Clean While Dealing with Data Variability - Joseph W. Yoder

  1. 1. 10/12/2015 1 Sustainable Architecture “Supporting Data Variability” Copyright 2014 Joseph W. Yoder & The Refactory, Inc. JDD – October 12th, 2015 Joseph W. Yoder -- www.refactory.com Sustaining Your Architecture Sustaining Architecture When dealing with data variability… “What can be done to help sustain our architecture in the long run?”
  2. 2. 10/12/2015 2 Sustaining Your Architecture Motivation: Need to Quickly Adapt to Change  Business Rules and Domain Elements are varying a lot and changing quickly:  New calculations for insurance policies and new types of policies offered  Online store catalog with new products and services and rules applying to them  New cell phone product and services…  Need quick ways to develop and adapt to these varying requirements Sustaining Your Architecture Core Ideas Separate what changes from what doesn’t change Protect core components Find the variability points and provide ways to support this variability in data  Description of changes (metadata)  Hook Points for variability  Dynamic Process Rules using Metadata 4
  3. 3. 10/12/2015 3 Sustaining Your Architecture Stuart Brand’s Shearing Layers  Buildings are a set of components that evolve in different timescales  Layers: site, structure, skin, services, space plan, stuff. Each layer has its own value, and speed of change (pace)  Buildings adapt because faster layers (services) are not obstructed by slower ones (structure) —Stuart Brand, How Buildings Learn Sustaining Your Architecture Yoder and Foote’s Software Shearing Layers “Factor your system so that artifacts that change at similar rates are together.”—Foote & Yoder, Ball of Mud, PLoPD4. • The platform • Infrastructure • Data schema • Standard frameworks, components, services • Abstract classes, Interfaces, APIs, Schemas • Classes, Lambdas • Code (functions, algorithms) • Data (metadata) LayersSlower Faster
  4. 4. 10/12/2015 4 Sustaining Your Architecture Sweep It Under the Rug Cover it up to keep other areas clean (Façade and other Wrapper Patterns) Sustaining Your Architecture Put a rug at the Front Door Protect Important Components! Keep other parts of the system clean Sometimes Glue code (Mediators) helps keep others parts of the system cleaner (Anti-Corruption Layer -- Eric Evans)
  5. 5. 10/12/2015 5 Sustaining Your Architecture Wipe your Feet at the Front Door ALIAS: ENCAPSULATE AND IGNORE KEEPING THE INTERNALS CLEAN Patterns for Sustaining Architecture PLoP 2012 Paper Sustaining Your Architecture Wipe your Feet at the Front Door Filtering and Cleansing Sequence to keep Place Order Interface Clean Protected Components S1 S2 S3 Sn Adapter/ Proxy/ Façade Frontdoor Wrapper Filtering, Cleansing, Security Checks ...
  6. 6. 10/12/2015 6 Sustaining Your Architecture THINGS ARE CHANGING: SYSTEMS NEED TO QUICKLY ADAPT TO VARIATIONS!!! Sustaining Your Architecture In my youth…two bad words M and R words Metadata and Reflection
  7. 7. 10/12/2015 7 Sustaining Your Architecture Metadata Sustaining Your Architecture The Power of Metadata Code is data, data is code. Everything is data. And data can drive behavior Meta data simply describes other data. “If something is going to vary in a predictable way, store the description of the variation in a database so that it is easy to change”—Ralph Johnson "Anything you can do, I can do Meta" ”Meta is Beta"
  8. 8. 10/12/2015 8 Sustaining Your Architecture What is Metadata “Data about Data” Document Type Book Authors Erich Gamma, John Vlissides, Ralph Johnson, and Richard Helm Year of Publication 1994 Title Design Patterns: Elements of Reusable Object-Oriented Software Page Count 395 Publisher Addison-Wesley Description Software engineering book describing recurring solutions to common problems in software design... Sustaining Your Architecture Metadata – the Three “M”s Metaobjects Markup Manifest Brian Foote: Metadata, The Good, The Bad, The Ugly Mini-SugarLoaf PLoP, São Paulo, Brazil - September 2011
  9. 9. 10/12/2015 9 Sustaining Your Architecture Metadata – Markup <xsl:stylesheet xmlns:xsl="http://www.w3.org/ 1999/XSL/Transform" version="1.0"> <xsl:key match="/publisher/books/book" use="author-ref/@ref" name="books-by-author"/> <xsl:template match="/"><bestsellers-list> <xsl:apply-templates select="/publisher/authors/author"> <xsl:sort select="sum(key('books-by- author', @id)/sold)"/> <xsl:sort select="last_name"/> </xsl:apply- templates> </bestsellers-list></xsl:template> <xsl:template match="author"> <copy> <name> <xsl:value-of select="last_name"/>, <xsl:value-of select="first_name"/> </name> <total_publications> <xsl:value-of select="count(key('books-by-author', @id))"/> </total_publications> <total_sold> <xsl:value-of select="sum(key('books-by-author', @id)/sold)"/> </total_sold> <rank> <xsl:value-of select="position()"/> </rank> </copy> </xsl:template> </xsl:stylesheet> Sustaining Your Architecture public class Car { // Injectable constructor @Inject public Car(Engine engine) { ... } // Injectable field @Inject private Provider<Seat> seatProvider; // Injectable package-private method @Inject void install(Windshield windshield, Trunk trunk) { ... } } @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) //can use in method only. public @interface Test {…} Metadata – Annotations
  10. 10. 10/12/2015 10 Sustaining Your Architecture JUnit Annotations Annotation Description @Test public void method() Annotation @Test identifies that method is a test method. @Before public void method() Will perform the method() before each test. This method can prepare the test environment, e.g. read input data, initialize the class) @After public void method() Test method must start with test @BeforeClass public void method() Will perform the method before the start of all tests. This can be used to perform time intensive activities such as connecting to or loading data from a database. @AfterClass public void method() Will perform the method after all tests have finished. This can be used to perform clean-up activities for example be used to disconnect to a database @Ignore Will ignore the test method. Useful if the underlying code has changed and the test has not yet been adapted or if running the test takes too long. @Test(expected=IllegalArgume ntException.class) Tests if named exception is thrown. @Test(timeout=100) Fails if the method takes longer then 100 milliseconds Sustaining Your Architecture Metadata – Most use 2*M Markup and Manifest are “good” enough for most successful meta-architectures… Metadata For Online Store Catalog Sweaters (size=(S,M,L,XL), color=(red, green, blue, yellow,…)) Canoes (length=float, width=float, color=(red, brown, blue, silver,…)) Video (name=string, rating=(P,PG,R,…), category=(Action, Drama,…))
  11. 11. 10/12/2015 11 Sustaining Your Architecture Custom Metadata Sustaining Your Architecture AOM and TypeSquare (code example) public class EntityType { private string _typeName; private Type _type; public EntityType (string name, Type type) { _typeName = name; _type = type; } public string Name { get{return _typeName;} set{_typeName = value;} }…} http://www.codeproject.com/ Christopher G. Lasater <entities> <entity name="President" entityType="TopLevelExecutiveType" actualAttributes="EmployeeId" /> <entity name="VicePresident" entityType="ExecutiveType" actualAttributes="EmployeeId" /> …. </entities>
  12. 12. 10/12/2015 12 Sustaining Your Architecture Dynamic Hook Points Eli Acherkan, Atzmon Hen-Tov, David H. Lorenz, Lior Schachter, Rebecca Wirfs-Brock, Joseph W. Yoder Asian PLoP 2012 When building dynamic systems, it is often the case that new behavior is needed which is not supported by the core architecture. One way to vary the behavior quickly is to provide well-defined variation points, called hook-points, in predefined places in the systems where new behavior can be dynamically looked up and invoke at runtime when desired. Can use metadata annotations to look up and invoke! Sustaining Your Architecture Dynamic Hook Points Eli Acherkan, Atzmon Hen-Tov, David H. Lorenz, Lior Schachter, Rebecca Wirfs-Brock, Joseph W. Yoder Asian PLoP 2012
  13. 13. 10/12/2015 13 Sustaining Your Architecture Paving over the Wagon Trail ALIAS: MAKE REPETITIVE TASKS EASIER STREAMLINING REPETITIVE CODE TASKS Create simple examples, templates, & scripts Develop a tool that generates code Identify and use existing tools or frameworks Develop a framework &/or runtime environment Develop a domain-specific language Patterns for Sustaining Architecture PLoP 2012 Paper Sustaining Your Architecture Paving over the Wagon Trail Patterns for Sustaining Architecture PLoP 2012 Paper
  14. 14. 10/12/2015 14 Alternative Computation Styles Imperative Traditional programming Define a series of steps with conditionals and loops to vary sequence Declarative Executable specifications Many options: DSLs (Special language) Decision tables State machines Production rule systems Adaptive Model Systems Domain Specific Languages, Martin Fowler and Rebecca Parsons Architecture of DSL Processing __________ __________ __________ __________ __________ DSL Script parse semantic model generate __________ __________ __________ __________ ______ __________ ______ _________ __________ DSL Script target code optional Domain Specific Languages, Martin Fowler and Rebecca Parsons ✗ DSLs can be a scripting language and work by means of parameterization, They don’t need to have a meta-level but they often do have one!!!
  15. 15. 10/12/2015 15 Sustaining Your Architecture IMPORT EXAMPLE: HOW TO DEAL WITH CLIENT VARIABILITY??? Sustaining Your Architecture Import Orders  Provided a standard format for imports  Lots of duplication  If Clients needed to vary from that format, we had to write or update a complete, self-contained custom import program  Imports that need special processing, or had client specific rules, required to write a complete, self-contained import program
  16. 16. 10/12/2015 16 Sustaining Your Architecture Motivation: Need to Quickly Adapt to Change  Business Rules or Domain Elements are changing quickly:  New calculations for insurance policies and new types of policies offered  Online store catalog with new products and services and rules applying to them  New cell phone product and services…  Need quick ways to develop and adapt to these changing requirements. Sustaining Your Architecture Import Validation [ValidationRule(“Joes Validation Rule")] public class JoesValidationRule : ValidationRule { public JoesValidationRule () : base() { } public override void Validate(ImportContext context) …} Validate Process Core Rules Load and Process Cleint Specific Rules • Dynamic Hook Point
  17. 17. 10/12/2015 17 Sustaining Your Architecture Architectural Practice: Support Data Variability  Separate what changes from stable part of system and provide support for adaptability/flexibility  Integrate new learning into your architecture  Refactoring  Redesign  Rework  Code clean up Sustaining Your Architecture Sustaining an Architecture  Minimize architectural debt: Support the ability to change/adapt what needs to change  Make what is too difficult, time consuming, or tedious easier  Decide at the most responsible moment, not the last possible moment  Learn and evolve Keep the system “livable” for its users and developers
  18. 18. 10/12/2015 18 Sustaining Your Architecture Sustainable Architecture Stewardship  Follow through  Ongoing attention  Not ignoring the little things that can undermine our ability to grow, change and adapt our systems Sustaining Your Architecture Indicators You’ve Paid Enough Attention to Architecture  Defects are localized  Stable interfaces  Consistency  Developers can easily add new functionality  New functionality doesn’t “break” existing architecture  Few areas that developers avoid because they are too difficult to work in  Able to incrementally integrate new functionality
  19. 19. 10/12/2015 19 Sustaining Your Architecture Continuous Inspection Asian PLoP 2014 Paper Sustaining Your Architecture Continuous Inspection Asian PLoP 2014 Paper CODE SMELL DETECTION METRICS (TEST COVERAGE, CYCLOMATIC COMPLEXITY, TECHNICAL DEBT, SIZES, …) APPLICATION SECURITY CHECKS ARCHITECTURAL CONFORMANCE AUTOMATE WHERE YOU CAN!!!
  20. 20. 10/12/2015 20 Sustaining Your Architecture Design Values Supporting Variability  Respect your system’s shearing layers  Understand the rates of what changes  Determine & support who should be able to make changes, when, and at what cost  Make what is too difficult, time consuming, or tedious easier  Create tools, leverage design patterns, use data to drive behavior…  Don’t overdesign!!! (Only design what you need when you need it) Sustaining Your Architecture Resources Adaptive Object Models  www.adaptiveobjectmodel.com  www.metaplop.org Teams That Innovate  www.teamsthatinnovate.org Refactoring www.refactory.com Hillside Patterns www.hillside.net
  21. 21. 10/12/2015 21 Dziękuję!!! joe@refactory.com Twitter: @metayoda

×