Data Supply Chain Pipeline

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stephen Moon
Senior Solutions Architect, Department of Defense Team
Data Supply Chain Pipeline

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Supply Change Management?
Wikipedia... “In commerce, supply chain management (SCM), the
management of the flow of goods and services, involves the movement
and storage of raw materials, of work-in-process inventory, and of
finished goods from point of origin to point of consumption.
Interconnected or interlinked networks, channels and node businesses
combine in the provision of products and services required by end
customers in a supply chain. Supply-chain management has been
defined as the "design, planning, execution, control, and monitoring of
supply chain activities with the objective of creating net value, building a
competitive infrastructure, leveraging worldwide logistics, synchronizing
supply with demand and measuring performance globally.”

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer Use Case
• An organization has tens, hundreds, or thousands of
disparate, heterogeneous systems, applications, or
devices across multiple organizational domains.
• Each organizational domain has multiple systems of
record storing data about the same entities and it’s
attributes
• Leaders need to be able to ask single-domain and multi-
domain questions and receive a single, accurate answer
• For example… what is my force readiness if a conflict
arises in <region>?
• This is a multi-domain question

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is the Importance of Data?
• Data is a critical organizational asset, not an IT function
asset
• Data + Context + Relationships = Information
• Information provides strategic, tactical, and operational
advantages by enabling execution of the mission more
rapidly with greater agility and precision than the
adversary.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Handling
! Without good quality data, the best modeling techniques in the world won’t deliver accurate results
• Prepare raw data for model training
• Training Sets, Development Sets, Test Sets
• Historical event data required to build a machine learning model should be stored in the data wareho
Importance of Data for Machine / Deep Learning

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Supply Chain Pipeline Mission Statement
Democratize data by creating an accurate and consistent
360° view of the organization with the purpose of
providing actionable strategic, tactical, and operational
insight in order to enable execution excellence.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Supply Chain Pipeline Operating Model
Current State Future State
Ross, Jeanne W, et al. Enterprise Architecture As Strategy: Creating a Foundation for Business Execution. Harvard Business Review
Press, 2006.
https://www.amazon.com/dp/B004OC07EE/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume
Velocity
Variety
Veracity
Value
Kilobytes (KB)
Megabytes (MB)
Gigabytes (GB)
Terabytes (TB)
Petabytes (PB)
Time-based
Event-driven
────────
KB/s
MB/s
GB/s
Structured | Semi-structured | Unstructured
Accuracy
Authenticity
Consistency
Reliability
Decision Making
Strategic
Tactical
Operational
The Five Vs of Data

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collect
&
Process
Organize
&
Catalog
Explore
&
Discover
Analyze
&
Report
Suppliers Consumers
Search
&
Prepare
Data Supply Chain Pipeline (DSCP)

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Name Build It and Make Them Come
Statement Customers should by able to access the data where it lives regardless of their own location
Name Minimal Disruption
Statement Minimize/Eliminate disruption to data producing systems, applications, or devices
Name Configure-to-Customize
Statement Leverage configurable components to meet at least 80% of the requirements
Name Decoupling
Statement Pipeline stages are independent of one another
DSCP Architecture & Design Principles

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Architecture

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Design

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Minimal Viable Product (MVP)
Initial Operating Capability (IOC)
Collection & Processing

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Collection
! There is no “tool” for collecting data from disparate,
heterogeneous suppliers
• Focus on cultivating the organization competencies and
the processes for engaging with suppliers
• Build tiger teams who understand the organizational
domains
• Develop templates for Memorandums of Understanding
(MoU) and Interface Control Documents (ICD)
• Use AWS Cloud Services to build and enable the right
technology solutions for data collection use cases

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Collection
The result will be a small set of common Collection &
Processing patterns that can be standardized, automated,
and scaled to hundreds or thousands of suppliers on AWS’
Cloud Services.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture Pattern Design Pattern Methodologies
Extraction, Transformation, & Loading Change Data Capture
• Push (Source initiated)
• Pull (Target initiated)
• Transaction Log (Synchronous /
Asynchronous)
• Slowly Changing Dimensions (Type 0:6)
Enterprise Application Integration Publish / Subscribe
• Topic-based
• Content-based
DSCP Collection Architecture & Design Patterns

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Interest
Influence
Control
Control
Applications/Systems which are owned and/or
directly managed by the collecting organization
Influence
Applications/Systems of which the collecting
organization is an internal or external stakeholder but
does not own or manage the application/system
Interest
Applications/Systems of which the collecting
organization has a concern for the data but does not
have control or influence over the application/system
Why is this important?
Determines how data is going to be collected!
DSCP Collection, Circles of Concern

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hard Rules Soft Rules
Quality rules that are independent of
business value or meaning.
Example 1
A social security number must be
numeric and contain exactly 9 digits.
Example 2
An address must be valid per the USPS.
Policy rules based on context that add
or alter business value or meaning.
Example 2
A mailing address for prescription
drugs cannot be a P.O. box.
Example 1
A social security number and date of
birth must be a unique combination.
DSCP Processing, Business Rules

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Technical
L0

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Technical
L1

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Technical L2,
Structured

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DSCP Next Steps
1) Engage your AWS Solutions Architect
2) Develop a set of questions
3) Identify the information required to answer the questions
4) Identify suppliers of the relevant data
5) Build the solution platform
6) Collect & Process the data
7) Start answering questions

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank You!

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
moonstep@amazon.com
Senior Solutions Architect
Department of Defense (DoD)
Worldwide Public Sector (WWPS)
Amazon Web Services (AWS)
Stephen Moon