AWS Community Day | Midwest 2018 Track 2 Building out a Global Data delivery platform - the business and technical use cases - Neil Dawkins, Scott Vina, and Tim Nettleton, Chicago

1. B U I L D I N G O U T A G L O B A L
D A T A D E L I V E R Y P L A T F O R M
Neil Dawkins

2. AGENDA
C U R R E N T C H A L L E N G E S & S O L U T I O N
T E C H N I C A L I M P L E M E N TAT I O N
C O S T S A N D B E N E F I T S
T E C H N O L O G Y D E E P D I V E
S U P P O R T

3. CURRENT CHALLENGES
Getting out of the Data Center business
Patching and supporting Operating systems
Disk space and network infrastructure issues
I N F R A S T R U C T U R E
Production processes
24x7 support
Ongoing maintenance (lists, re-runs due to data issues etc.)
Troubleshooting issues
O P E R AT I O N A L S U P P O R T
Timeframes
Developer resources
Managing priorities
N E W D E V E L O P M E N T R E Q U E S T S

4. SOLUTION?
Getting out of the Data Center business
Patching and supporting Operating systems
Disk space and network infrastructure issues
I N F R A S T R U C T U R E
Production processes
24x7 Support
Ongoing maintenance (Universe files, re-runs due to data issues etc.)
Troubleshooting issues
O P E R AT I O N A L S U P P O R T
Timeframes
Developer resources
Managing priorities
N E W D E V E L O P M E N T R E Q U E S T S

5. T E C H N I C A L
I M P L E M E N T A T I O N

6. TEAM LOCATIONS
D e v e l o p m e n t c u r r e n t l y d o n e i n C h i c a g o , S h e n z h e n a n d E u r o p e

7. Data Configuration
Data extraction and manipulation
Data delivery
DATA DELIVERY COMPONENTS

8. Management
Console
User
Interface Velo
Enterprise
Account
Management
System
Data APIConfiguration file
Client login
token
token
Data
points &
schedule
info
Data bundles
API
Requests
CONFIGURATION COMPONENT

12. Input & Output
Lambdas
Data API
Delivery
details
tokenData content
Output file
Data
points &
schedule
info
Enterprise
Account
Management
System
Status
message
CONFIGURATION COMPONENT
Configuration file

13. Feed
Messenger
Lambda
Output file
Status
message
Delivery
details
DATA DELIVERY COMPONENT
S3
Output file

14. OPTIMIZING PERFORMANCE
1MB Kinesis restriction resulted in abandoning Kafka/Kinesis solution
Initial AVRO file format was not conducive to larger data sets. API v2 team provide more of a messaging protocol
Initially built uncompressed objects which led to memory issues. Implemented Multi-part S3 upload feature which writes data in 5MB chunks
C H A L L E N G E S
From the clients perspective, they provide a scheduled start time and the production process should begin shortly afterwards as resources are available
Delivery would typically be expected with a reasonable time (approximately 30 minutes)
E X P E C TAT I O N S
On average, files are typically available within a minute
Initial Lambdas are triggered within 15 seconds of receiving the configuration file
API v2 takes 2 – 3 minutes to send data for much larger files
Output Lambdas take less than 2 minutes to generate larger files
W H AT W E A C H I E V E D

15. SUPPORT
Detailed logging
Troubleshoot issues
C L O U D WAT C H D A S H B O A R D S
Pizza tracker
Tracking messages displayed in UI
S N S M E S S A G E S T O C O N V E Y S TAT U S
Level I and II support
Status page for Services
C U S T O M D A S H B O A R D S F O R S E R V I C E S TAT U S

16. COSTS
Transparency around operating costs
Reports to detect unused services
C L O U D H E A LT H R E P O R T S W I T H S E R V I C E C O S T S
More insight into operating costs
Better able to determine profit margins
E S TA B L I S H P R I C I N G M O D E L W I T H M O R E C E R TA I N T Y
Implementation determines costs
Teams are forced to be cost conscious
I N C E N T I V E T O C R E AT E W E L L D E S I G N E D S O L U T I O N

17. BENEFITS
R E D U C E P R O D U C T I O N T I M E
M I N I M I Z E O P E R AT I O N A L C O S T S
R E D U C E C O M P L E X I T Y O F D E L I V E R I E S
I M P R O V E D E L I V E R Y T I M E L I N E S F O R C L I E N T R E Q U E S T S
I M P R O V E D P E R F O R M A N C E F O R I N T E R N A L P R O C E S S E S
R E D U C E O N P R E M I S E I N F R A S T R U C T U R E F O O T P R I N T
I M P R O V E D C L I E N T E X P E R I E N C E W I T H S E L F S E R V I C E M O D E L
F R E E U P D E V E L O P M E N T R E S O U R C E S
R E D U C T I O N O F I N C I D E N T S R E L AT E D T O I N F R A S T R U C T U R E

18. End State is a Fully Integrated and Simple Workflow
STEP TWO
Automated setup of
accounts and assignment of
data licensed for use
STEP THREE
Access to data is controlled
via robust entitlement
allowing for complete
insight into usage
STEP FOUR
Customer defines their data
access via flat file or API
based within a
componentized User
Interface
Integration with
Sales and
Marketing
Technology
STEP ONE
Contract closed in
salesforce kicks off the
onboarding process for
customer
Entitlement and
Account
Management
system shared
across M* Apps
Taps into data
accessible via
Data API v2
Morningstar
Design System
based UX

19. Q U E S T I O N S

20. T E C H N O L O G Y
D E E P D I V E
Technical Implementation – Tim Nettleton