We documented our journey into the Cloud with AWS

1. TDWI LAS VEGAS STRATEGY SUMMIT
tdwi.org/StrategySummitVegas/Presentations
Analytics at Cloud Scale
AVOID MIGRATION PAIN

2. Disclaimer/Goals
 I have no allegiance to any tech or tech company
 When it comes to our tech choices, good or bad, YMMV
 This was our journey, there are many *right* paths
 My goal is to highlight our wins and losses to share learnings
2

3. Our Business
Case
 Inability to scale quickly
 Restrictive innovation environment
 Difficulty managing budget
 Shrink the growing DevOps chasm
 Re-focus our tech ops talent pool
 Attraction/Retention of talented people
3

4. What We Wanted
 Elasticity – think pay-per-use
 Scalability – think quick upgrade/downgrade
 Eliminate Pets / Create Cattle
 Increase profitability and smooth spend
4

5. Phase 1: Lift and shift
GET OUT OF YOUR DATA CENTER
5

6. Our Tech Stack - Then
Prod, Stage, Test, Dev
 Compute: Racks of Dell R910’s 32 x 1TB x
16TB
 All Pets
 NLB’s, switches, SAN’s,
 Database: SQL Server
 Reporting: Microstrategy
 Frameworks: Angular, .NET (v4.6+)
6

7. Preparing for Lift
and Shift
 Secure your budget
 Choosing a cloud provider
 Design communications infrastructure
 Understanding the EC2/Series instance
types
 Server inventory/collapsing server
environment
 Identify Pets and Cattle and build out
server templates
7

8. Preparing for
Lift and Shift
 Zones/Regions
 Create backout plan
 Time it right: Contracts,
penalties, CapEx
 Phased the approach
over months
 AWS Well Architected
Framework and Azure
Architecture Center
8

9. Wins & Losses
9
Did Great!
 Server discovery
 Combining instances
 Tried many instance types
 Supportive executive team
 Timed it well
Not Awesome
 Hiring ahead
 Getting in front of training

10. Execution
10
 Test migration with most complex system
 Redeploy, don’t move
 Leave a couple of weeks between phases
 Continually look for cattle
 Consider ethical destruction of servers

11. Wins & Losses
11
Did Great!
 Set up BDP early -
 Reduced our footprint
 Dramatically destressed our System
Engineering team
 Excitement → engagement
Not Awesome
 Cloud engineering and Software
engineering weren’t working together
all the way through
 Didn’t embrace DevOps early
 Didn’t set up monitoring early
 Inexperience created tech debt

12. Phase 2: Rearchitect in
place
USING CLOUD PROVIDER CLOUD NATIVE SERVICES, TOOLS,
AND PLATFORMS
12

13. What did we
build/update/r
eplace?
 Introduce DevOps to teams – everything is
code
 Convert .NET to .NET Core
 Built data lake to burst load
 Replace ETL with S3, Step Functions,
Lambda, and ECS
 Replace SQL Server with Aurora, EMR,
DynamoDB
 Tune our infrastructure – Size & HoO
13

14. Wins & Losses
14
Did Great!
 moved away from using SQL Server/SSIS
for ETL; switched to using Step Functions,
Lambda, ECS, RedShift, and S3
 Re:Invent and MS Build for your technical
teams
 DevOps
Not Awesome
 Adopting new tech when it’s first
announced like RedShift or SQL DW – do
your due diligence
 Load balancer costs add up, use ALB’s not
ELB’s
 Didn’t port everything fully
 No approach to tagging

15. Monitoring
 SumoLogic
 LogicMonitor
 CloudWatch
 CloudHealth
 Sentry
15

16. Wins & Losses
16
Did Great!
 Defining metrics
 Getting our tooling in place
 Displaying it everywhere
 Using graphs to share stories
Not Awesome
 Not having our monitoring in place fast
enough

17. Phase 3: Rearchitect
again
LOOKING AT WHAT’S AVAILABLE IN THE MARKET
17

18. Cloud Scale
Tooling and
Processes
 Fix all of our tagging
 Set HOO for all instances
 Readjust instance types after Re:Invent
 HOOPA
 Snowflake
18

19. HOOPA
19

20. Snowflake Performance
20

21. Snowflake Financials
21

22. Wins & Losses
22
Did Great!
 Snowflake was evolutionary for us.
Faster execution times, lower costs, cost
directly correlated to usage
 put multiple endpoints on a single alb
Not Awesome
 Tools that didn’t work out the way we
hoped – RedShift, EMR, SWF

23. Our Tech Stack - Now
 Compute: EC2, ECS (Elastic Container Service) / docker,
Lambda
 Workflows: Step Functions, SWF*
 Database: Snowflake, SQL Server, Aurora PostgreSQL,
Aurora MySQL, Redshift*, DynamoDB, EMR (PySpark)*
 Eventing/Streams: SQS, SNS, Kinesis
 Reporting: MicroStrategy
 Frameworks: Angular, .NET (v4.6+, core, standard)
 Conf Mgmt/Orchestration: Chef, Ansible, CloudFormation
23

24. Contact
Information
Paul Birkbeck, VP, Software Engineering
pbirkbeck@spscommerce.com
linkedin.com/in/paulbirkbeck/
24