Let's Turn real User Data into a Science The next generation of testing best practices

1. September 14, 2015
“Let’s turn Real User Data into a Science!”
Dan Boutin – Senior Product Evangelist

2. © 2014 SOASTA. All rights reserved. March 3, 2015 2
Agenda
• Who are we, and why are we here?
• Performance Testing over the Years – History Lesson
• Now Let’s Talk Architecture!
• Trade-Offs & Deep Dive
• The Result!
• Session II: It’s Your Turn!

3. © 2014 SOASTA. All rights reserved. September 17, 2015 3
Who are We and Why are we here?
• We are….Performance Experts.
• …with a Data Science component
• We collect Billions of Real User Beacons
• …What’s a beacon?
• …Where do we get it?

4. © 2014 SOASTA. All rights reserved. September 17, 2015 4
Who are We and Why are we here?
• How Are We Different?
• User Experience Beacon Collection
All Of The Data
KG
All Of The Detail
All Of The Time
Kept Forever
• What do we do with these Billions of Real User Beacons?
• …we keep them….which could have been a challenge….

5. © 2014 SOASTA. All rights reserved. September 17, 2015 5
100 Billion
User Experiences Tested
10 Million
Tests Performed
Actual CloudTest
view
Who are We and Why are we here?

6. © 2014 SOASTA. All rights reserved.
How did all this start?
o 1989
o 1995
o 2007!

7. © 2014 SOASTA. All rights reserved.
Fear Factor
o “We don’t test in production.”

8. © 2014 SOASTA. All rights reserved.
Automated Grid Provisioning
Your environment must be flexible & scalable

9. © 2014 SOASTA. All rights reserved.
You need a Kill Switch – No Fear Factor!

10. © 2014 SOASTA. All rights reserved.
Real-time Performance Analysis
You need drill down by Load Generation Location

11. © 2014 SOASTA. All rights reserved.
Detailed Error Analysis
You need detailed error analysis - LIVE

12. © 2014 SOASTA. All rights reserved.
Multi-Test-Run Comparison
Compare results of a LIVE test with previous test executions
You need to know: Are we better than last time?

13. © 2014 SOASTA. All rights reserved.
Detailed Transaction, Page, and URL Analysis
• Detailed Transaction and Page Analysis of Web and Mobile Load Tests
• Detailed URL Analysis of Web and Mobile Load Tests
You need web & mobile analysis

14. © 2014 SOASTA. All rights reserved.
Run Globally Distributed Load Tests with Akamai
Your analytics should have visibility into your CDN

15. © 2014 SOASTA. All rights reserved.
Detailed Page Component Breakdown
Your analytics should have visibility into your CDN

16. © 2014 SOASTA. All rights reserved.
Reflects growth in cloud hours – Amazon only! (17 other providers!)
7 Year Growth of cloud testing: SOASTA & Amazon
….goodbye Fear Factor…

17. © 2014 SOASTA. All rights reserved.
Testing in Production – Why Not?
o What is the value added?
• CDN Tests (Not configured to serve up new content)
• Batch Jobs are not present in the lab
• Misconfigured App & Web servers
• Thread & Connection Pool settings
• Bandwidth Constraints
• Radically different performance on different database sizes

18. © 2014 SOASTA. All rights reserved.
Testing in Production – Why Not?
o So what should the process look like?

19. © 2014 SOASTA. All rights reserved.
o So what should the process look like?

20. Test Takeaway:
Building the tests

21. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015
Fix Your process! No more outdated test creation

22. © 2014 SOASTA. All rights reserved.
o So what should the process look like?

23. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015
Analyze the most common session paths of real
users

24. © 2014 SOASTA. All rights reserved.
How Do Users Move Through Your Site?

26. © 2014 SOASTA. All rights reserved.
New Way to Pinpoint Performance Problems

27. © 2014 SOASTA. All rights reserved.
Test Takeaway
What did we learn?
Revenue
Brand
Competitive advantage

28. © 2014 SOASTA. All rights reserved.
o So what should the process look like?

29. mPulse
What’s a Beacon?
www.w3.org/TR/Beacon
Total Beacons Collected since 6/2013:
~ 85 Billion
Run rate over 3B per week and growing
Projected ~ 175B by 1/1/166

30. Big Data Challenges
Data Scientists spend too much time ‘data wrangling’
“Data scientists, according to interviews and expert
estimates, spend from 50 percent to 80 percent of their
time mired in this more mundane labor of collecting and
preparing unruly digital data, before it can be explored for
useful nuggets.”
NY Times – August 17th, 2014

31. Big Data Challenges
Building a data science platform is very difficult
Infrastructure
•Choosing big data technologies and setting up a cluster can easily take 9
months or more
Data Pipeline
•Building a high performing big data schema requires specialized skills
•Extracting, transforming, and loading of data (data wrangling) is an
enormous time sink and a poor use of data scientists time
Analysis and Workflow
•Figuring out how you can ask questions of the data and how to visualize the
results takes time that data scientists should be using to generate actionable
insights from their studies

32. Julia Language & iJulia Notebook UI
Julia is a rising star in scientific programming
processing speed
support for parallel processing
compatibility with 400+ prebuilt statistical packages
large number and growing number of visualization libraries.
Trade-Offs & Deep Dive

33. Julia Language & iJulia Notebook UI
www.julialang.org
processing speed
support for parallel processing
compatibility with 400+ prebuilt statistical packages
large number and growing number of visualization libraries.
Where can I find Julia?

34. Trade-Offs & Deep Dive

35. © 2014 SOASTA. All rights reserved. September 17, 2015 35
Trade-Offs & Deep Dive
o Amazon Redshift is a fully managed, petabyte-
scale data warehouse service in the cloud.
• Columnar Database
• Extremely fast query times
• Attractive Economics

36. © 2014 SOASTA. All rights reserved. September 17, 2015 36
Now Let’s Talk Architecture

37. Data Science WorkbenchData Science without the data wrangling, and much more
Infrastructure
Data
PipelineAnalysis and Workflow
• Data Science Workbench comes with the
state-of-the-art technology you need to
analyze your customer experiences
• All of the real user beacon data is loaded into
Data Science Workbench into a highly
optimized schema ready for analysis
• Data science is done with Julia, a remarkably
fast and in-memory solution for analyzing huge
data-sets
• Access to an ever growing library of analysis
functions and visualizations based on
SOASTA’s and our customers’ expertise

39. © 2014 SOASTA. All rights reserved. September 17, 2015 39
The Result!
• Every customer beacon unpacked, transformed and loaded nightly by
SOASTA into a SOASTA designed Schema in Amazon Redshift. This
process designed, supplied and supported by SOASTA
• Amazon Redshift is an extremely inexpensive and powerful BIG DATA
database that can scale to almost 2 Petabytes in size. Amazon
estimates compute and storage costs of $1,000/TB/month for our
implementation
• An online, interactive explore, discover and develop interface based on
the Julia scientific programming language developed at MIT and the
iJulia Notebook UI
• SOASTA developed Functions & Statistical Models

40. Let’s take a look at what we have!
A peek at how we use these
Technologies

41. Our Data Science
Workbench

42. Discussion?
Our Data Science
Workbench

43. Our Data Science
Workbench

44. Our Data Science
Workbench

45. Let’s look deeper inside the DSWB

46. The future is here!

47. © 2014 SOASTA. All rights reserved.
Trivia Questions on Technology that apply today
@DanBoutinSOASTA

49. September 14, 2015
“Let’s turn Real User Data into a Science!”
Dan Boutin – Senior Product Evangelist