At the StampedeCon 2015 Big Data Conference: As a frequent recipient of the J.D. Powers award for excellence in customer service, T-Mobile takes great pride in the quality of care that we provide our customers. As smartphone technologies advance (and fragment), the challenge of providing quality technical support can be daunting. To address this challenge, T-Mobile is reinventing many of its traditional practices and embracing DevOps, cloud deployment and lambda architecture. Specifically: * Cassandra for fast and consistent writes (at scale), as well as low-latency reads * Apache Spark and EMR for processing data archived in S3 * Kafka for flexibility in data ingestion * Chef and CloudFormation Templates to automate deployments * Graphite and Riemann for monitoring The goals of this presentation are: * showcase how these technologies are helping T-Mobile be * successful in addressing these business challenges * share tactics for tackling customer preference management and data collection transparency * specific “lessons learned” while migrating to NoSQL, Big Data and The Cloud.

1. FROM SQL TO NOSQL
Timothy Shelton, T-Mobile USA
StampedeCon, 2015
1

2. Outline
• BUSINESS GOALS: Why do we need Big Data?
• TECHNOLOGY: What did we build?
• PRIVACY: Customer preference management
• LESSONS LEARNED: Toes were stubbed.
• Q&A
2

3. WHY DO WE NEED BIG DATA?
3

4. Business Goals
4
We want you to have a great experience with your device, and we want
to provide world-class service to quickly resolve any device problems.

5. Example Use Case
Symptom: “Battery keeps going dead”
Possible Root Causes:
• Bad application
• Misconfigured application
• Misconfigured device
• Incorrect charger
• Improper expectations
• Faulty hardware
5

6. Tackling Complexity
At the beginning of 2015, there
were an estimated 1.5 million
applications in the Google Play
Store
6
1Q14 4Q14 1Q15
Smartphones 6.9 8.0 8.0
Non-Smartphones 0.5 0.6 0.5
Mobile Broadband 0.1 0.4 0.3
Total Company 7.5 9.0 8.8
(Device Sales, Millions)
Challenge: Accurately
diagnose device problems,
educate customers, solve
problems quickly.
We sell a ton of devices:

7. By the numbers….
7
30,000
Average number of
technical calls received
daily
32
Average number of
customer-installed
applications per device
56,836,000
Total customers, as of
1Q2015 (combined)
100%
Percentage of
customers expecting
problem resolution

8. Solution: BIG DATA!
8
Device
GOAL
Provide Customer Care
Representatives visibility
into the device so they are
able to resolve problems.
REQUIREMENTS
1. Ability to gather
metrics from a device
2. Ingest the metrics
3. Store for a period of
time
4. Fast data retrieval

9. TECHNOLOGY
9

10. Technology: Version 1
10
Device SQL
Server
First attempt: RDBMS to store data.
Findings:
X Tables had to be de-normalized and indices
removed to maintain write performance
X Reads were barely achieved by batch
processing snapshots of the database
Effectively deployed a write-only database.
? ?
?
?

11. Technology: Version 2
11
Device
C*
Second attempt: Cassandra!
Findings:
 Stellar write performance
 Stellar read performance
 Relatively low TCO
X Application misused [beta] C* driver
X Application not designed to scale
X Heterogeneous architecture
(.NET/Linux)

12. Technology: Version 3
12
Device C*
Evolution: Lambda Architecture!
Goals:
 Enables multiple consumers of incoming data
 Allows for near-realtime processing (Spark Streaming)
 Archival to S3 for offline processing (Spark)
 Layered architecture takes advantage of Cloud

13. Culture: Fast and Lean
• Preference is rapid
prototyping and
proofs-of-concept
• We set aggressive
dates, then iterate
• Cloud deployments
align nicely with our
culture
13
Embrace automation
• CloudFormation
Templates to reduce
risk of error and
increase velocity
• Chef for deployment
and configuration

14. CUSTOMER PREFERENCE
MANAGEMENT
14

15. Transparency and Choice
• Clearly describe what
is being collected
• Be transparent about
why it is being
collected
• What are the intended
uses of the data?
• Be clear with whom
the data will be shared
(if anyone)
• Allow the customer to
opt-out
• Opt-in for Location-
Based Services
15

16. Transparency: How not to do it
16
“It’s in our privacy
policy!”

17. LESSONS LEARNED
17

18. Scale Thy Monitoring
18
This looks bad!
OMG, traffic is
being rejected!
What Worked What Didn’t
Chart all the things! In this instance, statsd was being
overloaded with traffic

19. Organize Thy Data
19
What Didn’t
No temporal organization made
querying this data more difficult every
day
What Worked
Archive all the things!

20. Cassandra: df -u
• Monitor your capacity
• Leave enough disk for compaction
• Disabling compactions is a terrible strategy
20

21. QUESTIONS?
21

22. Thanks!
T-Mobile is actively recruiting engineers – come
change the world with us!
22
@TimothyAShelton
http://www.t-mobile.com