The document discusses anomaly detection in telecommunications using Spark, detailing its importance, methodologies, and application use cases. It outlines the process of defining 'normal' behavior, setting adaptive thresholds for alerts, and constructing a streaming application to analyze real-time data. Key technologies mentioned include Apache Kafka, Spark Streaming, and various machine learning techniques to effectively detect anomalies.

1. © 2015 MapR Technologies ‹#›© 2016 MapR Technologies
Tugdual Grall
Technical Evangelist
@tgrall
Anomaly Detection in Telecom with Spark
Code Motion Amsterdam
12 - May - 2016

2. © 2016 MapR Technologies 2
{“about” : “me”}
Tugdual “Tug” Grall
• MapR
• Technical Evangelist
• MongoDB
• Technical Evangelist
• Couchbase
• Technical Evangelist
• eXo
• CTO
• Oracle
• Developer/Product Manager
• Mainly Java/SOA
• Developer in consulting firms
• Web
• @tgrall
• http://tgrall.github.io
• tgrall
• NantesJUG co-founder
• Pet Project :
• http://www.resultri.com
• tug@mapr.com
• tugdual@gmail.com

3. © 2016 MapR Technologies 3
Agenda
• Introduction
• Anomaly Detection : Why?
• Anomaly Detection : How?
• Use Cases and Demonstration: Telco Sample Application

4. © 2016 MapR Technologies 4
Anomaly Detection

5. © 2016 MapR Technologies 5
Who Needs Anomaly Detection?
Utility providers using
smart meters

6. © 2016 MapR Technologies 6
Who Needs Anomaly Detection?
Feedback from
manufacturing assembly
lines

7. © 2016 MapR Technologies 7
Who Needs Anomaly Detection?
Monitoring data traffic on
communication networks

8. © 2016 MapR Technologies 8
What is Anomaly Detection?
• The goal is to discover rare events
– especially those that shouldn’t have happened
• Find a problem before other people see it
– especially before it causes a problem for customers
• Why is this a challenge?
– I don’t know what an anomaly looks like (yet)

9. © 2016 MapR Technologies 9

10. © 2016 MapR Technologies 10
Looks pretty
anomalous
to me

11. © 2016 MapR Technologies 11
Basic idea:
Find “normal” first

12. © 2016 MapR Technologies 12
Steps in Anomaly Detection
• Build a model: Collect and process data for training a model
• Use the machine learning model to determine what is the normal
pattern
• Decide how far away from this normal pattern you’ll consider to
be anomalous
• Use the AD model to detect anomalies in new data
– Methods such as clustering for discovery can be helpful

13. © 2016 MapR Technologies 13
How hard is it to set an alert for anomalies?
Grey data is from normal events; x’s are anomalies.
Where would you set the threshold?

14. © 2016 MapR Technologies 14
Basic idea:
Set adaptive thresholds

15. © 2016 MapR Technologies 15
99.9%-ile

16. © 2016 MapR Technologies 16
With Spikes
99.9%-ile including spikes

17. © 2016 MapR Technologies 17
Online
Summarizer
99.9%-ile
t
x > t ? Alarm !
x
How Hard Can it Be?

18. © 2016 MapR Technologies 18
Key Steps in Anomaly Detection
• What is normal?
• What will you measure to identify things that are “far” from normal?
• How far is “far”, if something is to be considered anomalous?

19. © 2016 MapR Technologies 19
A lot more….
• Model normal, then find
anomalies
• t-digest for adaptive
threshold
• Probabilistic models for
complex patterns
-
0 5 10 15
−20246810
offset+noise+pulse1+pulse2
A
B

20. © 2016 MapR Technologies 20
https://www.mapr.com/ebook
Learn more about
Machine Learning & Anomaly Detection

21. © 2016 MapR Technologies 21
Yes…
but how do I build such application?

22. © 2016 MapR Technologies 22

23. © 2016 MapR Technologies 23

24. © 2016 MapR Technologies 24

25. © 2016 MapR Technologies 25

26. © 2016 MapR Technologies 26
Data flow and processing
1. Device to Antenna
2. Antenna to main data center
3. Application should:
✓Store the data
✓Analyse/process the data
✓Detect Anomalies and alert IT

27. © 2016 MapR Technologies 27
Data flow and processing
1. Device to Antenna
2. Antenna to main data center
3. Application should:
✓Store the data
✓Analyse/process the data
✓Detect Anomalies and alert IT
➡ Pure mobile GSM, LTE, 5G, …
➡ Streaming Technology
➡ Big Data Storage
➡ Distributed Processing
➡ Machine Learning

28. © 2016 MapR Technologies 28
Architecture
Streams
HDFS/MapR-FS
HBase/MapR-DB JSON
Streaming
Streaming
SQL Engine
Analytics
JDBC/ODBC

29. © 2016 MapR Technologies 29

30. © 2016 MapR Technologies 30
• Cluster Computing Platform
• Extends “MapReduce” with
extensions
– Streaming
– Interactive Analytics
• Run in Memory

31. © 2015 MapR Technologies ‹#›@tgrall
Spark components
Spark SQL
Spark Streaming
(Streaming)
MLlib
(Machine Learning)
Spark Core (General execution engine)
GraphX
(Graph Computation)
Mesos
Distributed File System (HDFS, MapR-FS, S3, …)
Hadoop YARN

32. © 2016 MapR Technologies 32
Spark Resilient Distributed Datasets “RDD”
Sensor RDD
W
Executor
P4
W
Executor
P1 P3
W
Executor
P2
sc.textFile P1
8213034705,
95, 2.927373,
jake7870, 0……
P2
8213034705,
115, 2.943484,
Davidbresler2,
1….
P3
8213034705,
100, 2.951285,
gladimacowgirl,
58…
P4
8213034705,
117, 2.998947,
daysrus, 95….

33. © 2016 MapR Technologies 33
Spark Resilient Distributed Datasets
Transformation
Filter()
Action
Count()
RDD
newRDD
Value

34. © 2015 MapR Technologies@tgrall
Transformations
• Process an RDD, returns an RDD
• Examples :
• map() : one value => another value
• mapToPair() : one value => a tuple
• filter() : filters values/tuples on a given condition
• groupByKey() : groups values by key
• reduceByKey() : aggregates values by key
• join(), cogroup(), … : joins RDDs

35. © 2015 MapR Technologies@tgrall
Actions
• Process an RDD, returns a value
• Examples :
• count() : counts number of items in dataset
• first() : returns first entry
• take(n) : returns array of the n first elements
• foreach() : applies a function on each element
• collect() : returns all elements
• saveAsTextFile() : saves in files each element

36. © 2015 MapR Technologies@tgrall

37. © 2015 MapR Technologies@tgrall
Apache Kafka
• Feeds of messages are organised in
topics
• Processes that publish messages are
called producers
• Processes that subscribed to topic
and process messages are
consumers
• A Kafka cluster is made of one or
more brokers (== node)

38. © 2016 MapR Technologies 38
Broker 1
Topic A Topic B
Broker 2
Topic A Topic B
Broker 3
Topic A Topic B
Producer
Producer
Producer
Consumer
Consumer
Consumer

39. © 2016 MapR Technologies 39
What is Spark Streaming?
• Enables scalable, high-throughput, fault-tolerant stream
processing of live data
• Extension of the core Spark
Data Sources Data Sinks

40. © 2016 MapR Technologies 40
Spark Streaming Architecture
• Divide data stream into batches of X seconds (micro batching)
• Called DStream = sequence of RDDs
Spark
Streaming
input data
stream
DStream RDD batches
Batch
interval
data from
time 0 to 1
data from
time 1 to 2
RDD @ time 2
data from
time 2 to 3
RDD @ time 3RDD @ time 1

41. © 2016 MapR Technologies 41
Demonstration
https://github.com/mapr-demos/telco-anomaly-detection-spark

42. © 2016 MapR Technologies 42
Sample Code
• Universe dealing with Antenna & Users (Akka / Actors)
• Antenna Send Data to Spark (Kafka/Streams & Spark Streaming)
• Aggregate CDR Data by Tower (Spark & MapR DB)
• Analyse Tower Behaviour and Send Alerts when needed (Spark &
Kafka/Streams)

43. © 2016 MapR Technologies 43
Conclusion
• Build a streaming based application to capture data in real time
• Apache Kafka / MapR Streams
• Store data into a scalable data store
• MapR-FS/DB, Hadoop, NoSQL with Spark Support
• Use Spark & Spark Streaming to process data in real time
• Run Analytics jobs using Spark or SQL on “Hadoop” (Apache
Drill)

44. © 2016 MapR Technologies 44
Interesting Skills to Add to your Resume
• Apache Kafka
• Apache Spark
• NoSQL
• Machine Learning Technics

45. © 2016 MapR Technologies© 2016 MapR Technologies@tgrall 45
IoT : Racing Cars
Producers Consumers
sensors data
Real Time
Analytics
https://github.com/mapr-demos/racing-time-series

46. © 2016 MapR Technologies 46
Free eBooks
http://mapr.com/ebook

47. © 2016 MapR Technologies 47

48. © 2016 MapR Technologies 48
Q&A
@tgrall maprtech
tug@mapr.com
Engage with us!
MapR
maprtech
mapr-technologies