Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kappa architecture in the telecom industry

288 views

Published on

High-volume event streams (traditional network data, media, IoT sensor data, activity events on social media, etc.) are becoming widespread in the telecom industry. In particular, live analysis of telco log files and performance metrics allows network operators to observe the status of the system and identify possible problems using online aggregations and machine-learning algorithms. (Offline batch analysis of streams using tools like MapReduce is often too slow to respond to things happening right now; hence, it is not the best choice.)

Ignacio Manuel Mulas Viela and Nicolas Seyvet demonstrate an analytics pipeline setup for a telco use case that processes an unbounded dataset of logs and performance metrics. Raw data, logs, and cloud telemetry information are extracted from a production cloud infrastructure using Collectd, Openstack Ceilometer, and Logstash. This is piped into a distributed messaging system, Kafka, then analyzed by Apache Flink—a distributed stream analysis framework that is capable of analyzing thousands of messages per second, extracting insights that can be monitored by humans—and visualized using the ELK (Elasticsearch, Logstash, Kibana) stack.

Ignacio and Nicolas discuss the challenges and benefits of building an analytics pipeline following the Kappa architecture paradigm using the aforementioned tools and demonstrate Kappa’s value through an example use case. The use case analyzes and extracts statistical information from a stream of data and uses machine-learning techniques to develop an advanced anomaly detector, using two online machine-learning algorithms implemented on top of Flink: the online k-means detector and the Bayesian detector.

Published in: Data & Analytics
  • Be the first to comment

Kappa architecture in the telecom industry

  1. 1. Kappa Architecture In The Telco Industry Ignacio Mulas Viela Nicolas Seyvet
  2. 2. Ericsson Internal | 2011-10-19 | Page 4 Once Upon A Time… Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 “I want an advanced real-time analytics system to monitor my cloud infrastructure.” … By your most precious client
  3. 3. Ericsson Internal | 2011-10-19 | Page 5 › Data source – Events (metrics, logs) from physical and virtual servers › Analytics: – Real-time – Statistical analysis – Anomaly or novelty detection High Level View … Flink-forward | Ignacio Mulas | 12-October-2015 Data source Analytics
  4. 4. Ericsson Internal | 2011-10-19 | Page 6 › Bounded  A start and an end Finite, ingestion stops › Unbounded  A start but no end Infinite, ever-growing Data Set Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 t3 t2 t1 t0… tn t3 t2 t1 t0…t∞ Unbounded Bounded
  5. 5. Ericsson Internal | 2011-10-19 | Page 7 › Twitter’s Nathan Mars › But – Two independent pipelines – Complex maintenance – Complex merge Lambda Architecture Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  6. 6. Ericsson Internal | 2011-10-19 | Page 8Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  7. 7. Ericsson Internal | 2011-10-19 | Page 9 Kappa Architecture Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  8. 8. Ericsson Internal | 2011-10-19 | Page 10 › New model to abstract data processing – Millwheel, Spark Streaming, Dataflow, Stratosphere (Flink) › Stream engines › Correctness - Strong consistency - Exactly-once-processing › Resilience, fault tolerance › Tools that can deal with time * › APIs The (Short) Evolution Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  9. 9. Ericsson Internal | 2011-10-19 | Page 11 Principles Kappa Architecture Everything is a stream Immutable data sources Single analytics framework Stream replay Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  10. 10. Ericsson Internal | 2011-10-19 | Page 12 › Stream representation – Unbounded dataset composed by a sequence of events › Data pipeline: – Sequence of transformations on an unbounded data set that generates another set with more insightful data – UNIX pipes Basics Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 … Pub/Sub
  11. 11. Ericsson Internal | 2011-10-19 | Page 13 Our Stack Kafka Elastic Search Kibana Flink Analytics job 1 Analytics job 2 … raw data results job 1 … … Data sources Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 …
  12. 12. Ericsson Internal | 2011-10-19 | Page 14 First Data Pipeline Raw data Statistical analysis DashboardEnriched data Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  13. 13. Ericsson Internal | 2011-10-19 | Page 15 › Event time, which is when an event occurred › Processing time, which is when an event is observed in the system Time Event time Processingtime reality skew Time drifts Unordered events Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  14. 14. Ericsson Internal | 2011-10-19 | Page 16 Event Time e0e1e2e3 … t0t1t2t3 <tp0,e0><tp1,e1><tp2,e2><tp3,e3> <te0,e0><te1,e1><te2,e2><te3,e3> EventTimeExstractor() enableTimestamps() <te0,e0><te1,e1><te2,e2><te3,e3> w2 w1 w0 window() Flink-forward | Ignacio Mulas | 12-October-2015 Execution time Te0 + window + watermark Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 e: event tp: processing time te: event time
  15. 15. Ericsson Internal | 2011-10-19 | Page 17 2nd Client meeting… Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 “I want an advanced real-time analytics system to monitor my cloud infrastructure.” … By your most precious client It is nice, but… I cannot look at thousands of numbers simultaneously, can you do better?
  16. 16. Ericsson Internal | 2011-10-19 | Page 18 › Machine learning – Automatically detect anomalies in the infrastructure – Learn using raw and advanced metrics › … add a new transformation to my unbounded data! Advanced Data Pipeline … Stats ML analyticsData source Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  17. 17. Ericsson Internal | 2011-10-19 | Page 19 › Unsupervised machine learning › Create a statistical model for “normal” behavior – Poisson: count-based parameters – Gaussian: value-based parameters › Model adapts over time Bayesian Detector OK ANOMALYANOMALY Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016
  18. 18. Ericsson Internal | 2011-10-19 | Page 20 Log-Frequency Novelty Detector … … Frequency_i+1 Frequency_2 Frequency_n Phase 1: LEARN! Phase 2: DETECT! … Frequency_1 OK NOK Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 Time window Events … … … … History
  19. 19. Ericsson Internal | 2011-10-19 | Page 21 Multi-Variable Detector t0t0t0 hk hi hm … if… .keyBy(host) -slave Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 … … …
  20. 20. Ericsson Internal | 2011-10-19 | Page 22 Improved Data Pipeline Raw data Bayesian novelty detector Dashboard Anomalies Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 Statistical analysis Enriched dataRaw data
  21. 21. Ericsson Internal | 2011-10-19 | Page 23 3rd Client meeting Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 “I want an advanced real-time analytics system to monitor my cloud infrastructure.” … By your most precious client Great! I can now spot when and where changes occur … I´ll buy it! 
  22. 22. Ericsson Internal | 2011-10-19 | Page 24 › Tools, abstractions and APIs unifying stream/batch › Consistency, resiliency, fault-tolerance › Event time handling › Kappa architecture simplifies Big Data – One stack, many pipelines (batch/stream) – Flexible/extensible architecture › Machine learning can be applied on unbounded data sets – Treated as a complex transformation – Some caveats Summary Flink-forward | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 12-October-2015Flink Meetup | Ignacio Mulas | 26-November-2015Strata London | Ignacio Mulas & Nicolas Seyvet | 3 – June – 2016 Stream Batch Καρρα
  23. 23. Please, feel free to contact us if you have suggestions/comments/questions ignacio.mulas.viela@ericsson.com / @ immulvi nicolas.seyvet@ericsson.com / @NicolasSeyvet Thank you!

×