Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ignacio Mulas Viela – Kappa Architecture in the Telecom Industry

7,878 views

Published on

Flink Forward 2015

Published in: Technology

Ignacio Mulas Viela – Kappa Architecture in the Telecom Industry

  1. 1. Kappa architecture in the telecom industry Ignacio Mulas Viela Ericsson Research
  2. 2. Ericsson Internal | 2011-10-19 | Page 2 › Who? › What? › Why? › How? – Architecture – Time handling › Demo on local machine › Learning › Q&A Proposed agenda Flink-forward | Ignacio Mulas | 12-October-2015
  3. 3. Ericsson Internal | 2011-10-19 | Page 3 › Ericsson Research – Management of Complex Systems – Analytics technologies applied to the ICT industry › EIT ICT EUROPA project › Contributors: Who? Flink-forward | Ignacio Mulas | 12-October-2015
  4. 4. Ericsson Internal | 2011-10-19 | Page 4 ericsson MOBILE INFRASTRUCTURE AND CLOUD OPERATIONS & BUSINESS SUPPORT SOLUTIONS TELECOM SERVICES TV PLATFORMS 65,000 2.5 billion 1 billion Subscribers managed by us Subscribers supported by us Services professionals
  5. 5. Ericsson Internal | 2011-10-19 | Page 5 One day in the life of a medium sized network (~10M customers) Web pages 700,000,000 Videos 40,000,000 Radio sessions (RAB) 120,000,000 Handovers (HSDSCH-CC) 300,000,000 Internet sessions (PDP) 66,000,000 Sum data 10à100 TB/day Real-time data rate 100,000à1,000,000 events/second +200 more types of events Flink-forward | Ignacio Mulas | 12-October-2015
  6. 6. Ericsson Internal | 2011-10-19 | Page 6 Why? KAPPA ARCHITECTURE Everything is a Stream Immutable data sources Single analytics framework Stream replay Flink-forward | Ignacio Mulas | 12-October-2015
  7. 7. Ericsson Internal | 2011-10-19 | Page 7 ›  Real-time analysis of metrics and logs of a cloud infrastructure –  Analysis of system behavior ›  Extraction of statistical characteristics of system’s logs and metrics –  Anomaly detection ›  Clustering of node states and deviations (anomalies) What? … Flink-forward | Ignacio Mulas | 12-October-2015 Analytics
  8. 8. Ericsson Internal | 2011-10-19 | Page 8 How? Kafka Elastic Search Kibana Flink Pre-processing Streaming k-means Classification Time drift analysis raw data pre-processed data analysis results … … Data sources Flink-forward | Ignacio Mulas | 12-October-2015
  9. 9. Ericsson Internal | 2011-10-19 | Page 9 Data pipeline Raw data Pre- processing Feature Vectors Streaming k-means Centroids Predictor Dashboard Timedrift analysis OK/NOK Timedrifts Flink-forward | Ignacio Mulas | 12-October-2015
  10. 10. Ericsson Internal | 2011-10-19 | Page 10 Data pipeline Flink-forward | Ignacio Mulas | 12-October-2015 Raw data Pre- processing Feature Vectors Streaming k-means Centroids Predictor Dashboard Timedrift analysis OK/NOK Timedrifts
  11. 11. Ericsson Internal | 2011-10-19 | Page 11 Time challenge How would you count 20 seconds without your own clock? How would you count 20 seconds? Trivial! Looking at my clock… 13 15 14 Not trivial… Two things to consider: - Cannot ask all at the same time - Everyone has a different time
  12. 12. Ericsson Internal | 2011-10-19 | Page 12 Processor Time (tp) -  Use machine clock -  Things to consider: -  Differences between hosts -  Default Time handling Event Time (te) ›  Use time of events ›  Things to consider: –  Time drifts –  Unordered events ›  Watermarks t0t0t0te0 tp VS Flink-forward | Ignacio Mulas | 12-October-2015
  13. 13. Ericsson Internal | 2011-10-19 | Page 13 Event time e0e1e2e3 … t0t1t2t3 <t0,e0><t1,e1><t2,e2><t3,e3> <te0,e0><te1,e1><te2,e2><te3,e3> EventTimeExstractor() enableTimestamps() <te0,e0><te1,e1><te2,e2><te3,e3> w2 w1 w0 window() Flink-forward | Ignacio Mulas | 12-October-2015 Te0 + window + watermark
  14. 14. Ericsson Internal | 2011-10-19 | Page 14 Data pipeline Flink-forward | Ignacio Mulas | 12-October-2015 Raw data Pre- processing Feature Vectors Streaming k-means Centroids Predictor Dashboard Timedrift analysis OK/NOK Timedrifts
  15. 15. Ericsson Internal | 2011-10-19 | Page 15 Tips and tricks
  16. 16. Ericsson Internal | 2011-10-19 | Page 16 Local deployment Kafka Elastic Search Kibana Flink LOGSTASh Data dump VM1 VM2 vagrant up
  17. 17. Ericsson Internal | 2011-10-19 | Page 17 Good ›  Flink is a natural candidate to address Kappa architecture (everything is a stream!) ›  Good integration with Kafka (StreamExecutionEnvironment) ›  Nice representation of jobs (using graph) ›  Easy to start running a cluster Things to improve ›  API not consistent in all the cases –  Timeout on sources and step function –  Reverse of other of parameters in different functions of the API ›  Easy to debug in a single machine but a bit hard in the distributed setup ›  Hard to see contextual information on slaves in the cluster learnings Flink-forward | Ignacio Mulas | 12-October-2015
  18. 18. Please, feel free to contact me if you have suggestions/ comments/questions @ ignacio.mulas.viela@ericsson.com Thank you! quESTIONS?

×