High-volume event streams (traditional network data, media, IoT sensor data, activity events on social media, etc.) are becoming widespread in the telecom industry. In particular, live analysis of telco log files and performance metrics allows network operators to observe the status of the system and identify possible problems using online aggregations and machine-learning algorithms. (Offline batch analysis of streams using tools like MapReduce is often too slow to respond to things happening right now; hence, it is not the best choice.)
Ignacio Manuel Mulas Viela and Nicolas Seyvet demonstrate an analytics pipeline setup for a telco use case that processes an unbounded dataset of logs and performance metrics. Raw data, logs, and cloud telemetry information are extracted from a production cloud infrastructure using Collectd, Openstack Ceilometer, and Logstash. This is piped into a distributed messaging system, Kafka, then analyzed by Apache Flink—a distributed stream analysis framework that is capable of analyzing thousands of messages per second, extracting insights that can be monitored by humans—and visualized using the ELK (Elasticsearch, Logstash, Kibana) stack.
Ignacio and Nicolas discuss the challenges and benefits of building an analytics pipeline following the Kappa architecture paradigm using the aforementioned tools and demonstrate Kappa’s value through an example use case. The use case analyzes and extracts statistical information from a stream of data and uses machine-learning techniques to develop an advanced anomaly detector, using two online machine-learning algorithms implemented on top of Flink: the online k-means detector and the Bayesian detector.