Successfully reported this slideshow.
Your SlideShare is downloading. ×

Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 10 Ad

Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021

Download to read offline

Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021

Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream

The first project profiled in this talk (transforming email reports into self-serving dashboards) involved generating query-based reports for the sales team replacing email reports. Messages were consumed from Apache Kafka, aggregated and inserted into InfluxDB based on Kafka metadata timestamp. The Socialgist team ran into a problem of counts mismatch for high-traffic Kafka topics and solved it by randomizing timestamp and building cache.

The second project profiled (detecting anomalies in-stream) involved detecting strange behaviors in graphs of data streams (that are internally considered as anomalies). Data is pulled from Elasticsearch, run through the anomaly detection model, and stored in InfluxDB. The results stored in InfluxDB are represented in Grafana, and alerts are fired into the Slack channel. This project helped Socialgist predict the behaviors of streams and identify alerts before any other system could.

Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021

Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream

The first project profiled in this talk (transforming email reports into self-serving dashboards) involved generating query-based reports for the sales team replacing email reports. Messages were consumed from Apache Kafka, aggregated and inserted into InfluxDB based on Kafka metadata timestamp. The Socialgist team ran into a problem of counts mismatch for high-traffic Kafka topics and solved it by randomizing timestamp and building cache.

The second project profiled (detecting anomalies in-stream) involved detecting strange behaviors in graphs of data streams (that are internally considered as anomalies). Data is pulled from Elasticsearch, run through the anomaly detection model, and stored in InfluxDB. The results stored in InfluxDB are represented in Grafana, and alerts are fired into the Slack channel. This project helped Socialgist predict the behaviors of streams and identify alerts before any other system could.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021 (20)

Advertisement

More from InfluxData (20)

Recently uploaded (20)

Advertisement

Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] | Transforming Email Reports into Self-Serving Dashboards & Detecting Anomalies In-Stream | InfluxDays NA 2021

  1. 1. A platform that provides the tools & services to connect research & analytics companies to the data which they depend upon.
  2. 2. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Project Overview : Transforming Email Reports • Consume messages from Apache Kafka • Aggregate and store in Influxdb • Goal is to generate self serving dashboards • High volume mismatch issue solved by • adding unique tag • building point cache • randomizing timestamp
  3. 3. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Architecture & Usage of InfluxDB
  4. 4. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Self Serving Dashboard
  5. 5. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Project Overview : Anomaly Detection • into a problem of counts mismatch for high traffic Kafka topics and we solved it by randomizing timestamp and building cache. • Collecting tons of data from various sources • Storing count of collected data in InfluxDB • Goal is to find anomalies in data • There are three types of anomalies: • sudden drops • volume decay • deadman
  6. 6. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Usage of InfluxDB • into a problem of counts mismatch for high traffic Kafka topics and we solved it by randomizing timestamp and building cache. • Storing results of Anomaly Detection into InfluxDB • Alerts - Tick scripts • Using dynamic thresholds • Alert handling: • slack channel • email (sms)
  7. 7. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Visualizing • Slack alert • which stream • anomaly strength • threshold • Visualization in Grafana • Obtaining data from InfluxDB
  8. 8. © 2021 InfluxData Inc. All Rights Reserved. Q&A
  9. 9. © 2021 InfluxData Inc. All Rights Reserved. Thank You

×