Slide 3 Fast Data processing with kafka, rfx and redis

217 views

Published on

my slide in big data class

Published in: Data & Analytics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
217
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Slide 3 Fast Data processing with kafka, rfx and redis

  1. 1. Fast Data Processing with RFX Simplify Fast Data Processing trieunt@fpt.com.vn tantrieuf31@gmail.com
  2. 2. Today topic : We would talk about all things in this red circle
  3. 3. Demo first https://github.com/rfxlab/pageview-analytics-with-rfx
  4. 4. Content at glance 1. BEAM✲ methodology for agile data warehouse 2. Introduction to Fast Data 3. Problem “Fast Data in web analytics” 4. Examples for fast data design pattern (RFX or Reactive Function X) 4.1. Event data actor 4.2. Event data agent 4.3. Event data collector 4.4. Event data router 4.5. Event data processor 4.6. Event data storage 4.7. Event data query 4.8. Event data reactor 5. Demo “Fast Data in web analytics” with source code explanation
  5. 5. 1 - BEAM✲ methodology
  6. 6. 1 - BEAM✲ methodology for Agile Data Warehouse BEAM✲ stands for Business Event Analysis & Modelling, and it’s a methodology for gathering business requirements for Agile Data Warehouses and building those warehouses. It was developed by Lawrence Corr (@LawrenceCorr) and Jim Stagnitto (@JimStag), and published in their book Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema.
  7. 7. Example with BEAM✲
  8. 8. Goal: Modeling all business events and put into a database in agile way
  9. 9. 2 - Fast Data
  10. 10. Introduction to Fast Data
  11. 11. 3 - Problems in Practice
  12. 12. Problems “Fast Data in web analytics” 1. Counting pageview of website 2. Counting unique user of website 3. Sending email when pageview is unnormal (simple DDOS attack detection)
  13. 13. 4 - Thinking with RFX
  14. 14. ● A design pattern to solve big fast data problems ● A collection of Open Source Tools ● The mission of RFX 1. Build data product quickly with design patterns 2. Apply BEAM✲ for agile data pipeline 3. React to critical events in near-real-time What is RFX or Reactive Function X ?
  15. 15. RFX framework What ? ● The Java framework, is built from open source projects: ○ Based on core Akka Actor ( http://akka.io ) ○ Lightweight DAO with Spring JDBC ( https://spring.io ) ○ Netty ( http://netty.io ) and VertX ( http://vertx.io/ ) ○ Common utils class for Apache { Kafka, Hadoop , Spark } ○ Common utils class for NoSQL ( Redis ( http://redis.io ), MongoDB ) ● a R&D project, started since 11/2013 for fast data processing Why ? ● Divide Java code into modules: ○ common infrastructure code ( rfx-stream ) ○ business logic code ( check valid data stream ) ○ machine learning code ( automation & optimization ) ● Focus on best practices and reusability ● Foundation for scalability (system and business) ● Test-driven development for Real-Time Analytics ● Continuous integration & improvement
  16. 16. Your business logic here
  17. 17. Reactive Function (X) Philosophy
  18. 18. Core elements of rfx-stream
  19. 19. Core backend modules rfx-track: ● collecting all events from log agent rfx-stream: ● processing stream data (PipelineProcessing pattern) ● processing real-time analytics ● processing business logic (by reactive function) rfx-cronjob: ● synchronizing real-time data to report database (by parsing data in Redis and update to Report database)
  20. 20. Core frontend modules rfx-report: ● visualizing data in real-time ● monitoring real-time event rfx-agent: ● tracking user activity: heatmap data, pageview, ... ● logging user activity to rfx-track (via network protocol: HTTP, TCP or UDP)
  21. 21. How to solve problems with RFX ?
  22. 22. Use Cases in “Fast Data in web analytics” 1. Counting pageview of website 2. Counting unique user of website 3. Sending email when pageview is unnormal (simple DDOS attack detection)
  23. 23. Apply RFX into Pageview Analytics 1.1. Event data actor: a web user 1.2. Event data agent: RFX-track-js 1.3. Event data collector: RFX-track-server 1.4. Event data queue: Apache Kafka 1.5. Event data processor: RFX-stream 1.6. Event data storage: Redis, MySQL 1.7. Event data query: RFX-data-api 1.8. Event data reactor: RFX-reactor
  24. 24. Demo and Explanation for code and concepts
  25. 25. Readings ● http://www.decisionone.co.uk/press/agile-data-warehouse-design-sampler.pdf ● http://www.slideshare.net/votrongdao/agile-data-warehouse-34427798 ● Apache Kafka Installation Video | How To Setup Apache Kafka https://youtu.be/Fg8cTsEk7Gc ● https://www.tutorialspoint.com/apache_kafka/ ● https://kafka.apache.org/quickstart ● http://xyu.io/2015/07/13/building-a-faster-etl-pipeline-with-flume-kafka-and-hive/ ● http://blog.cloudera.com/blog/2015/06/architectural-patterns-for-near-real-time-data-pr ocessing-with-apache-hadoop/ ● https://www.oreilly.com/ideas/drivetrain-approach-data-products

×