Analytics Platforms

1. BACK END
ANALYTIC PLATFORMs
Nguyen Thi Kim Tuyen
Version 1.0
01/2013

2. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

3. Targets
 Analytic systems
 KPI
 Monitoring (access log , error log)
 Real-time analytics / batch processing analytic
al tasks for Client
 Depends GNT infrastructure , Scalability

4. Approaches
 Refer log platforms' achitectures of Facebook ,
Twitter , ...
 Community reviews of each component
 Adapt needs

5. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

6. Analytic platform(1/10)
 Tracker
 Action log, error log (nginx)
 Web log (Play framework)
 Game user activities log (event-driven logs)
 Database log (Cassandra, Redis, commit log)
 Page taging / logfile analytics
 Collector
 ETL
 Analyzer
 Reporter

7. Analytic platform(2/10)
Facebook
 Facebook
 Web -> Scribe -> Ptail -> Puma -> HBase
 http://www.slideshare.net/slrash/2011-
0630hadoopsummit-v5-8469751
 => Collection layer (Flume/Scribe) → Filter
layer (Flume) → Batching layer (Coprocessor)

8. Analytic platform(3/10)
Facebook

9. Analytic platform(4/10 )Facebook
Caligraphus-HDFS-ZK = { HDFS, Zookeeper,
Hbase, Hive }

10. Analytic platform(5/10) Facebook
Ptail = Parallel Tail
Concurent read : HDFS 2.0 : add sync : lower write-to-read latency
Ptail : read blocking data being written , < 10s latency

11. Analytic platform(6/10) Facebook
PUMA write-path

12. Analytic platform(7/10) Facebook
PUMA read-path

13. Analytic platform (8/10)
Twitter

14. Analytic platform(9/10)
Twitter

15. Analytic platform(10/10)

16. Agenda
 Targets
 Approaches
 Analytic Platform
 Map/Reduce
 GNT Game analytic system(current/testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

17. Map Reduce(1/2)

18. Map Reduce(2/2)

19. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current &
testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

20. GNT Game Analytic System(1/2) (Current)

21. GNT Game Analytic System(2/2)
current
 Limitation :
– Javscript implementation : limit 1 JS execution
/1 server at time
– Scalability : not scale except in case of
sharding
 Improving : integration Mongo + Hadoop
 http://www.slideshare.net/iammutex/the-
elephant-in-the-room-mongo-db-hadoop

22. GNT Game Analytic System(1/4)

23. GNT Game Analytic System(2/4)
testing FlumeNG
 flume.conf : 192.168.30.183
 t-game-web183.sources = tail-nginx tail-play
 t-game-web183.sinks = avro-sink-nginx183 avro-sink-play183
 t-game-web183.channels = mem-channel-nginx183 mem-channel-
play183
 t-game-web183.sources.tail-nginx.type = exec
 t-game-web183.sources.tail-nginx.command = tail -F
/var/log/nginx/access.log
 t-game-web183.sources.tail-nginx.channels = mem-channel-nginx183
 t-game-web183.channels.mem-channel-nginx183.type = memory
 t-game-web183.sinks.avro-sink-nginx183.type = avro
 t-game-web183.sinks.avro-sink-nginx183.hostname = 192.168.30.185
 t-game-web183.sinks.avro-sink-nginx183.port = 10183
 t-game-web183.sinks.avro-sink-nginx183.channel = mem-channel-
nginx183

24. GNT Game Analytic System(3/4)
testing FlumeNG
 flume.conf : 192.168.30.185
 t-game-cass185.sinks.hdfs-sink-
nginx183.type = hdfs
 t-game-cass185.sinks.hdfs-sink-
nginx183.hdfs.path =
hdfs://namenode/flume/webdata/nginx183

25. GNT Game Analytic System(4/4)
testing Hadoop

26. Mixed Solutions
 Case 1 : old system : Mongo + Hadoop
 Case 2 : FlumeNG + Hadoop + HBase
 Case 3 : Batch processing : Hadoop HDFS
(not use FlumeNG)

27. GNT Game Analytic system proposal

28. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

29. Hadoop(1/19)

30. Hadoop(2/19)

31. Hadoop(3/19)

32. Hadoop(4/19)

33. Hadoop(5/19)

34. Hadoop(6/19)

35. Hadoop(7/19)

36. Hadoop(8/19)

37. Hadoop(9/19)

38. Hadoop(10/19)

39. Hadoop(11/19)

40. Hadoop(12/19)

41. Hadoop(13/19)

42. Hadoop(14/19)

43. Hadoop(15/19)

44. Hadoop(16/19)

45. Hadoop(17/19)

46. Hadoop(18/19)

47. Hadoop(19/19)

48. Agenda
 Targets
 Methodologies
 Log platforms
 Map/Reduce
 GNT analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

49. Hadoop components(1/2)

50. Hadoop components(2/2)
 Flume HBase
 Sqoop Zookeeper
 Hue Oozies
 Pig Whirr
 Hive Snappy
 Hbase Mahout

51. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

52. HBase
 Facebook , Twitter using it . Why ?
 New feature : Coprocessor

53. HBase

54. HBase
Data model
 Table
 Row
 Column
 Cell
 Versions
 Row-key design

55. HBase vs Cassandra
 TODO

56. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

57. Collectors
 Flume NG (Collector + ETL)
 Scribe
 Chukwa
 Which one is suitable for us ?

58. Flume/FlumeNG(1/10)
architecture

59. Flume/FlumeNG(2/10)
Concepts
 Network stream :
 Avro/Syslog/Netcat
 Source / Channel /Sink
 Decorator
 Flow
 Event
 Flume agent
 Flume avro client / log4j Appender

60. Flume/FlumeNG(3/10)
 Flume source :
 Avro
 Netcat
 Syslog (TCP/UDP)
 Exec
 Thrift/Avro legacy
 Custom

61. Flume/FlumeNG(4/10)
 Flume Sink :
 HDFS
 Avro
 Logger
 … , FileRoll, Custom
 Flume Channel :
 Memory
 JDBC channel
 Recoverable memory channel

62. FlumeFlumeNG(5/10)
Consolidation

63. Flume/FlumeNG(6/10)
Multiplexing

64. Flume/FlumeNg(7/10)
Reliability & Failure handling

65. Flume/FlumeNG(8/10)
Failure handling

66. Flume/FlumeNG(9/10)

67. Flume/FlumeNG(10/10)
plugin (decorator)
 TODO
 Flume with HBase sink
 https://groups.google.com/a/cloudera.org/grou
p/cdh-
user/browse_thread/thread/5ee135ad0e720ea
9/c5bffc83f97fdd3c?hl=vi&lnk=gst&q=flume-
ng#c5bffc83f97fdd3c

68. Agenda
 Targets
 Approaches
 Analytic platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

69. Chukwa(1/2)

70. Chukwa(2/2)

72. Agenda
 Targets
 Approaches
 Log platforms
 Map/Reduce
 GNT Game analytic system (current & testing)
 Appendix
 Hadoop/Hadoop components
 Hbase
 Collectors (FlumeNG/Chukwa/Scribe)

73. Flume vs Scribe
 TODO

74. Future issues
 Manage analytic jobs
 Message queue : Kafka , ZeroMQ
 Monitoring memory , flume agent , hadoop
cluster , ...
 Scalability

75. Referrences
 Google docs :
 Cloudera
 Hadoop begins
 Hbase begins
 FlumeNG
 Log center
 Wikipedia :
 ...