This document discusses real-time analytics of URL page views (PV) and unique visitors (UV) as an example of real-time big data analysis. It describes collecting logs, analyzing logs to count PV/UV for URLs, storing the counts in Redis and Cassandra, and storing in Greenplum for search, aggregation and ranking. It also discusses fault tolerance, the lambda architecture, and Twitter's Summingbird for streaming MapReduce.

1. 실시간 URL UV/PV 집계 사례를 통해 보는
'빅데이터 실시간 데이터 분석'
다음커뮤니케이션 유대은
moongtook@daumcorp.com
moongtook@hanmail.net
moongtook@gmail.com

2. 빅데이터 분석
Batch vs Real Time

3. Big data analytics - Batch (Hadoop)
Query = Function (All Data)
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/
http://www.slideshare.net/Hadoop_Summit/realtime-analytics-with-storm

4. Big data analytics - Batch (Hadoop)
MapReduce Job = Function (All Data)
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/
http://www.slideshare.net/Hadoop_Summit/realtime-analytics-with-storm

5. Big data analytics - Batch (Hadoop)
http://www.slideshare.net/Hadoop_Summit/realtime-analytics-with-storm

6. Big data analytics - Batch (Hadoop)
http://www.slideshare.net/Hadoop_Summit/realtime-analytics-with-storm

7. Big data analytics - Real Time (Storm)
Query = Function (Data Stream)
Data Stream을 바라보고 실시간으로 바로 분석
Fast, Incremental algorithm

8. Big data analytics - Real Time (Storm)
Topology = Function (Data Stream)
Strom은 Data Stream을 바라보며
실시간으로 데이터를 처리하기 위한 좋은 인프라

9. https://github.com/nathanmarz/storm

10. bolt
spout
http://www.infoq.com/presentations/Storm

11. A spout is a source of streams
A bolt consumes any number of
input streams, does some
processing
http://www.infoq.com/presentations/Storm

12. Storm - cluster

13. Storm - cluster
distributed realtime computation infra

14. URL UV/PV 실시간 집계 사례

15. 로그수집
https://github.com/moongtook/kestrel_tail

16. 로그수집

17. 로그수집
https://github.com/moongtook/kestrel_tail

18. 로그분석

19. 로그분석

20. 로그 하나 꺼내오기

21. 로그 하나 꺼내오기

22. URL의 UV/PV 카운트 올리기

23. URL의 UV/PV 카운트 올리기
Inside of Redis

24. URL의 UV/PV 카운트 저장하기

25. URL의 UV/PV 카운트 저장하기
Cassandra column family
row key 1
super column 1
super column 2
...
column name 1
column name 1
column name2
...
column value
row key 2
column name2
column value
column value
column value
...
super column 1
super column 2
...
column name 1
column name 1
column name2
...
column value
...
column name2
column value
column value
column value
...
...
...
...

26. URL의 UV/PV 카운트 저장하기
md5( reversed url) + date
6ed6a80a1623
65e78e2716d4
9508d974_201
2-10-24
Henessy column family schema
...
minutely_pv
minutely_uv
hourly_pv
hourly_uv
daily_pv
daily_uv
minutely_pv
...
...
212
202
5220
4576
233997
155723
151
...
...
...
20:01
...
minutely_pv
minutely_uv
hourly_pv
hourly_uv
daily_pv
daily_uv
minutely_pv
...
...
...
...
20:01
...
bc2ed9981fae
01adda327bcd
7e2a3576_201
2-10-24
...
20:02
...
388
383
9839
8163
597338
299751
364
...
...
...
...
20:02
...
...
...

27. Greenplum에도 저장하기
Search, Aggregation, Ranking을 위해
지난 1분동안 UV/PV 변화가 있었던 컨텐츠만...

28. Greenplum에도 저장하기
Secondary Index Pattern
2012-10-24_20_01
bc2ed9981fae01adda327bcd7e2a3576_2012-10-24
null
null
...
6ed6a80a162365e78e2716d49508d974_2012-10-24
bc2ed9981fae01adda327bcd7e2a3576_2012-10-24
...
2012-10-24_20_03
6ed6a80a162365e78e2716d49508d974_2012-10-24
...
2012-10-24_20_02
...
null
null
...
6ed6a80a162365e78e2716d49508d974_2012-10-24
bc2ed9981fae01adda327bcd7e2a3576_2012-10-24
...
null
null
...
...
...

29. URL UV/PV 실시간 집계 사례

40. Fault-tolerant
장애 허용 시스템(Fault tolerant system)은
구성 부품의 일부가 고장나도 정상적으로 처리를
수행하는 시스템 이다. - 위키백과
http://ko.wikipedia.org/wiki/장애_허용_시스템
http://en.wikipedia.org/wiki/Fault-tolerant_design

41. Human Fault-tolerant

42. Human Fault-tolerant
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/
http://strataconf.com/strata2013/public/schedule/detail/27610

44. URL UV/PV 실시간 집계 사례

45. URL UV/PV 실시간 집계 사례

46. Lamda architecture
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/

47. Lamda architecture
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/

48. Lamda architecture
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/

49. Lamda architecture
Big data - Nathan Marz and James Warren, http://www.manning.com/marz/

50. Lamda architecture
Twitter summingbird - https://speakerdeck.com/sritchie/summingbird-streaming-mapreduce-at-twitter

51. Lamda architecture
Twitter summingbird - https://speakerdeck.com/sritchie/summingbird-streaming-mapreduce-at-twitter

52. 끝!