This document provides an overview of Big Data concepts including architecture, use cases, and examples. It discusses how Big Data can help tell what will happen, find unexpected relationships, monitor situations, and fix problems. Real examples mentioned include Google leaving China, input methods, Google ads, Netflix recommendations, and Walmart personalized offers. The Big Data architecture described includes collecting data, using queues to decouple systems, processing data in Hadoop clusters, storing data in data stores like HDFS, HBase and Elasticsearch, and visualizing results. Cloud services can also be used to build projects quickly.

1. BIGData, BIGDream
BIN ZHU

2. Who am I？
• 15years experience
• Telecom
• Enterprise ERP
• National Library
• Internet
• Java
• Database DBA, Designer, developer
• BI, Report, Data Mining
• Architect
• Team leader, Dev Manager, Product Owner

3. What is Big Data？
大 快 杂 疑

4. What the Big Data can do for us
• TellYou What Is Likely to Happen
• Find Unexpected Relationships
• Monitor a Situation as it Develops
• Fix a Problem Before it Becomes a Crisis

5. Real Examples
• Google left China
• Input methods

6. Real Examples
• Google Ads
• Google Map Service

7. Real Examples
• Netflix movie recommendation

8. Real Examples
• Walmart shopping Suggestion and customized flyer
• Credit Card Fraud

9. Big Data Architecture
Collecting Queue Processing Data Store Visualization

10. Collecting Queue Processing Data Store Visualization

11. Big Data Architecture
Collecting Queue Processing Data Store Visualization

12. Collecting data

13. Big Data Architecture
Collecting Queue Processing Data Store Visualization

14. Queue
• Queue is an ordered list of elements of similar data types
• Queue is a FIFO( First in First Out ) structure

15. Why Queue?
• Decoupling, simplify the system
• Building asynchronous applications
• Make the application much stronger

16. Popular Queues

17. Positives and Negatives of each Queue

18. Kafka architecture

19. Big Data Architecture
Collecting Queue Processing Data Store Visualization

20. Processing Engine

21. Cluster in HA mode (99.99%)
52mins down time in a year
• Availability
• Reliability
• No Single Point of Failure
• Scalable

22. Hadoop Cluster

23. Hadoop HDFS (Hadoop Distributed File System)

24. Hadoop Yarn (Cluster Resource Management)

25. HowYarn works?

26. Apache Storm
Apache Storm is a free and open source
distributed real-time computation system.
Use Case
• Real-time analytics
• Online machine learning
• Continuous computation
• Distributed RPC
• ETL
Main feature
• Scalable
• Fault-tolerant

27. Big Data Architecture
Collecting Queue Processing Data Store Visualization

28. Data Store

29. Data Store – RDBMS

30. Data Store – NoSQL

31. RDBMS VS NoSQL

32. RDBMS Cluster
Master – Slave
Master – Master
Sharding
Separate DB

33. NoSQL Cluster (Cassandra)

34. Collecting Queue Processing Data Store Visualization

35. Real Case

36. ELK
ElasticSearch
Logstash
Kibana

37. Cache Redis

38. Cloud

39. What is behind the Cloud?

40. Use Cloud Services to build your Project Quickly

41. Cloud
Public Cloud？Private Cloud？

43. Real Job Description in Montreal
Company: Hortonworks
Position: SOLUTIONS ENGINEER
SKILLS
• You already know and love core Apache Hadoop (HDFS &YARN) and can talk to the benefits of a centralized
architecture for both data management and data access
• Solution Architecture/Engineering experience as a field of practice (able to listen to
customer requirements, whiteboard and propose solution architecture, and get hands-on with the tech
to design, build, and demonstrate real business value)
• Streaming experience (Nifi, Kafka, Storm, Spark, Solr, etc.)
• NoSQL experience (HBase, Cassandra, MongoDB, etc.) Maybe you can even debate one option over another?
• EDW experience – theTeradatas, Netezzas, GreenPlum/HAWQ, Exadatas of the world
• Integration Products experience (Tibco, MuleSoft, IBM, Oracle, Spring Integration, etc.)
• Start-up company experience
• HortonworksCertified - Admin and/or Developer
http://hortonworks.com/careers/details/102293/

44. Questions