This document provides an overview of building a business intelligence (BI) system using a data lake ecosystem. It discusses using Hadoop, Hive, Teradata, Tableau and Jenkins together. The goals are to deal with big data problems like high volume, variety and velocity of data in a cost effective way. Sample architectures are proposed to handle ETL processes, data storage and querying, and visualization. Key considerations for choosing components include storage and management costs, processing time, and balancing needs with costs. The document concludes by suggesting that a good framework can help support business growth over time within a given cost curve.

1. BUILD YOUR BI SYSTEM
PRACTICE IN DATA LAKE ECOSYSTEM
Bryan@Vpon Data

2. • Experience
Vpon Data Engineer
TWM, Keywear, Nielsen
• Bryan’s notes for data analysis
http://bryannotes.blogspot.tw
• Spark.TW
• Linikedin
https://tw.linkedin.com/pub/bryan-yang/7b/763/a79
ABOUT ME

9. AGENDA
• User Story
• Data Lake
• Frame Work of BI

10. DEAL WITH BIG DATA

11. SMALL RETAILER

12. MORE COMPLEX AND
BIG…
http://www.slideteam.net/technology-powerpoint-templates/mobile-phones.html

13. 3 KINDS OF PROBLEMS
https://kavyamuthanna.wordpress.com/category/big-data/

14. BIG DATA BIG PROBLEM
http://www.mn.uio.no/ifi/studier/masteroppgaver/nd/masteroppgave_cloud_bigdata_hpc.html

15. BIG DATA BIG COST
• The cost of data storage
What does the data keep?
How long?
• The cost of data management
Is the machine and infra easy to maintain?
Data Flow(ETL)?
• The time cost of data processing
How long will the users can wait?
Accessibility of the data
Human costs you can not see

16. A REAL CASE

17. SO MANY ADHOC QUERIES
SALES
MARKETING
FINANCE
BUSINESS

18. EVEN A SIMPLE QUERY
Q: HI, PLEASE TELL ME HOW MANY
USERS FROM THE BEGINNING?
A:SELECT COUNT(1)
FROM LOG

21. ttps://myreelpov.wordpress.com/2012/12/23/which-story-do-you-prefer-life-of-pi/life-of-pi-2
Your Life
Boss
Family and Lover
Customers
Data Ocean

23. Overviews
Business intelligence (BI) is the set of techniques and tools for
the transformation of raw data into meaningful and useful
information for business analysis purposes. —Wikipedia

24. DIFFERENT FEATHERS
Price Perfomance Accessibility
Hadoop Low Median Low
SQL Server Low-Median Depends on Median
Data
Warehouse
High High Median
BI System High Depends on High

25. http://www.datalytyx.com/big-data-data-lakes/

27. WHY DATA LAKE
tp://thesologuide.com/332/the-seesaw-of-success-when-taking-a-rest-is-bes

29. HIVE
• Create at Facebook
• Data warehouse in Hadoop ecosystem
• HiveQL(SQL like interface)
• Metastore(Save the schema of data,
schema on read)
• UDF

30. http://www.stratapps.net/intro-hive.php

31. http://hortonworks.com/blog/hive-cheat-sheet-for-sql-users/

32. ONE MORE THING

33. TERADATA
• Massively Parallel Processing
• Each processor handles different threads
of the program, and Each processor itself
has its own operating disk
• Teradata SQL is fully certified at the SQL
92

36. http://www.slideshare.net/alam7/module-02-teradata-basics

37. https://www.safaribooksonline.com/library/view/teradata-architecture-for

38. TABLEAU
• Visualization Tool
• Connect with kinds of database
• VizQL
• Tableau Server

39. http://www.clearpeaks.com/blog/tableau/tableau-8-2-new-features

41. https://www.youtube.com/watch?v=fYpy04vmG_o

43. m/services/business-intelligence-services/tableau-consulting/table

44. JENKINS
• Manage ETL processes
• Free & Many Plugins
• Monitor Jobs Status and dependency
• Communication with Git and SVM
• Email alert

45. User Interface回到首頁
管理選單
建置中項目
Joblist 建置資訊
建置狀態
下次建置項目及時間
ip:port
Job Name List
Job Name
List

46. Build Steps
call python script
call the remote shell
call local shell script

47. Build Graph
Job Name Job Name Job Name
Job Name Job Name Job Name
Job Name Job Name Job Name

48. LET’S PUT IT ALL
TOGETHER

49. Hadoop
Cluster 1
Hadoop
Cluster 2
Teradata
Tableau
Server
User
Data Transfer
Request
ETL
Live Query Too Slow
Data Slicing

50. Hadoop
Cluster 1
Hadoop
Cluster 2
Teradata
Tableau
Server
User
Data Transfer
Request
ETL
Extract Data Insufficient Space
Data Slicing

51. Hadoop
Cluster 1
Hadoop
Cluster 2
Teradata
Tableau
Server
User
Data Transfer
Request
ETL
Extract Data
Every Day
Table
View
Statistical Tables
Data Slicing

52. USER EXPERIENCE
TUNING
0
30
60
90
120
150
HIVE TERADATA BI
120X Faster

55. HOW TO CHOOSE THE
COMPONENT IN YOUR BI
FRAMEWORK ?
• The cost of data storage
• The cost of data management
• The time cost of data processing

56. CONSIDERINGS AND
SUGGESTIONS
• Time is money
• HDD space/ money for the time
• Understanding the components and
relationships
• Get balance of the needs and costs
• Good framework will help business growth

57. COST CURVE
Business Growth
CostofBusinessGrowth

58. Hardware
*More Nodes
*More Memories
*Graph Card
…
Software
*Spark
*Tez
*Tachyon
*Algorithm
…
IN THE FUTURE
Cloud
*EC2
*Big Query
*Bluemix
*SAP
…

59. THANK YOU FOR YOUR
LISTENING
Special Thank
Vpon
Hood, Meiyen, Gil and OPS Team

60. Q & A