DataTalks #4: Построение хранилища данных на основе платформы hadoop / Игорь Нахват

•Download as PPTX, PDF•

2 likes•678 views

В докладе Игорь продемонстрирует архитектуру хранилища, базирующуюся на технологиях от Cloudera и Oracle. Он расскажет об опыте интеграции множества источников данных с помощью самописных решений и использованием специализированных инструментов, как Apache NiFi. Выступление заинтересует технических специалистов, которые уже знакомы со стеком Hadoop. Интересуетесь анализом данных? Присоединяйтесь к нашей группе на Facebook: https://www.facebook.com/groups/DataTalks/

BUILDING A DATA WAREHOUSE
WITH HADOOP
10.10.2015
IGOR NAKHVAT, DATA INTEGRATION ENGINEER

CONTENT TABLE
I. Building a Data Warehouse with Hadoop
A.Data sources
B.Data storage
C.Data flow
D.ETL tool
E.Conclusions

3
DATA SOURCES
GamesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

4
SPA Payment ForumeSport
Clan warsUpdate
DATA SOURCES
ServicesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

5
DATA SOURCES
GeographyDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

6
DATA SOURCES
Total: 294222
72
Tables
Total: 1264
Relational databasesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

7
DATA SOURCES
Non - Relational data sourcesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

8
• It is an open architecture.
• Cost effective.
• Many interfaces to data (SQL, Spark, Java, Scala, Python).
• Many ways/formats for storing the data.
• Many tools available for the data analytics.
DATA STORAGE
Why Hadoop?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

9
Keep in mind
• Lack of employees
• Security
DATA STORAGE
Why Hadoop?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

10
DATA STORAGE
Hadoop ecosystem
SQOOP
PIG
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

11
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

12
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

13
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

14
DATA FLOW
ShellSQL CSV + GZIP
Check count
rows
Compute
stats
Parquet
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

15
DATA FLOW
Parquet
Shell
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

16
DATA FLOW
Parquet
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

17
DATA FLOW
Shell
Aggregation
Presentation
Audience
Balance
Finance
Data
scientist
Manager
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

18
ETL TOOL
Continuous integration tool
+ =
ETL toolPlugins
5 - 20 Hours 1000+ Jobs
Jenkins
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

19
ETL TOOL
Apache NiFi
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

20
Apache NiFi
• Drag – n – drop works!
• Great visualization.
• Data provenance.
• Flow can be modified at runtime.
ETL TOOL
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

21
Apache NiFi
Keep in mind
• Multiuser development.
• No templates.
• NiFi is not an orchestration tool.
ETL TOOL
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

22
CONCLUSIONS
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS
• Hadoop is good for data warehousing
• Poor Hadoop security
• Impala (SQL on Hadoop) performs and scales
• Data format choice is a key (Avro, Parquet)

Microsoft has been a leader in the enterprise analytics space for years. In 2014, Microsoft had already created R language functionality within Azure Machine Learning. On April 6, 2015, Microsoft and closed on a deal to acquire Revolution Analytics, a company focusing on scalable processing solutions initiated by the well-known R language. Many data science projects and initial demos do not need high-volume solutions: however, having a high-volume answer for the R language allows for planning or working toward the largest data science solutions. This presentation describes the add-value for the Revolution Analytics acquisition. The talk covers 1) an overview of current data science technologies from Microsoft; 2) a description of the R language; 3) a brief review of the add-value for R with Azure Machine Learning, and 4) a description of the performance architecture and demo of the language constructs developed by Revolution Analytics. Most of the presentation will be focused on sections two and four. It is anticipated that these technologies will be partially if not fully integrated into SQL Server 2016.

Hadoop introKeith Davis

MATLAB and Scientific Data: New Features and Capabilities

The HDF-EOS Tools and Information Center

Join the journey of a data scientist on the way to industrialization... From notebook to proof of concept, from proof of concept to production, we will cover what happened at Air France. It won’t be golden rules, but a true story. What is exactly industrializing data science? How to package data science models? How to articulate data scientists and data engineers roles? Is continuous integration a wild dream for data scientists? This journey will feed you with key concepts which worked at Air France, and might give you a new light to guide you through your own data science journey. Pauline Ballereau - Air France & Nicolas Laille - Xebia https://dataxday.fr/ video available: https://www.youtube.com/watch?v=ESx6wR6g4ukx

Guacamole Fiesta: What do avocados and databases have in common?

ArangoDB Database

First, our CTO, Frank Celler, does a quick overview of the latest feature developments and what is new with ArangoDB. Then, Senior Graph Specialist, Michael Hackstein talks about multi-model database movement, diving deeper into main advantages and technological benefits. He introduces three data-models of ArangoDB (Documents, Graphs and Key-Values) and the reasons behind the technology. We have a look at the ArangoDB Query language (AQL) with hands-on examples. Compare AQL to SQL, see where the differences are and what makes AQL better comprehensible for developers. Finally, we touch the Foxx Microservice framework which allows to easily extend ArangoDB and include it in your microservices landscape.

Tuning and Optimizing U-SQL Queries (SQLPASS 2016)

Michael Rys

HDF5 High Level and Lite Libraries

The HDF-EOS Tools and Information Center

GDAL Enhancement for ESDIS Project

The HDF-EOS Tools and Information Center

SQL Now! How Optiq brings the best of SQL to NoSQL data.

Julian Hyde

A talk given at the August, 2013 NoSQL Now! conference in San Jose, California. 2013 is the year of that SQL meets Hadoop & NoSQL. There is a surprisingly good fit between thirty-something SQL and the upstart millennial data technologies. SQL helps you deliver faster value, improve integration with visualization and analytics tools, and bring your data to a larger audience. Julian Hyde, author of Mondrian and Optiq, and a contributor to Apache Drill and Cascading Lingual, shows how to quickly build a SQL interface to a NoSQL system using the Optiq framework. As a case study, he shows how Optiq has allows the Mondrian in-memory analysis engine (part of Pentaho's business intelligence suite) to leverage the memory and compute power of a cluster and pull data from hybrid SQL and NoSQL sources.

DataTalks #4: Что такое предиктивная аналитика и кому она нужна / Надежда Руч...

DataTalks #4: Построение хранилища данных на основе платформы hadoop / Игорь Нахват

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to DataTalks #4: Построение хранилища данных на основе платформы hadoop / Игорь Нахват

Similar to DataTalks #4: Построение хранилища данных на основе платформы hadoop / Игорь Нахват (20)

More from WG_ Events

More from WG_ Events (20)

Recently uploaded

Recently uploaded (20)

DataTalks #4: Построение хранилища данных на основе платформы hadoop / Игорь Нахват