Integrated Data Warehouse with Hadoop and Oracle Database

Building the Integrated DataWarehouseWith Oracle Database and HadoopGwen Shapira, Senior Consultant

Why Pythian•  Recognized Leader: •  Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration •  Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments•  Expertise: •  One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors •  Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC•  Global Reach & Scalability: •  24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response © 2012 – Pythian

About Gwen Shapira •  Oracle ACE Director •  13 Years with pager •  7 as Oracle DBA •  Senior Consultant: •  Has MacBook, will travel. •  @gwenshap •  http://www.pythian.com/news/author/ shapira/ © 2012 – Pythian

Agenda•  What is Big Data?•  Why do we care about Big Data?•  Why your DWH needs Hadoop?•  Examples of Hadoop in the DWH•  How to integrate Hadoop into your DWH•  Avoiding major pitfalls © 2012 – Pythian

What is Big Data?

Your DWH needs Hadoop

Technical Challenges•  Storage capacity•  Storage throughput Scalable storage •  Pipeline throughput•  Processing power•  Parallel processing Massive Parallel Processing •  System Integration•  Data Analysis Ready to use tools © 2012 – Pythian

Hadoop Benefits•  Reliable solution based on unreliable hardware•  Designed for large files•  Load data first, structure later•  Designed to maximize throughput of large scans•  Designed to maximize parallelism•  Designed to scale•  Flexible development platform•  Solution Ecosystem © 2012 – Pythian

Hadoop in the Data WarehouseUse Cases and Customer Stories

Connecting the (big) Dots

Sqoop Import Examples•  Sqoop import -‐-‐connect jdbc:oracle:thin:@//dbserver: 1521/masterdb -‐-‐username hr -‐-‐table emp -‐-‐where “start_date > ’01-‐01-‐2012’” •  Sqoop import jdbc:oracle:thin:@//dbserver:1521/masterdb -‐-‐username myuser -‐-‐table shops -‐-‐split-‐by shop_id Must be indexed or -‐-‐num-‐mappers 16 partitioned to avoid 16 full table scans © 2012 – Pythian

Less Flexible Export • 100row batch inserts • Commit every 100 batches • Parallel export • Update mode Example: sqoop export -‐-‐connect jdbc:oracle:thin:@//dbserver:1521/masterdb -‐-‐table bar -‐-‐export-‐dir /results/bar_data © 2012 – Pythian

Fuse-DFS•  Mount HDFS on Oracle server: •  sudo yum install hadoop-0.20-fuse •  hadoop-fuse-dfs dfs:// <name_node_hostname>:<namenode_port> <mount_point>•  Use external tables to load data into Oracle•  File Formats may vary•  All ETL best practices apply © 2012 – Pythian

Oracle Loader for Hadoop•  Load data from Hadoop into Oracle•  Map-Reduce job inside Hadoop•  Converts data types.•  Partitions and sorts•  Direct path loads•  Reduces CPU utilization on database © 2012 – Pythian

Oracle Direct Connector to HDFS•  Create external tables of files in HDFS•  PREPROCESSOR HDFS_BIN_PATH:hdfs_stream •  All the features of External Tables•  Tested (by Oracle) as 5 times faster (GB/s) than FUSE-DFS © 2012 – Pythian

How not to Fail

Use Hadoop Efficiently•  Understand your bottlenecks: •  CPU, storage or network?•  Reduce use of temporary data: •  All data is over the network •  Written to disk in triplicate.•  Eliminate unbalanced workloads•  Offload work to RDBMS•  Fine-tune optimization with Map-Reduce © 2012 – Pythian

Thank you and Q&ATo contact us… sales@pythian.com 1-877-PYTHIAN To follow us… http://www.pythian.com/news/ http://www.facebook.com/pages/The-Pythian-Group/163902527671 @pythian @pythianjobs http://www.linkedin.com/company/pythian © 2012 – Pythian

Integrated Data Warehouse with Hadoop and Oracle Database

by Chen (Gwen) Shapira on Oct 05, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 12

Statistics

Integrated Data Warehouse with Hadoop and Oracle Database Presentation Transcript

https://twitter.com	8
https://si0.twimg.com	3
http://instacurate.com	1