Uploaded byChien Chung Shen
Hadoop Essential for Oracle Professionals
This document provides an overview of Hadoop and the Hadoop ecosystem. It discusses key Hadoop concepts like HDFS, MapReduce, YARN and data locality. It also summarizes SQL on Hadoop using tools like Hive, Impala and Spark SQL. The document concludes with examples of using Sqoop and Flume to move data between relational databases and Hadoop.
More Related Content
Similar to Hadoop Essential for Oracle Professionals
![[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...](https://cdn.slidesharecdn.com/ss_thumbnails/xer2bb6nrdc8pdpev0pc-8-251204082258-7c2fa4a1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...](https://cdn.slidesharecdn.com/ss_thumbnails/8brigo2sbu6qur6gxrra-7-251205085715-6ae07d24-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...](https://cdn.slidesharecdn.com/ss_thumbnails/7nw2xxixrxqdxvrb5wca-6-251205085714-ab09a2ac-thumbnail.jpg?width=640&height=640&fit=bounds)
Hadoop Essential for Oracle Professionals
- 1. Hadoop Essentials for Oracle Professionals 申建忠(FrankChien-Chung Shen) Oracle Authorized Trainer MySQL Authorized Trainer Cloudera Hadoop Authorized Trainer
- 2. Big Data • Volume –Data amount • Variety – Data type • Velocity – Data processing Speed Value
- 3. Hadoop Common(MR v1) HDFS (HadoopDistributed FileSystem) Map Reduce (mapred API/Framework/Resource Management)
- 4. Hadoop Common(MR v2) HDFS MapReduce (mapreduce API/Framework) YARN-Yet Another Resource Negotiator (YARN API/Resource Management)
- 5. Hadoop EcoSystem HDFS(Hadoop DistributedFileSystem) Map Reduce (V1) YARN (Yet Another Resource Negotiator) Impala Spark Storm HBase Map Reduce (V2) mapred (Java Api) mapeduce Giraph ZooKeeper HUE Ozzie Solr Flume MahoutHive Pig Sqoop HttpFS WebHDFS Hadoop fsKafka
- 6. Hadoop EcoSystem HDFS(Hadoop DistributedFileSystem) Map Reduce (V1) YARN (Yet Another Resource Negotiator) Impala Spark Storm HBase Map Reduce (V2) mapred (Java Api) mapeduce Giraph ZooKeeper HUE Ozzie Solr Flume SQL PigLatin SQL Scala/ Java/ Python Java Logs Message RDBMS HTTP/S MahoutHive Pig Sqoop HttpFS WebHDFS Hadoop fsKafka
- 7. Hadoop EcoSystem HDFS(Hadoop DistributedFileSystem) Map Reduce (V1) YARN (Yet Another Resource Negotiator) Impala Mahout Spark Storm HBase Map Reduce (V2) Hive Pig mapred (Java Api) mapeduce Giraph ZooKeeper HUE Ozzie Solr Flume Sqoop HttpFS WebHDFS Hadoop fs Batch SQL Interactive SQL Machine Learning Streaming Cluster Coordination Workflow Graph Processing Scripting Language In-Memory Processing NoSQL Database Browser GUI Search Data Collector Data Exchange Kafka Data Collector Http Access Http Access Command Line
- 8. Server Process Computing Layer Instance DBWR Oracle Server StorageLayer Database block blockblock block
- 9. Instance DBWR Real Application Cluster Database block DBWR Instance blockblock blockblock block Server Process coordinator Server process Server Process Parallel slave Parallel slave block block
- 10. I/O is theproblem • Faster I/O – ASM – Exadata • Less I/O – Index Access – Compression – Partitioned Table/Index – Materialized View/View Log – Clustered Table – Index Organized Table – Exadata
- 11. /dev/sdb1 /dev/sdc1/dev/sdd1 /dev/sde1/dev/sdf1/dev/sdg1 DATA Physical block(4k) Allocation unit 1/2/4/8/16/32/64M metadata +ASM metadata metadata metadata /dev/sdh /dev/sdh1 Oracle ASM /dev/sdb
- 12. SQL> create tablespacets1 datafile ‘+DATA’ size 3M; /dev/sdb1 /dev/sdc1/dev/sdd1 /dev/sde1 /dev/sdf1/dev/sdg1 DATA ASM volume /acfs_data acfs Physical block(4k) Allocation unit 1/2/4/8/16/32/64M metadata orcl +ASM metadata metadata metadata /dev/sdh /dev/sdh1 /dev/sdh2 Primary extent Mirror extent Primary extent
- 13. Instance DBWR Exadata Database DBWR Instance block block Server Process Server Process Exadata CellExadata Cell Exadata Cell Exadata Cell Smart Scans Exadata Cell block block block block block Filtering block block block block block block
- 14. Hadoop Concepts • MovingProcessing is faster than moving Data • Data Locality • HDFS was designed for many millions of large files, not billions of small files • Write Once, Read Many times • Default Replication(3)
- 15. Why not MovingData • Moving data from storage layer to computing layer • Can’t start processing until data movement completion • More nodes, more contentions
- 16. Storage Layer Data NodeData Node Data Node Data Node Name Node
- 17. HDFS $ hadoop put/home/frank/data1.txt Data Node1 Data Node2 Data Node3 Data Node4 Name Node Datanode:1,2,3 /user/frank/data1.txt Replications:3 Blocksize:64M 150M 64M Datanode:1,2,3
- 18. HDFS $ hadoop put/home/frank/data1.txt Data Node1 Data Node2 Data Node3 Data Node4 Name Node Datanode:1,2,3 Datanode:2,3,4 /user/frank/data1.txt Replications:3 Blocksize:64M 150M 64M 64M
- 19. HDFS $ hadoop put/home/frank/data1.txt Data Node1 Data Node2 Data Node3 Data Node4 Name Node Datanode:1,2,3 Datanode:2,3,4 Datanode:1,2,4 /user/frank/data1.txt Replications:3 Blocksize:64M 150M 64M 64M 22M
- 20.
- 21. YARN Node Manager Node Manager Node Manager Node Manager Resource Manager $ hadoop jarwordcount.jar /user/frank/data1.txt /user/frank/wc_data1_output AM Application Master
- 22. YARN Node Manager Node Manager Node Manager Node Manager Resource Manager $ hadoop jarwordcount.jar /user/frank/data1.txt /user/frank/wc_data1_output AM Mapper Mapper Mapper
- 23. YARN Node Manager Node Manager Node Manager Node Manager Resource Manager $ hadoop jarwordcount.jar /user/frank/data1.txt /user/frank/wc_data1_output AM Mapper Mapper Mapper Reduce r Reduce r
- 24. Data Locality Data Node NodeManager Data Node Node Manager Data Node Node Manager Data Node Node Manager Name Node Resource ManagerJVM daemons
- 25. Suggested Host Configurations DataNode Node Manager Data Node Node Manager Data Node Node Manager Data Node Node Manager Name Node Resource ManagerJVM daemons
- 26. Mapper Mapper Mapper Reducer Reducer Input Spilt Input Spilt Input Spilt Part- 00000 Part- 00001 Shuffle And Sort Map ReduceArchitecture JAR(computing) HDFS://user/frank/data1.txt HDFS://user/frank/wc_data1_output
- 27. the cat saton the mat the aardvark sat on the sofa Mapper Mapper (the, 1)(cat, 1)(sat, 1) (on, 1)(the, 1)(mat, 1) (the, 1)(aardvark, 1) (sat, 1)(on, 1)(the, 1) (sofa, 1) Reducer Reducer aardvark, 1 cat, 1 mat, 1 on, 2 sat, 2 sofa, 1 the, 4 (aardvark, 1) (cat, 1) (mat, 1) (on, 1, 1) (sat, 1, 1),(sofa, 1),(the, 1, 1, 1, 1) Shuffle Sort
- 28. NoSQL NoSQL No OnlySQL No, SQL Time
- 29. SQL on Hadoop 0 OperationalSQL < 100 msec Batch SQL > 20 min Interactive SQL 100 msec ~ 20 min 100 msec 20 min
- 30. SQL on Hadoop •Batch SQL – Hive • Based on MapReduce • Based on Tez(Hortonworks) • Interactive SQL – Spark SQL(Hive based on Spark) – Impala(Cloudera) – Apache Drill – Presto(Facebook) • Operational SQL – Apache Phoenix(SQL on Hbase)
- 31. Empid Ename SalaryHire_date Schema on Write Create table before inserting data into
- 32. 001 002 Frank Linda 25149191 85091233 20000 19000 M F 2000-11-01 2001-01-05 Schema on Write EmpidEname Salary Hire_date 001 Frank 20000 2000-11-01 002 Linda 19000 2001-01-05 Missing Missing
- 33. Schema on Read EmpidEname Job Salary Dept Hire_date 001 002 Frank Linda 25149191 85091233 20000 19000 M F 2000-11-01 2001-01-05
- 34. Schema on Read EmpidEname Job Salary Dept Hire_date 001 002 Frank Linda 25149191 85091233 20000 19000 M F 2000-11-01 2001-01-05
- 35. Sqoop /user/cloudera/t1/part-00000 /user/cloudera/t1/part-00001 RDBMS HDFS t1 JDBC Map Reduce job MapReduce Hive Select id From t1 Where name=‘Frank’ MapReduce job MetaStore Hive Architecture Data Dict. Table definition Table Data
- 36. /user/cloudera/t1/part-00000 /user/cloudera/t1/part-00001 Database HDFS t1(external table) Oracle SQLConnector for HDFS Data Dict. Table Data Instance Oracle SQL Connector for HDFS Data Pump files Delimited text files Delimited text files in Apache Hive tables
- 37.
- 38. Data Warehouse basedon RDBMS OLTP Data Warehouse Extract/Transform/Load Web Server BI Tools Ad Hoc Query ReportUser Access logs
- 39. Integrated RDBMS withHadoop OLTP Data Warehouse Web Server Hadoop EcoSystem User SqoopSqoop BI Tools Ad Hoc Query Report Access logs
- 40.