More Related Content
Similar to Big data technologies by Emerging India Analytics
Similar to Big data technologies by Emerging India Analytics (20)
Recently uploaded
Recently uploaded (20)
Big data technologies by Emerging India Analytics
- 2. 2 Vision To become leading consulting and training provider in the field of Data Analytics, Machine Learning, Big Data in India & Overseas. Mission To create value for our customers by providing consulting services and to impart high quality training & skill enhancement programs for employability. About Us Emerging India is promoted by professionals from IIT’s, IIM’s, MBAs and experts from Education and IT Industry. We are one of the India’s fastest growing Analytics/ IT consulting and training companies. We offer services in both consulting and training domain including NASSCOM certified professional programs (designed to bridge the gap between academics and Industry) and Data Analytics/ Cyber Security/ IoT/ Robotics/ AI/ Blockchain consulting solutions. We are also proud NASSCOM member and NASSCOM SSC Licensed Training Partner for the northern region in India.. As NASSCOM licensed training partner, Emerging India is proudly taking NASSCOM SSC initiatives to the next level in the field of Data Analytics to enhance the technical skills of students & working professionals.
- 3. Mrs. Rakhi Singh, Delivery Head (NASSCOM Certified Trainer) Mr. Mayank Jain, Big Data Developer and Analyst Mr. Kapil Sharma Center-Head cum Trainer (Certified by North-western University) Speakers:
- 4. What is HDFS HDFS stands for Hadoop Distributed File System Built on top of Ext3/Ext4 file System Designed to store large amount of data reliably and efficiently Ensure 100% data availability (High Availability Cluster) Do not permit Update Operation Built for OLAP, not for OLTP
- 22. Architecture of HDFS Ext3/Ext4 SecondaryName Node Name Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node Data Node HDFS Master Slave
- 23. MR Operation A.txt 100mb Client1 a.txt code1 A . t x t A .t x t
- 24. MR Framwork Mapper Reducer < Key_In,Val_In,Key_Out,Val_out> < Key_In,Val_In,Key_Out,Val_out> Input Patitioner Output Hi Hello Hi Hadoop World Hive Hadoop Hive Hi Hello World a.txt Hi Hello Hi Hadoop World Hive Hadoop Hive Hi Hello World block1 block2 Mapper1 Mapper2 1,Hi 2,Hello 3,Hi 4,Hadoop 5,World 1,Hive 2,Hadoop 3,Hive 4,Hi 5,Hello 6,World Reducer1 Hi,1 Hello,1 Hi,1 Hadoop,1 World,1 Hive,1 Hadoop,1 Hive,1 Hi,1 Hello,1 World,1
- 25. MR Framwork Mapper Reducer < Key_In,Val_In,Key_Out,Val_out> < Key_In,Val_In,Key_Out,Val_out> Input Patitioner Output Reducer1 Hi,1 Hello,1 Hi,1 Hadoop,1 World,1 Hive,1 Hadoop,1 Hive,1 Hi,1 Hello,1 World,1 Hi,<1,1,1> Hello,<1,1> Hadoop,<1,1> World,<1,1> Hive,<1,1> Hi,3 Hello,2 Hadoop,2 World,2 Hive,2
- 26. HIVE Driver Execution Engine =MapReduce Compiler Translator Client Submits SQL Convert SQL to Map Reduce Metastore Derby
- 27. Hive Database Types – Internal (Managed table) & External Internal(default): In case of drop operation, it will delete data + metadata External Table: It drop only metadata Optimization Technique Partitioning Bucketing
- 28. Introduction to HBase HBase is a Nosql, non-relational, distributed column-oriented database on top of Hadoop. NoSQL - NoSQL database are databases that doesn't use SQL engine as query engine. Hbase Daemons Daemons are services that run on individual machines and communicate with each other HMaster — Master server of HBase, contains all meta data. HRegionserver — Slave server of Hbase, contains the actual data. HQuorumpeer — Zookeeper daemons for co-ordination service. Advantages of using HBase Provides a highly scalable database with nativity with hadoop. Nodes can be added on the fly.
- 29. HBase ( LSM Tree)
- 31. HBase Data Model
- 33. Introduction to Spark Key Features • RDD • DAG • Dataframe • Lazzy
- 35. Our Location A H-196,304,Iind Floor Sector 63, ,Noida – 201301 Our Phone +91 120-4169097 +91 8860599698 Email / Website info@emergingindiagroup.com https://www.emergingindiagro up.com Get in Touch with Us We would be glad to hear from you !