Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
By: Khalid Imran
Big Data :
Technology Stack
Agenda
▪ Big Data Stack : In a Nutshell
▪ Data Layer
▪ Data Processing Layer
▪ Data Ingestion Layer
▪ Data Presentation La...
Big Data Technology Stack : In a nutshell
3
Data Layer
4
Hadoop Distributed File System (HDFS)
HDFS is a scalable, fault-tolerant Java based distributed file system t...
Data Processing Layer
5
Hadoop MapReduce
Hadoop Map/Reduce is a software framework for distributed processing of large dat...
Data Processing Layer
6
Apache Hive
Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing da...
Data Processing Layer
7
Apache Cassandra
Cassandra is another open-source distributed NoSQL database. It is highly scalabl...
Data Processing Layer
8
Apache Spark
Apache Spark is an open source cluster computing framework for large-scale data proce...
Data Ingestion Layer
9
Apache Flume
Apache Flume is a distributed, reliable, and available service for efficiently collect...
Data Presentation Layer
10
Kibana
Kibana is an analytics and visualization plugin that works with ElasticSearch. It provid...
Operations & Scheduling Layer
11
Ambari
Ambari is an open framework that helps in provisioning, managing and monitoring of...
About Me – Khalid Imran
12
A tester by passion, I’ve spent the past 16+ years testing disparate systems, learning new doma...
Upcoming SlideShare
Loading in …5
×

Big Data Technology Stack : Nutshell

14,518 views

Published on

View the Big Data Technology Stack in a nutshell. This Big Data Technology Stack deck covers the different layers of the Big Data world and summarizes the major technologies in vogue today.

Published in: Data & Analytics
  • Be the first to comment

Big Data Technology Stack : Nutshell

  1. 1. By: Khalid Imran Big Data : Technology Stack
  2. 2. Agenda ▪ Big Data Stack : In a Nutshell ▪ Data Layer ▪ Data Processing Layer ▪ Data Ingestion Layer ▪ Data Presentation Layer ▪ Operations & Scheduling Layer ▪ Security & Governance 2
  3. 3. Big Data Technology Stack : In a nutshell 3
  4. 4. Data Layer 4 Hadoop Distributed File System (HDFS) HDFS is a scalable, fault-tolerant Java based distributed file system that is used for storing large volumes of data in inexpensive commodity hardware. Amazon Simple Storage Service (S3) S3 is a cloud based scalable, distributed file system offering from Amazon. It can be utilized as the data layer in big data applications, coupled with other required components. IBM General Parallel File System (GPFS) / Spectrum Scale GPFS is a high-performance clustered file system developed by IBM.
  5. 5. Data Processing Layer 5 Hadoop MapReduce Hadoop Map/Reduce is a software framework for distributed processing of large data sets on compute clusters of commodity hardware. It is a sub-project of the Apache Hadoop project. The framework takes care of scheduling tasks, monitoring them and re-executing any failed tasks. A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. Apache Pig Pig is a high-level platform for creating MapReduce programs used with Hadoop. Apache Pig allows Apache Hadoop users to write complex MapReduce transformations using a simple scripting language called Pig Latin. Pig translates the Pig Latin script into MapReduce so that it can be executed on the data. Pig Latin can be extended using UDF (User Defined Functions) which the user can write in Java, Python, JavaScript, Ruby or Groovy and then call directly from the language.
  6. 6. Data Processing Layer 6 Apache Hive Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. Hive is used to explore, structure and analyze data, then turn it into actionable business insight. Apache Hive supports analysis of large datasets stored in Hadoop's HDFS and compatible file systems such as Amazon S3 file system. It provides an SQL-like language called HiveQL with schema on read and transparently converts queries to map/reduce, Apache Tez and Spark jobs. Apache HBase HBase is an open-source NoSQL database that provides real-time read/write access to large datasets with extremely low latency as well as fault tolerance. HBase runs on top of HDFS. HBase provides a strong consistency model, and range-based partitioning. Reads, including range-based reads, tend to scale much better on HBase, whereas writes do not scale as well as they do on Cassandra..
  7. 7. Data Processing Layer 7 Apache Cassandra Cassandra is another open-source distributed NoSQL database. It is highly scalable, fault tolerant and can be used to manage huge volumes of data. Cassandra's consistency model is based on Amazon's Dynamo: it provides eventual consistency. This is very appealing for some applications where you want to guarantee the availability of writes. Similarly, Cassandra tends to provide very good write scaling. Apache Storm Storm is a distributed real-time computation system for processing large volumes of high-velocity data. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Apache Solr Apache Solr is the open source platform for searches of data stored in HDFS in Hadoop. Solr powers the search and navigation features of many of the world’s largest Internet sites, enabling powerful full-text search and near real-time indexing. Apache Solr can be used for rapidly finding tabular, text, geo-location or sensor data that is stored in Hadoop.
  8. 8. Data Processing Layer 8 Apache Spark Apache Spark is an open source cluster computing framework for large-scale data processing. Studies have shown that Spark can run up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk for program execution. It provides in-memory computations for increased speed and data processing over MapReduce. It runs on top of existing Hadoop cluster and can access Hadoop data store (HDFS), as well as also process structured data from Hive and streaming data from HDFS, Flume, Kafka, Twitter and other sources. Apache Mahout Apache Mahout is a library of scalable machine-learning algorithms that can be implemented on top of Apache Hadoop and it utilizes the MapReduce paradigm. Machine learning is a discipline of artificial intelligence focused on enabling machines to learn without being explicitly programmed, and it is commonly used to improve future performance based on previous outcomes. Mahout provides the tools and algorithms to automatically find meaningful patterns in those big data sets stored in the HDFS.
  9. 9. Data Ingestion Layer 9 Apache Flume Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of streaming data (e.g. application logs, sensor and machine data, geo-location data and social media) into the HDFS. It has a simple and flexible architecture based on streaming data flows; and is robust and fault tolerant with comes with configurable reliability mechanisms for failover and recovery. Apache Kafka Kafka is a high throughput distributed messaging system. Kafka maintains feeds of messages in categories called topics. Producers are processes that publish messages to a Kafka topic. Consumers are processes that subscribe to topics and process the feed of published messages. Kafka is run as a cluster comprising of one or more servers each of which is called a broker. Apache Sqoop Apache Sqoop is a tool designed to transfer data between Hadoop and relational databases or mainframes. Sqoop can be used to import data from a RDBMS or a mainframe into HDFS, transform the data using Hadoop MapReduce, and then export the data back into an RDBMS.
  10. 10. Data Presentation Layer 10 Kibana Kibana is an analytics and visualization plugin that works with ElasticSearch. It provides real-time summary and charting of streaming data. The visualization capabilities it provides allow users to different charts, plots and maps of large volumes of data.
  11. 11. Operations & Scheduling Layer 11 Ambari Ambari is an open framework that helps in provisioning, managing and monitoring of Apache Hadoop clusters. It simplifies the deployment and maintenance of hosts. Ambari also includes an intuitive web interface that allows one to easily provision, configure and test all the Hadoop services and core components. It also comes with the powerful Ambari Blueprints API that can be utilized for automating cluster installations without any user intervention. Apache Oozie Apache Oozie provides operational service capabilities for a Hadoop cluster, specifically around job scheduling within the cluster. Oozie is a Java based web application that is primarily used to schedule Apache Hadoop jobs. Oozie can combine multiple jobs sequentially into one logical unit of work. It can be integrated with the Hadoop stack, and supports Hadoop jobs for various Apache tools such as MapReduce, Apache Pig, Apache Hive, and Apache Sqoop. Apache ZooKeeper Apache ZooKeeper provides operational services for a Hadoop cluster. It provides a distributed configuration service, a synchronization service and a naming registry for distributed systems that can use Zookeeper to store and mediate updates to important configuration information.
  12. 12. About Me – Khalid Imran 12 A tester by passion, I’ve spent the past 16+ years testing disparate systems, learning new domains, developing innovative solutions, designing test strategies, challenging conventional methods, proving new techniques and embracing emerging tools & technologies. The breadth and depth of my experience cuts across functional testing, non-functional testing, manual, automation, test and project execution methodologies, licensed and open stack tools, platforms, devices, programming languages, custom-built test harnesses and utilities, delivery management, client engagement, on-site, off-shore team dynamics and more. I am currently heading the 1400+ QA strong testing practice at Cybage as a QA Evangelist. I manage the Testing Centre of Excellence (TCoE), lead a team of architects and specialists and assist in deliveries across the organization, pre-sales and business development, solutioning and consultancy, training and process improvement group. I hold multiple certifications namely: CSQA, CSM and CPISI. I welcome any questions or feedback you may have on this presentation.

×