First NL-HUG: Large-scale data processing at SARA with Apache Hadoop
•Download as ODP, PDF•
1 like•1,072 views
An introduction to large-scale data processing, and a short overview of SARA's Hadoop related activities.
Report
Share
Report
Share
![Large-scale data processing [at SARA] [with Apache Hadoop] Evert Lammerts February 9, 2012, Netherlands Hadoop User Group](https://image.slidesharecdn.com/presentation-120209045015-phpapp01/85/First-NL-HUG-Large-scale-data-processing-at-SARA-with-Apache-Hadoop-1-320.jpg)
Recommended
Large-Scale Data Storage and Processing for Scientists with Hadoop
1. The document discusses large-scale data storage and processing options for scientists in the Netherlands, focusing on Hadoop and its components HDFS and MapReduce.
2. HDFS provides a distributed file system that stores very large datasets across clusters of machines, while MapReduce allows processing of datasets in parallel across a cluster.
3. A case study is described that uses HDFS for storage of a 2.7TB text file and MapReduce for analyzing the data to study category evolution in Wikipedia articles over time.
Notes on data-intensive processing with Hadoop Mapreduce
These are slides used in a guest lecture at Eindhoven Technical University, on May 30th, 2012. Adapted from slides by Jimmy Lin.
Hadoop @ Sara & BiG Grid
This document discusses large-scale data processing using Apache Hadoop at SARA and BiG Grid. It provides an introduction to Hadoop and MapReduce, noting that data is easier to collect, store, and analyze in large quantities. Examples are given of projects using Hadoop at SARA, including analyzing Wikipedia data and structural health monitoring. The talk outlines the Hadoop ecosystem and timeline of its adoption at SARA. It discusses how scientists are using Hadoop for tasks like information retrieval, machine learning, and bioinformatics.
Introduction to SARA's Hadoop Hackathon - dec 7th 2010
This document summarizes an agenda for the SARA Hadoop Hackathon on December 7, 2010. It provides background on Hadoop and how it relates to earlier technologies like Nutch and MapReduce. It then outlines the agenda for the day which includes introductions, presentations on MapReduce at University of Twente and a kickoff for the hackathon project building period. An optional tour of the SARA facilities is also included. The day will conclude with presentations of hackathon results.
Introduction NL-HUG (April)
Data science can help organizations extract value from their data. The document discusses various institutions in Amsterdam focused on data and computing including Science Park Amsterdam, Nikhef which conducted research for the Large Hadron Collider, SARA which maintains a national supercomputer and internet infrastructure, CWI which was one of SARA's founding institutions, and BiG Grid which operates a national computing grid. It also mentions upcoming talks on sensor data and information retrieval and encourages participants to share thoughts and potentially use the new Hadoop cluster.
Hadoop: Distributed data processing
The document summarizes a presentation given by Amr Awadallah of Cloudera on Hadoop. It discusses how current storage systems are unable to perform computation, and how Hadoop addresses this through its marriage of HDFS for scalable storage and MapReduce for distributed processing. It provides an overview of Hadoop's history and design principles such as managing itself, scaling performance linearly, and moving computation to data.
Hadoop: Distributed Data Processing
Hadoop is a scalable distributed system for storing and processing large datasets across commodity hardware. It consists of HDFS for storage and MapReduce for distributed processing. A large ecosystem of additional tools like Hive, Pig, and HBase has also developed. Hadoop provides significantly lower costs for data storage and analysis compared to traditional systems and is well-suited to unstructured or structured big data. It has seen wide adoption at companies like Yahoo, Facebook, and eBay for applications like log analysis, personalization, and fraud detection.
Hadoop for beginners free course ppt
This is a power point presentation on Hadoop and Big Data. This covers the essential knowledge one should have when stepping into the world of Big Data.
This course is available on hadoop-skills.com for free!
This course builds a basic fundamental understanding of Big Data problems and Hadoop as a solution. This course takes you through:
• This course builds Understanding of Big Data problems with easy to understand examples and illustrations.
• History and advent of Hadoop right from when Hadoop wasn’t even named Hadoop and was called Nutch
• What is Hadoop Magic which makes it so unique and powerful.
• Understanding the difference between Data science and data engineering, which is one of the big confusions in selecting a carrier or understanding a job role.
• And most importantly, demystifying Hadoop vendors like Cloudera, MapR and Hortonworks by understanding about them.
This course is available for free on hadoop-skills.com
Recommended
Large-Scale Data Storage and Processing for Scientists with Hadoop
1. The document discusses large-scale data storage and processing options for scientists in the Netherlands, focusing on Hadoop and its components HDFS and MapReduce.
2. HDFS provides a distributed file system that stores very large datasets across clusters of machines, while MapReduce allows processing of datasets in parallel across a cluster.
3. A case study is described that uses HDFS for storage of a 2.7TB text file and MapReduce for analyzing the data to study category evolution in Wikipedia articles over time.
Notes on data-intensive processing with Hadoop Mapreduce
These are slides used in a guest lecture at Eindhoven Technical University, on May 30th, 2012. Adapted from slides by Jimmy Lin.
Hadoop @ Sara & BiG Grid
This document discusses large-scale data processing using Apache Hadoop at SARA and BiG Grid. It provides an introduction to Hadoop and MapReduce, noting that data is easier to collect, store, and analyze in large quantities. Examples are given of projects using Hadoop at SARA, including analyzing Wikipedia data and structural health monitoring. The talk outlines the Hadoop ecosystem and timeline of its adoption at SARA. It discusses how scientists are using Hadoop for tasks like information retrieval, machine learning, and bioinformatics.
Introduction to SARA's Hadoop Hackathon - dec 7th 2010
This document summarizes an agenda for the SARA Hadoop Hackathon on December 7, 2010. It provides background on Hadoop and how it relates to earlier technologies like Nutch and MapReduce. It then outlines the agenda for the day which includes introductions, presentations on MapReduce at University of Twente and a kickoff for the hackathon project building period. An optional tour of the SARA facilities is also included. The day will conclude with presentations of hackathon results.
Introduction NL-HUG (April)
Data science can help organizations extract value from their data. The document discusses various institutions in Amsterdam focused on data and computing including Science Park Amsterdam, Nikhef which conducted research for the Large Hadron Collider, SARA which maintains a national supercomputer and internet infrastructure, CWI which was one of SARA's founding institutions, and BiG Grid which operates a national computing grid. It also mentions upcoming talks on sensor data and information retrieval and encourages participants to share thoughts and potentially use the new Hadoop cluster.
Hadoop: Distributed data processing
The document summarizes a presentation given by Amr Awadallah of Cloudera on Hadoop. It discusses how current storage systems are unable to perform computation, and how Hadoop addresses this through its marriage of HDFS for scalable storage and MapReduce for distributed processing. It provides an overview of Hadoop's history and design principles such as managing itself, scaling performance linearly, and moving computation to data.
Hadoop: Distributed Data Processing
Hadoop is a scalable distributed system for storing and processing large datasets across commodity hardware. It consists of HDFS for storage and MapReduce for distributed processing. A large ecosystem of additional tools like Hive, Pig, and HBase has also developed. Hadoop provides significantly lower costs for data storage and analysis compared to traditional systems and is well-suited to unstructured or structured big data. It has seen wide adoption at companies like Yahoo, Facebook, and eBay for applications like log analysis, personalization, and fraud detection.
Hadoop for beginners free course ppt
This is a power point presentation on Hadoop and Big Data. This covers the essential knowledge one should have when stepping into the world of Big Data.
This course is available on hadoop-skills.com for free!
This course builds a basic fundamental understanding of Big Data problems and Hadoop as a solution. This course takes you through:
• This course builds Understanding of Big Data problems with easy to understand examples and illustrations.
• History and advent of Hadoop right from when Hadoop wasn’t even named Hadoop and was called Nutch
• What is Hadoop Magic which makes it so unique and powerful.
• Understanding the difference between Data science and data engineering, which is one of the big confusions in selecting a carrier or understanding a job role.
• And most importantly, demystifying Hadoop vendors like Cloudera, MapR and Hortonworks by understanding about them.
This course is available for free on hadoop-skills.com
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
This document provides an introduction to big data and the installation of a single-node Apache Hadoop cluster. It defines key terms like big data, Hadoop, and MapReduce. It discusses traditional approaches to handling big data like storage area networks and their limitations. It then introduces Hadoop as an open-source framework for storing and processing vast amounts of data in a distributed fashion using the Hadoop Distributed File System (HDFS) and MapReduce programming model. The document outlines Hadoop's architecture and components, provides an example of how MapReduce works, and discusses advantages and limitations of the Hadoop framework.
Seminar_Report_hadoop
This document provides an introduction and overview of Hadoop, an open-source framework for distributed storage and processing of large datasets across clusters of computers. It discusses how Hadoop uses MapReduce and HDFS to parallelize workloads and store data redundantly across nodes to solve issues around hardware failure and combining results. Key aspects covered include how HDFS distributes and replicates data, how MapReduce isolates processing into mapping and reducing functions to abstract communication, and how Hadoop moves computation to the data to improve performance.
Hadoop technology doc
Hadoop is an open source framework that allows for the distributed processing of large data sets across clusters of computers. It uses a MapReduce programming model where the input data is distributed, mapped and transformed in parallel, and the results are reduced together. This process allows for massive amounts of data to be processed efficiently. Hadoop can handle both structured and unstructured data, uses commodity hardware, and provides reliability through data replication across nodes. It is well suited for large scale data analysis and mining.
EclipseCon Keynote: Apache Hadoop - An Introduction
Todd Lipcon explains why you should be interested in Apache Hadoop, what it is, and how it works. Todd also brings to light the Hadoop ecosystem and real business use cases that evolve around Hadoop and the ecosystem.
Introduction to Hadoop Technology
This document discusses Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It describes how Hadoop uses HDFS for distributed storage and fault tolerance, YARN for resource management, and MapReduce for parallel processing of large datasets. It provides details on the architecture of HDFS including the name node, data nodes, and clients. It also explains the MapReduce programming model and job execution involving map and reduce tasks. Finally, it states that as data volumes continue rising, Hadoop provides an affordable solution for large-scale data handling and analysis through its distributed and scalable architecture.
Introduction to Hadoop part1
Accompanying slides for the class “Introduction to Hadoop” at the PRACE Autumn school 2020 - HPC and FAIR Big Data organized by the faculty of Mechanical Engineering of the University of Ljubljana (Slovenia).
Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...
Bigdata Hadoop, Its components and a Hadoop project is described in Details.
Visit http://hadoop-beginners.blogspot.com to see Hadoop Tutorials.
Thanks for the visit. :)
Hadoop Report
The document provides an abstract for a paper on the Hadoop framework. It discusses how Hadoop is a software framework that supports data-intensive distributed applications under an open source license. It was inspired by Google's MapReduce and Google File System papers. The paper will represent the history, development, and current situation of Hadoop technology. It is now maintained by the Apache Software Foundation via Cloudera. The paper will include chapters on an introduction to Hadoop, its history, key technologies like MapReduce and HDFS, other related Apache projects, and instructions for setting up a single node Hadoop cluster.
Hadoop Presentation - PPT
This document outlines the objectives, key concepts, and curriculum for a Big Data and Hadoop training module. The objectives are to understand what Big Data is, the Hadoop ecosystem and its features, career opportunities, and the training curriculum. It defines Big Data, Hadoop, and the Hadoop ecosystem. It discusses the V's of Big Data and domains where Big Data is applicable. It also outlines job roles in the Big Data industry, potential employers, career paths, and the 10-module training curriculum covering topics like Hadoop, MapReduce, Pig, Hive, HBase, Zookeeper and Oozie.
Hadoop: The Default Machine Learning Platform ?
Apache Hadoop, since its humble beginning as an execution engine for web crawler and building search indexes, has matured into a general purpose distributed application platform and data store. Large Scale Machine Learning (LSML) techniques and algorithms proved to be quite tricky for Hadoop to handle, ever since we started offering Hadoop as a service at Yahoo in 2006. In this talk, I will discuss early experiments of implementing LSML algorithms on Hadoop at Yahoo. I will describe how it changed Hadoop, and led to generalization of the Hadoop platform to accommodate programming paradigms other than MapReduce. I will unveil some of our recent efforts to incorporate diverse LSML runtimes into Hadoop, evolving it to become *THE* LSML platform. I will also make a case for an industry-standard LSML benchmark, based on common deep analytics pipelines that utilize LSML workload.
Hadoop
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. It uses a simple programming model called MapReduce that automatically parallelizes and distributes work across nodes. Hadoop consists of Hadoop Distributed File System (HDFS) for storage and MapReduce execution engine for processing. HDFS stores data as blocks replicated across nodes for fault tolerance. MapReduce jobs are split into map and reduce tasks that process key-value pairs in parallel. Hadoop is well-suited for large-scale data analytics as it scales to petabytes of data and thousands of machines with commodity hardware.
Big data and hadoop
Big data is being collected from many sources like the web, social networks, and businesses. Hadoop is an open source software framework that can process large datasets across clusters of computers. It uses a programming model called MapReduce that allows automatic parallelization and fault tolerance. Hadoop uses commodity hardware and can handle various data formats and large volumes of data distributed across clusters. Companies like Cloudera provide tools and services to help users manage and analyze big data with Hadoop.
CityLABS Workshop: Working with large tables
This document discusses working with large tables and big data processing. It introduces distributed computing as an approach to process large datasets by distributing data across multiple nodes and parallelizing operations. The document then outlines using Apache Hadoop and the MK Data Hub cluster to distribute data storage and processing. It demonstrates how to use tools like Hue, Hive, and Pig to analyze tabular data in a distributed manner at scale. Finally, hands-on examples are provided for computing TF-IDF statistics on the large Gutenberg text corpus.
Hadoop and big data
This presentation helps you understand the basics of Hadoop.
What is Big Data?? How google search so fast and what is MapReduce algorithm? all these questions will be answered in the presentation.
Hadoop
This document is a curriculum seminar report on Hadoop submitted by a computer science student to their professor. It includes sections on the need for new technologies to handle large and diverse datasets, the history and origin of Hadoop, descriptions of the key Hadoop components like HDFS and MapReduce, and comparisons of Hadoop to RDBMS systems and discussions of its disadvantages. The report provides an overview of Hadoop for educational purposes.
Hadoop bigdata overview
This document provides an overview of Hadoop, an open source framework for distributed storage and processing of large datasets across clusters of computers. It discusses that Hadoop was created to address the challenges of "Big Data" characterized by high volume, variety and velocity of data. The key components of Hadoop are HDFS for storage and MapReduce as an execution engine for distributed computation. HDFS uses a master-slave architecture with a NameNode master and DataNode slaves, and provides fault tolerance through data replication. MapReduce allows processing of large datasets in parallel through mapping and reducing functions.
Hadoop Technologies
The document discusses various Hadoop technologies including HDFS, MapReduce, Pig/Hive, HBase, Flume, Oozie, Zookeeper, and HBase. HDFS provides reliable storage across multiple machines by replicating data on different nodes. MapReduce is a framework for processing large datasets in parallel. Pig and Hive provide high-level languages for analyzing data stored in Hadoop. Flume collects log data as it is generated. Oozie manages Hadoop jobs. Zookeeper allows distributed coordination. HBase provides a fault-tolerant way to store large amounts of sparse data.
Sf NoSQL MeetUp: Apache Hadoop and HBase
What is Apache Hadoop, what does it do? How does Hadoop Work? What is Apache HBase? Hadoop and HBase use cases.
Hadoop tutorial
Hadoop is a software framework that allows for distributed processing of large data sets across clusters of computers. It uses MapReduce and HDFS to parallelize tasks, distribute data storage, and provide fault tolerance. Applications of Hadoop include log analysis, data mining, and machine learning using large datasets at companies like Yahoo!, Facebook, and The New York Times.
Neo4j vs giraph
This document surveys and compares three large-scale graph processing platforms: Apache Giraph, Hadoop-MapReduce, and Neo4j. It analyzes their programming models and performance based on previous studies. Hadoop was found to have the worst performance for graph algorithms due to its lack of optimizations for graphs. Giraph was generally the fastest platform due to its in-memory computations and message passing model. Neo4j performed well for small graphs due to its caching but did not scale as well as distributed platforms for large graphs. The document concludes that distributed graph-specific platforms like Giraph outperform generic platforms for most graph problems.
Hadoop and BigData - July 2016
This document provides an overview of Hadoop and Big Data. It begins with introducing key concepts like structured, semi-structured, and unstructured data. It then discusses the growth of data and need for Big Data solutions. The core components of Hadoop like HDFS and MapReduce are explained at a high level. The document also covers Hadoop architecture, installation, and developing a basic MapReduce program.
Big data and hadoop
This document provides an overview of big data and Hadoop. It discusses what big data is, its types including structured, semi-structured and unstructured data. Some key sources of big data are also outlined. Hadoop is presented as a solution for managing big data through its core components like HDFS for storage and MapReduce for processing. The Hadoop ecosystem including other related tools like Hive, Pig, Spark and YARN is also summarized. Career opportunities in working with big data are listed in the end.
More Related Content
What's hot
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
This document provides an introduction to big data and the installation of a single-node Apache Hadoop cluster. It defines key terms like big data, Hadoop, and MapReduce. It discusses traditional approaches to handling big data like storage area networks and their limitations. It then introduces Hadoop as an open-source framework for storing and processing vast amounts of data in a distributed fashion using the Hadoop Distributed File System (HDFS) and MapReduce programming model. The document outlines Hadoop's architecture and components, provides an example of how MapReduce works, and discusses advantages and limitations of the Hadoop framework.
Seminar_Report_hadoop
This document provides an introduction and overview of Hadoop, an open-source framework for distributed storage and processing of large datasets across clusters of computers. It discusses how Hadoop uses MapReduce and HDFS to parallelize workloads and store data redundantly across nodes to solve issues around hardware failure and combining results. Key aspects covered include how HDFS distributes and replicates data, how MapReduce isolates processing into mapping and reducing functions to abstract communication, and how Hadoop moves computation to the data to improve performance.
Hadoop technology doc
Hadoop is an open source framework that allows for the distributed processing of large data sets across clusters of computers. It uses a MapReduce programming model where the input data is distributed, mapped and transformed in parallel, and the results are reduced together. This process allows for massive amounts of data to be processed efficiently. Hadoop can handle both structured and unstructured data, uses commodity hardware, and provides reliability through data replication across nodes. It is well suited for large scale data analysis and mining.
EclipseCon Keynote: Apache Hadoop - An Introduction
Todd Lipcon explains why you should be interested in Apache Hadoop, what it is, and how it works. Todd also brings to light the Hadoop ecosystem and real business use cases that evolve around Hadoop and the ecosystem.
Introduction to Hadoop Technology
This document discusses Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It describes how Hadoop uses HDFS for distributed storage and fault tolerance, YARN for resource management, and MapReduce for parallel processing of large datasets. It provides details on the architecture of HDFS including the name node, data nodes, and clients. It also explains the MapReduce programming model and job execution involving map and reduce tasks. Finally, it states that as data volumes continue rising, Hadoop provides an affordable solution for large-scale data handling and analysis through its distributed and scalable architecture.
Introduction to Hadoop part1
Accompanying slides for the class “Introduction to Hadoop” at the PRACE Autumn school 2020 - HPC and FAIR Big Data organized by the faculty of Mechanical Engineering of the University of Ljubljana (Slovenia).
Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...
Bigdata Hadoop, Its components and a Hadoop project is described in Details.
Visit http://hadoop-beginners.blogspot.com to see Hadoop Tutorials.
Thanks for the visit. :)
Hadoop Report
The document provides an abstract for a paper on the Hadoop framework. It discusses how Hadoop is a software framework that supports data-intensive distributed applications under an open source license. It was inspired by Google's MapReduce and Google File System papers. The paper will represent the history, development, and current situation of Hadoop technology. It is now maintained by the Apache Software Foundation via Cloudera. The paper will include chapters on an introduction to Hadoop, its history, key technologies like MapReduce and HDFS, other related Apache projects, and instructions for setting up a single node Hadoop cluster.
Hadoop Presentation - PPT
This document outlines the objectives, key concepts, and curriculum for a Big Data and Hadoop training module. The objectives are to understand what Big Data is, the Hadoop ecosystem and its features, career opportunities, and the training curriculum. It defines Big Data, Hadoop, and the Hadoop ecosystem. It discusses the V's of Big Data and domains where Big Data is applicable. It also outlines job roles in the Big Data industry, potential employers, career paths, and the 10-module training curriculum covering topics like Hadoop, MapReduce, Pig, Hive, HBase, Zookeeper and Oozie.
Hadoop: The Default Machine Learning Platform ?
Apache Hadoop, since its humble beginning as an execution engine for web crawler and building search indexes, has matured into a general purpose distributed application platform and data store. Large Scale Machine Learning (LSML) techniques and algorithms proved to be quite tricky for Hadoop to handle, ever since we started offering Hadoop as a service at Yahoo in 2006. In this talk, I will discuss early experiments of implementing LSML algorithms on Hadoop at Yahoo. I will describe how it changed Hadoop, and led to generalization of the Hadoop platform to accommodate programming paradigms other than MapReduce. I will unveil some of our recent efforts to incorporate diverse LSML runtimes into Hadoop, evolving it to become *THE* LSML platform. I will also make a case for an industry-standard LSML benchmark, based on common deep analytics pipelines that utilize LSML workload.
Hadoop
Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. It uses a simple programming model called MapReduce that automatically parallelizes and distributes work across nodes. Hadoop consists of Hadoop Distributed File System (HDFS) for storage and MapReduce execution engine for processing. HDFS stores data as blocks replicated across nodes for fault tolerance. MapReduce jobs are split into map and reduce tasks that process key-value pairs in parallel. Hadoop is well-suited for large-scale data analytics as it scales to petabytes of data and thousands of machines with commodity hardware.
Big data and hadoop
Big data is being collected from many sources like the web, social networks, and businesses. Hadoop is an open source software framework that can process large datasets across clusters of computers. It uses a programming model called MapReduce that allows automatic parallelization and fault tolerance. Hadoop uses commodity hardware and can handle various data formats and large volumes of data distributed across clusters. Companies like Cloudera provide tools and services to help users manage and analyze big data with Hadoop.
CityLABS Workshop: Working with large tables
This document discusses working with large tables and big data processing. It introduces distributed computing as an approach to process large datasets by distributing data across multiple nodes and parallelizing operations. The document then outlines using Apache Hadoop and the MK Data Hub cluster to distribute data storage and processing. It demonstrates how to use tools like Hue, Hive, and Pig to analyze tabular data in a distributed manner at scale. Finally, hands-on examples are provided for computing TF-IDF statistics on the large Gutenberg text corpus.
Hadoop and big data
This presentation helps you understand the basics of Hadoop.
What is Big Data?? How google search so fast and what is MapReduce algorithm? all these questions will be answered in the presentation.
Hadoop
This document is a curriculum seminar report on Hadoop submitted by a computer science student to their professor. It includes sections on the need for new technologies to handle large and diverse datasets, the history and origin of Hadoop, descriptions of the key Hadoop components like HDFS and MapReduce, and comparisons of Hadoop to RDBMS systems and discussions of its disadvantages. The report provides an overview of Hadoop for educational purposes.
Hadoop bigdata overview
This document provides an overview of Hadoop, an open source framework for distributed storage and processing of large datasets across clusters of computers. It discusses that Hadoop was created to address the challenges of "Big Data" characterized by high volume, variety and velocity of data. The key components of Hadoop are HDFS for storage and MapReduce as an execution engine for distributed computation. HDFS uses a master-slave architecture with a NameNode master and DataNode slaves, and provides fault tolerance through data replication. MapReduce allows processing of large datasets in parallel through mapping and reducing functions.
Hadoop Technologies
The document discusses various Hadoop technologies including HDFS, MapReduce, Pig/Hive, HBase, Flume, Oozie, Zookeeper, and HBase. HDFS provides reliable storage across multiple machines by replicating data on different nodes. MapReduce is a framework for processing large datasets in parallel. Pig and Hive provide high-level languages for analyzing data stored in Hadoop. Flume collects log data as it is generated. Oozie manages Hadoop jobs. Zookeeper allows distributed coordination. HBase provides a fault-tolerant way to store large amounts of sparse data.
Sf NoSQL MeetUp: Apache Hadoop and HBase
What is Apache Hadoop, what does it do? How does Hadoop Work? What is Apache HBase? Hadoop and HBase use cases.
Hadoop tutorial
Hadoop is a software framework that allows for distributed processing of large data sets across clusters of computers. It uses MapReduce and HDFS to parallelize tasks, distribute data storage, and provide fault tolerance. Applications of Hadoop include log analysis, data mining, and machine learning using large datasets at companies like Yahoo!, Facebook, and The New York Times.
Neo4j vs giraph
This document surveys and compares three large-scale graph processing platforms: Apache Giraph, Hadoop-MapReduce, and Neo4j. It analyzes their programming models and performance based on previous studies. Hadoop was found to have the worst performance for graph algorithms due to its lack of optimizations for graphs. Giraph was generally the fastest platform due to its in-memory computations and message passing model. Neo4j performed well for small graphs due to its caching but did not scale as well as distributed platforms for large graphs. The document concludes that distributed graph-specific platforms like Giraph outperform generic platforms for most graph problems.
What's hot (20)
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
EclipseCon Keynote: Apache Hadoop - An Introduction
EclipseCon Keynote: Apache Hadoop - An Introduction
Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...
Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...
Similar to First NL-HUG: Large-scale data processing at SARA with Apache Hadoop
Hadoop and BigData - July 2016
This document provides an overview of Hadoop and Big Data. It begins with introducing key concepts like structured, semi-structured, and unstructured data. It then discusses the growth of data and need for Big Data solutions. The core components of Hadoop like HDFS and MapReduce are explained at a high level. The document also covers Hadoop architecture, installation, and developing a basic MapReduce program.
Big data and hadoop
This document provides an overview of big data and Hadoop. It discusses what big data is, its types including structured, semi-structured and unstructured data. Some key sources of big data are also outlined. Hadoop is presented as a solution for managing big data through its core components like HDFS for storage and MapReduce for processing. The Hadoop ecosystem including other related tools like Hive, Pig, Spark and YARN is also summarized. Career opportunities in working with big data are listed in the end.
Big Data and Hadoop
Big Data raises challenges about how to process such vast pool of raw data and how to aggregate value to our lives. For addressing these demands an ecosystem of tools named Hadoop was conceived.
Hadoop Developer
This document outlines the modules and topics covered in an Edureka course on Hadoop. The 10 modules cover understanding Big Data and Hadoop architecture, Hadoop cluster configuration, MapReduce framework, Pig, Hive, HBase, Hadoop 2.0 features, and Apache Oozie. Interactive questions are also included to test understanding of concepts like Hadoop core components, HDFS architecture, and MapReduce job execution.
Hadoop
This document provides an overview of big data, Hadoop, and related concepts:
- Big data refers to large datasets that cannot be processed efficiently by traditional systems due to their size. Sources include social media, smartphones, machines, and log files.
- Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It implements the MapReduce programming model.
- Key Hadoop components include HDFS for storage, MapReduce for distributed processing, and related projects like Pig, Hive, HBase, Flume, Oozie, and Sqoop. Companies use Hadoop for applications involving large datasets, such as log analysis, recommendations, and business intelligence
Hadoop
Hadoop is an open-source framework for storing and processing large datasets in a distributed computing environment. It allows for massive data storage, enormous processing power, and the ability to handle large numbers of concurrent tasks across clusters of commodity hardware. The framework includes Hadoop Distributed File System (HDFS) for reliable data storage and MapReduce for parallel processing of large datasets. An ecosystem of related projects like Pig, Hive, HBase, Sqoop and Flume extend the functionality of Hadoop.
Chattanooga Hadoop Meetup - Hadoop 101 - November 2014
Josh Patterson is a principal solution architect who has worked with Hadoop at Cloudera and Tennessee Valley Authority. Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. It allows for consolidating mixed data types at low cost while keeping raw data always available. Hadoop uses commodity hardware and scales to petabytes without changes. Its distributed file system provides fault tolerance and replication while its processing engine handles all data types and scales processing.
Big Data Concepts
This presentation Simplify the concepts of Big data and NoSQL databases & Hadoop components.
The Original Source:
http://zohararad.github.io/presentations/big-data-introduction/
How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
Hadoop was developed to solve problems with data warehousing systems at Yahoo and Facebook that were limited in processing large amounts of raw data in real-time. Hadoop uses HDFS for scalable storage and MapReduce for distributed processing. It allows for agile access to raw data at scale for ad-hoc queries, data mining and analytics without being constrained by traditional database schemas. Hadoop has been widely adopted for large-scale data processing and analytics across many companies.
Introduction to Apache Hadoop Eco-System
The document provides information about Hadoop, its core components, and MapReduce programming model. It defines Hadoop as an open source software framework used for distributed storage and processing of large datasets. It describes the main Hadoop components like HDFS, NameNode, DataNode, JobTracker and Secondary NameNode. It also explains MapReduce as a programming model used for distributed processing of big data across clusters.
Hadoop and Big Data: Revealed
Content presented at a talk on Aug. 29th. Purpose is to inform a fairly technical audience on the primary tenets of Big Data and the hadoop stack. Also, did a walk-thru' of hadoop and some of the hadoop stack i.e. Pig, Hive, Hbase.
Tools and techniques for data science
R is an open source programming language and software environment for statistical analysis and graphics. It is widely used among data scientists for tasks like data manipulation, calculation, and graphical data analysis. Some key advantages of R include that it is open source and free, has a large collection of statistical tools and packages, is flexible, and has strong capabilities for data visualization. It also has an active user community and can integrate with other software like SAS, Python, and Tableau. R is a popular and powerful tool for data scientists.
Hadoop: An Industry Perspective
This document provides an overview of Hadoop and how it can be used for data consolidation, schema flexibility, and query flexibility compared to a relational database. It describes the key components of Hadoop including HDFS for storage and MapReduce for distributed processing. Examples of industry use cases are also presented, showing how Hadoop enables affordable long-term storage and scalable processing of large amounts of structured and unstructured data.
Big data or big deal
The document discusses Hadoop and big data technologies. It begins with an introduction to big data concepts and the various Hadoop components like HDFS, MapReduce, YARN, Hive, Pig and Mahout. It then explains how big data is different from traditional data warehousing through the concept of schema-on-read. Finally, it provides recommendations on tools for working with big data technologies locally and in the cloud, as well as sources of inspiration like sandbox environments, Apache projects and GitHub.
Where does hadoop come handy
This document discusses Hadoop and big data. It begins with definitions of big data and how Hadoop can help with large, complex datasets. It then discusses how Hadoop works with other tools like Pig and Hive. The document outlines different scenarios for big data and whether Hadoop is suitable. It also discusses how big data frameworks have evolved from Google papers. Finally, it provides examples of big data use cases and how education is being democratized with big data tools.
Big data ppt
The document discusses big data, including what it is, sources of big data like social media and stock exchange data, and the three Vs of big data - volume, velocity, and variety. It then discusses Hadoop, the open-source framework for distributed storage and processing of large datasets across clusters of computers. Key components of Hadoop include HDFS for distributed storage, MapReduce for distributed computation, and YARN which manages computing resources. The document also provides overviews of Pig and Jaql, programming languages used for analyzing data in Hadoop.
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
In this interactive webinar, we'll walk through use cases on how you can use advanced analytics like SAS Visual Statistics and In-Memory Statistic with Hortonworks’ data platform (HDP) to reveal insights in your big data and redefine how your organization solves complex problems.
Hadoop
This document discusses big data and the Hadoop platform. It provides an overview of Hadoop versions 1 and 2. Hadoop version 1 included the Hadoop Distributed File System (HDFS) for large-scale storage and the MapReduce programming model. Hadoop version 2 introduced YARN to address MapReduce limitations like scalability and availability. The document also briefly describes the Hadoop ecosystem of related projects like Hive for data warehousing and Pig for data flows. It provides examples of Twitter and Facebook's use of Hadoop for big data analytics.
Hadoop
This document provides an overview of Hadoop, including its history, components, uses, ecosystem, supported operating systems, and alternatives. It describes how Hadoop was created by Doug Cutting and became an Apache project in 2008. The core components of Hadoop are HDFS for storage and MapReduce for distributed processing. Hadoop can be used for applications like log analysis, machine learning, and real-time data processing. Common Hadoop tools include Hive, Pig, HBase, Zookeeper, and Sqoop.
Hadoop_Its_Not_Just_Internal_Storage_V14
This document provides an overview of Hadoop storage perspectives from different stakeholders. The Hadoop application team prefers direct attached storage for performance reasons, as Hadoop was designed for affordable internet-scale analytics where data locality is important. However, IT operations has valid concerns about reliability, manageability, utilization, and integration with other systems when data is stored on direct attached storage instead of shared storage. There are tradeoffs to both approaches that depend on factors like the infrastructure, workload characteristics, and priorities of the organization.
Similar to First NL-HUG: Large-scale data processing at SARA with Apache Hadoop (20)
Chattanooga Hadoop Meetup - Hadoop 101 - November 2014
Chattanooga Hadoop Meetup - Hadoop 101 - November 2014
How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
Predictive Analytics and Machine Learning …with SAS and Apache Hadoop
Recently uploaded
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/how-axelera-ai-uses-digital-compute-in-memory-to-deliver-fast-and-energy-efficient-computer-vision-a-presentation-from-axelera-ai/
Bram Verhoef, Head of Machine Learning at Axelera AI, presents the “How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-efficient Computer Vision” tutorial at the May 2024 Embedded Vision Summit.
As artificial intelligence inference transitions from cloud environments to edge locations, computer vision applications achieve heightened responsiveness, reliability and privacy. This migration, however, introduces the challenge of operating within the stringent confines of resource constraints typical at the edge, including small form factors, low energy budgets and diminished memory and computational capacities. Axelera AI addresses these challenges through an innovative approach of performing digital computations within memory itself. This technique facilitates the realization of high-performance, energy-efficient and cost-effective computer vision capabilities at the thin and thick edge, extending the frontier of what is achievable with current technologies.
In this presentation, Verhoef unveils his company’s pioneering chip technology and demonstrates its capacity to deliver exceptional frames-per-second performance across a range of standard computer vision networks typical of applications in security, surveillance and the industrial sector. This shows that advanced computer vision can be accessible and efficient, even at the very edge of our technological ecosystem.GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Recently uploaded (20)
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
First NL-HUG: Large-scale data processing at SARA with Apache Hadoop
- 1. Large-scale data processing [at SARA] [with Apache Hadoop] Evert Lammerts February 9, 2012, Netherlands Hadoop User Group
- 2. Who's who?
- 5. An introduction to scale @ SARA
- 6. An introduction to Hadoop & MapReduce
- 8. Why large-scale data processing? An introduction to scale @ SARA An introduction to Hadoop & MapReduce Hadoop @ SARA
- 9. (Jimmy Lin, University of Maryland / Twitter, 2011)
- 10. (IEEE Intelligent Systems, 03/04-2009, vol 24, issue 2, p8-12)
- 11. s/knowledge/data/g* HTTP logs, Click data, Query logs, CRM data, Financial data, Social networks, Archives, Crawls, and many more You already have your data (*Jimmy Lin, University of Maryland / Twitter, 2011)
- 15. Anybody with the know-how can generate insights!
- 16. Note: “ the know-how ” = Data Science DevOps Programming algorithms Domain knowledge
- 17. Why large-scale data processing? An introduction to scale @ SARA An introduction to Hadoop & MapReduce Hadoop @ SARA
- 18. SARA the national center for scientific computing Facilitating Science in The Netherlands with Equipment for and Expertise on L arge-Scale Computing , L arge-Scale Data Storage , H igh-Performance Networking , eScience , and Visualization
- 19. Large-scale data != new
- 22. MIMD: Multiple Instruction Multiple Data
- 23. MISD: Multiple Instruction Single Data
- 24. SISD: Single Instruction Single Data (Von Neumann)
- 26. Data parallelism
- 27. Compute @ SARA
- 28. What's different about Hadoop? No more do-it-yourself parallelism – it's hard! But rather linearly scalable data parallelism Separating the what from the how (NYT, 14/06/2006)
- 29. Why large-scale data processing? An introduction to scale @ SARA An introduction to Hadoop & MapReduce Hadoop @ SARA
- 30. A bit of history Nutch* 2002 2004 MR/GFS** 2006 2004 Hadoop * http://nutch.apache.org/ ** http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/gfs.html
- 31. http://wiki.apache.org/hadoop/PoweredBy 2010 - 2012: A Hype in Production
- 33. Move processing to the data
- 34. Process data sequentially, avoid random reads
- 35. Seamless scalability (Jimmy Lin, University of Maryland / Twitter, 2011)
- 37. Extract something of interest
- 38. Create an ordering in intermediate results
- 40. Generate output MapReduce: functional abstraction of step 2 & step 4 (Jimmy Lin, University of Maryland / Twitter, 2011)
- 42. reduce (k', v') -> <k', v'>* All values associated with a single key are sent to the same reducer The framework handles the rest
- 43. The rest? Scheduling, data distribution, ordering, synchronization, error handling...
- 44. An overview of a Hadoop cluster
- 45. The ecosystem Hbase , Hive, Pig, HCatalog, Giraph, Elephantbird, and many others...
- 46. Why large-scale data processing? An introduction to scale @ SARA An introduction to Hadoop & MapReduce Hadoop @ SARA
- 47. Timeline 2009: Piloting Hadoop on Cloud 2010: Test cluster available for scientists 6 machines * 4 cores / 24 TB storage / 16GB RAM Just me! 2011: Funding granted for production service 2012: Production cluster available (~March) 72 machines * 8 cores / 8 TB storage / 64GB RAM Integration with Kerberos for secure multi-tenancy 3 devops, team of consultants
- 48. Architecture
- 49. Components Hadoop, Hive, Pig, Hbase, HCatalog - others?
- 52. Machine Learning
- 53. Econometry
- 54. Bioinformatics
- 55. Computational Ecology / Ecoinformatics
- 56. Machine learning: Infrawatch, Hollandse Brug
- 57. Structural health monitoring 145 sensors 100 Hz 60 seconds 60 minutes 24 hours 365 days x x x x x = large data (Arno Knobbe, LIACS, 2011, http://infrawatch.liacs.nl)
- 59. e.g. Twitter gardenhose data
- 61. e.g. del.ico.us & flickr tags
- 62. Finding named entities: [person company place] names
- 65. Personalization
- 66. Semantic web
- 67. Interest from industry We're opening shop. Come and pilot.
- 70. Hadoop is probably not the best
- 71. Hadoop has momentum
- 73. The data center is your computer
- 74. Where is the data scientist? Much to learn & teach!
- 75. Any questions? [email_address] @eevrt @sara_nl