Hadoop: Beyond MapReduce
•Download as PPTX, PDF•
1 like•1,832 views
Overview of the above and beyond MapReduce, for the HPC/science community. Key point: move up the stack, reuse what is there. But: some of these people are capable of writing their own YARN apps, so they should be encouraged to do so if they see a need.
Report
Share
Report
Share
![© Hortonworks Inc.
Hadoop MapReduce
1. Map: events <k,v>* pairs
2. Reduce: <k,[v1, v2,.. vn]> <k,v'>
•Map trivially parallelisable on blocks in a file
•Reduce parallelise on keys
•MapReduce engine can execute Map and
Reduce sequences against data
•HDFS provides data location for work
placement
Page 2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Llnl talk
These slides are from a recent talk I gave at Lawrence Livermore Labs.
The talk gives an architectural outline of the MapR system and then discusses how this architecture facilitates large scale machine learning algorithms.
Introduction to Yarn
YARN was introduced as part of Hadoop 2.0 to address limitations in the original MapReduce (MR1) architecture like scalability bottlenecks and underutilization of resources. YARN introduces a global ResourceManager and per-node NodeManagers to allocate cluster resources to distributed applications. It allows various distributed processing frameworks beyond MapReduce to share common cluster resources. Applications request containers for ApplicationMasters that then negotiate resources from YARN to run application components in containers across nodes. Existing MapReduce jobs can also run unchanged on YARN.
Introduction to map reduce
MapReduce provides a programming model for processing large datasets in a distributed, parallel manner. It involves two main steps - the map step where the input data is converted into intermediate key-value pairs, and the reduce step where the intermediate outputs are aggregated based on keys to produce the final results. Hadoop is an open-source software framework that allows distributed processing of large datasets across clusters of computers using MapReduce.
Tez: Accelerating Data Pipelines - fifthel
This document provides an overview of Tez, an Apache project that provides a framework for executing data processing jobs on Hadoop clusters. Tez allows expressing data processing jobs as directed acyclic graphs (DAGs) of tasks and executes these tasks in a optimized manner. It addresses limitations of MapReduce by providing a more flexible execution engine that can optimize performance and resource utilization.
TechEvent Operating MapR Hadoop Cluster for a year
The document discusses operating a MapR Hadoop cluster for a year to handle large volumes of automotive test data. Key aspects included setting up a 24-node cluster running Red Hat Linux with MapR-FS and YARN, separating control and worker nodes, and configuring services for high availability. Administration required deep knowledge of Hadoop components and customization to evolving requirements. Lessons included planning security from the start, maintaining performance and stability, and close collaboration with development teams.
Map Reduce
This document provides an overview of MapReduce, including:
- MapReduce is a programming model for processing large datasets in parallel across clusters of computers.
- It works by breaking the processing into map and reduce functions that can be run on many machines.
- Examples are given like word counting, distributed grep, and analyzing web server logs.
EMR AWS Demo
The document outlines how to run an Elastic MapReduce job on AWS. It discusses uploading input data and a MapReduce jar file to S3 storage. It then shows how to create an EMR Hadoop cluster with a specified number of workers and hardware configuration. Finally, it demonstrates submitting a MapReduce job to the cluster, monitoring its status, and terminating the cluster once complete.
Join optimization in hive
This document discusses optimization techniques for map join in Hive. It describes:
1) Previous approaches to common join and map join in Hive and their limitations.
2) Optimized map join techniques like uploading small tables to distributed cache and performing local joins to avoid shuffle.
3) Using JDBM for hash tables caused performance issues so alternative approaches were evaluated.
4) Automatically converting common joins to optimized map joins based on table sizes and joining conditional.
5) Compression and archiving of hash tables to distributed cache to reduce bandwidth overhead.
6) Performance evaluations showing improvements from the optimized techniques.
Recommended
Llnl talk
These slides are from a recent talk I gave at Lawrence Livermore Labs.
The talk gives an architectural outline of the MapR system and then discusses how this architecture facilitates large scale machine learning algorithms.
Introduction to Yarn
YARN was introduced as part of Hadoop 2.0 to address limitations in the original MapReduce (MR1) architecture like scalability bottlenecks and underutilization of resources. YARN introduces a global ResourceManager and per-node NodeManagers to allocate cluster resources to distributed applications. It allows various distributed processing frameworks beyond MapReduce to share common cluster resources. Applications request containers for ApplicationMasters that then negotiate resources from YARN to run application components in containers across nodes. Existing MapReduce jobs can also run unchanged on YARN.
Introduction to map reduce
MapReduce provides a programming model for processing large datasets in a distributed, parallel manner. It involves two main steps - the map step where the input data is converted into intermediate key-value pairs, and the reduce step where the intermediate outputs are aggregated based on keys to produce the final results. Hadoop is an open-source software framework that allows distributed processing of large datasets across clusters of computers using MapReduce.
Tez: Accelerating Data Pipelines - fifthel
This document provides an overview of Tez, an Apache project that provides a framework for executing data processing jobs on Hadoop clusters. Tez allows expressing data processing jobs as directed acyclic graphs (DAGs) of tasks and executes these tasks in a optimized manner. It addresses limitations of MapReduce by providing a more flexible execution engine that can optimize performance and resource utilization.
TechEvent Operating MapR Hadoop Cluster for a year
The document discusses operating a MapR Hadoop cluster for a year to handle large volumes of automotive test data. Key aspects included setting up a 24-node cluster running Red Hat Linux with MapR-FS and YARN, separating control and worker nodes, and configuring services for high availability. Administration required deep knowledge of Hadoop components and customization to evolving requirements. Lessons included planning security from the start, maintaining performance and stability, and close collaboration with development teams.
Map Reduce
This document provides an overview of MapReduce, including:
- MapReduce is a programming model for processing large datasets in parallel across clusters of computers.
- It works by breaking the processing into map and reduce functions that can be run on many machines.
- Examples are given like word counting, distributed grep, and analyzing web server logs.
EMR AWS Demo
The document outlines how to run an Elastic MapReduce job on AWS. It discusses uploading input data and a MapReduce jar file to S3 storage. It then shows how to create an EMR Hadoop cluster with a specified number of workers and hardware configuration. Finally, it demonstrates submitting a MapReduce job to the cluster, monitoring its status, and terminating the cluster once complete.
Join optimization in hive
This document discusses optimization techniques for map join in Hive. It describes:
1) Previous approaches to common join and map join in Hive and their limitations.
2) Optimized map join techniques like uploading small tables to distributed cache and performing local joins to avoid shuffle.
3) Using JDBM for hash tables caused performance issues so alternative approaches were evaluated.
4) Automatically converting common joins to optimized map joins based on table sizes and joining conditional.
5) Compression and archiving of hash tables to distributed cache to reduce bandwidth overhead.
6) Performance evaluations showing improvements from the optimized techniques.
TEZ-8 UI Walkthrough
This document discusses the Tez UI and debugging tools. It provides an overview of the Tez UI, including how to view DAGs, track progress, view task logs and counters, and diagnose failed tasks. It also describes helper tools like swimlanes and a Tez log parser that can be used for post-hoc analysis of Tez applications.
Scaling graphite to handle a zerg rush
This document discusses scaling issues with Graphite and solutions implemented at Similarweb to handle high volumes of metrics. Key points:
1) Graphite struggled with high IOPS and a single-threaded carbon-cache. Replacing carbon-cache with the multi-threaded go-carbon and using SSDs helped address IOPS bottlenecks.
2) carbon-relay was replaced with the faster C implementation carbon-c-relay to load balance metrics among go-carbon instances.
3) statsd was replaced with the C implementation statsite for better performance and capabilities like quantiles.
4) The final setup consisted of statsite sending to multiple carbon-c-relay and go-carbon instances, handling
MapReduce basic
MapReduce is a programming model for processing large datasets in parallel. It works by breaking the dataset into independent chunks which are processed by the map function, and then grouping the output of the maps into partitions to be processed by the reduce function. Hadoop uses MapReduce to provide fault tolerance by restarting failed tasks and monitoring the JobTracker and TaskTrackers. MapReduce programs can be written in languages other than Java using Hadoop Streaming.
From Oracle to the Web - Automating Spatial Data Updates
The document discusses how EPA automated the monthly refresh of spatial data in its Facility Registry Service (FRS) map service. It used FME to extract data from Oracle Spatial into a file geodatabase, then updated the underlying data for an ArcGIS Server map service. A Python script in FME packaged and deployed the new geodatabase, refreshed the map service, and notified other workflows. This automated a previously manual process and improved performance by moving from Oracle to a file-based system. EPA plans to trigger the FME workflow directly from Oracle and integrate it into other refresh flows.
Map Reduce
The document provides an overview of MapReduce, including:
1) MapReduce is a programming model and implementation that allows for large-scale data processing across clusters of computers. It handles parallelization, distribution, and reliability.
2) The programming model involves mapping input data to intermediate key-value pairs and then reducing by key to output results.
3) Example uses of MapReduce include word counting and distributed searching of text.
Qubole Overview at the Fifth Elephant Conference
The document discusses some of the challenges of managing Hadoop clusters in the cloud, including setting up infrastructure components like the Hive metastore and determining optimal cluster sizing. It then presents some solutions offered by Qubole's data platform, like auto-scaling clusters and running periodic jobs. The document also covers techniques for improving query performance, such as using HDFS as a cache layer and storing data in columnar format for faster access compared to JSON or CSV files stored in S3.
Introduction to Yarn
YARN allows applications to share resources in a cluster by separating resource management from job scheduling and monitoring. It introduces the ResourceManager for resource management and per-application ApplicationMasters for negotiating resources and monitoring applications. The NodeManager runs on slave nodes and manages containers which are allocated by the ResourceManager and used by applications to execute tasks. YARN supports high availability, rolling upgrades, multi-tenancy and fault tolerance.
Map reduce presentation
MapReduce is a programming model for processing large datasets in a distributed system. It involves a map step that performs filtering and sorting, and a reduce step that performs summary operations. Hadoop is an open-source framework that supports MapReduce. It orchestrates tasks across distributed servers, manages communications and fault tolerance. Main steps include mapping of input data, shuffling of data between nodes, and reducing of shuffled data.
Hive
The document discusses Hive, a data warehouse infrastructure built on top of Hadoop. It provides the following key points:
- Hive provides a mechanism for analysts to query large datasets using a SQL-like language called HiveQL instead of using Java programs. This allows analysts who do not know Java to work with Hadoop.
- Hive includes components like a shell, compiler, execution engine, and metastore to allow SQL-like queries over data stored in Hadoop files. It uses a metastore to manage metadata about tables and their physical storage.
- Hive represents data as tables partitioned into buckets or ranges. It uses a data model similar to relational databases to organize data and query it using SQL-
Analysing of big data using map reduce
This document discusses efficient analysis of big data using the MapReduce framework. It introduces the challenges of analyzing large and complex datasets, and describes how MapReduce addresses these challenges through its map and reduce functions. MapReduce allows distributed processing of big data across clusters of computers using a simple programming model.
Introduction To Map Reduce
This document provides an overview of MapReduce, a programming model developed by Google for processing and generating large datasets in a distributed computing environment. It describes how MapReduce abstracts away the complexities of parallelization, fault tolerance, and load balancing to allow developers to focus on the problem logic. Examples are given showing how MapReduce can be used for tasks like word counting in documents and joining datasets. Implementation details and usage statistics from Google demonstrate how MapReduce has scaled to process exabytes of data across thousands of machines.
Tuning up with Apache Tez
This document discusses tuning a pipeline with Apache Tez. It provides an overview of Tez, including that it aims to enhance scenarios not well served by MapReduce by supporting Hive and Pig. It details lessons learned from using Tez, such as some Pig tasks not compiling and occasional freezing, lack of DistributedCache support for S3, and poor Amazon support. The document concludes by stating that Tez is good for early adopters of Pig and Hive and those with bounded workloads.
Hadoop Summit 2014 - recap
Hadoop Summit 2014 covered developments in YARN, Tez, Spark, BlinkDB, Summingbird, Storm, and machine learning. YARN now supports additional workloads beyond MapReduce. Tez is a new execution engine that provides performance gains of 2-3x for Hive and Pig queries. Spark is an emerging platform for interactive queries and streaming data. BlinkDB enables interactive queries on very large datasets by using sampling. Summingbird and Storm allow stream processing on Hadoop. Machine learning libraries are focusing on sparse data representation and deep learning techniques.
The Meta of Hadoop - COMAD 2012
What do you talk about to a hall full of database gurus? Instead of science - my talk focused on the art. What made Hadoop successful? What can we learn from it? What principles work well in building software for large scale services? What are some interesting unsolved problems in a world overrun by open-source (and VC investments :-))
Hadoop eco system-first class
This document provides an introduction to distributed programming and Apache Hadoop. It discusses sequential, asynchronous, concurrent, and distributed programming. It then describes how Hadoop works as an open source framework for distributed applications, using a cluster of nodes to scale linearly and process large amounts of data reliably in a simple way. Key concepts covered include MapReduce programming, Hadoop installation and usage, and working with files in the Hadoop Distributed File System.
The Future of Sharding
The document discusses different approaches to scaling databases, including sharding and partitioning. It provides examples of sharding data across multiple tables based on a hashed key. It also demonstrates range partitioning a single table into multiple sub-tables based on time intervals, using PostgreSQL's native partitioning. The document inserts sample data and runs queries to showcase these different scaling techniques.
Hadoop - Introduction to map reduce programming - Reunião 12/04/2014
Aaron Myers introduces MapReduce and Hadoop. MapReduce is a distributed programming paradigm that allows processing of large datasets across clusters. It works by splitting data, distributing it across nodes, processing it in parallel using map and reduce functions, and collecting the results. Hadoop is an open source software framework for distributed storage and processing of big data using MapReduce. It includes HDFS for storage and Hadoop MapReduce for distributed computing. Developers write MapReduce jobs in Java by implementing map and reduce functions.
Facebook Retrospective - Big data-world-europe-2012
This document provides a retrospective on data infrastructure at Facebook from 2007-2011 written by the ex-Facebook data infrastructure lead. It summarizes the goals of building a universal data logging and computing platform, the state and growth of the Hadoop cluster from 10TB to 50PB, and key components like Hive, Scribe, and reporting tools that helped various teams access and analyze data. It also discusses challenges around query performance, unnecessary duplication, and a lack of APIs that were missed opportunities. The overall message is that building useful services around the software was more important than the software itself.
Scrap Your MapReduce - Apache Spark
Remember the last time you tried to write a MapReduce job (obviously something non trivial than a word count)? It sure did the work, but has lot of pain points from getting an idea to implement it in terms of map reduce. Did you wonder how life will be much simple if you had to code like doing collection operations and hence being transparent* to its distributed nature? Did you want/hope for more performant/low latency jobs? Well, seems like you are in luck.
In this talk, we will be covering a different way to do MapReduce kind of operations without being just limited to map and reduce, yes, we will be talking about Apache Spark. We will compare and contrast Spark programming model with Map Reduce. We will see where it shines, and why to use it, how to use it. We’ll be covering aspects like testability, maintainability, conciseness of the code, and some features like iterative processing, optional in-memory caching and others. We will see how Spark, being just a cluster computing engine, abstracts the underlying distributed storage, and cluster management aspects, giving us a uniform interface to consume/process/query the data. We will explore the basic abstraction of RDD which gives us so many awesome features making Apache Spark a very good choice for your big data applications. We will see this through some non trivial code examples.
Session at the IndicThreads.com Confence held in Pune, India on 27-28 Feb 2015
http://www.indicthreads.com
http://pune15.indicthreads.com
Cloud Optimized Big Data
What makes a big-data platform 'cloud-optimized'. Here's our (Qubole's) shot at it. @Cloud-Asia 2014.
Availability and Integrity in hadoop (Strata EU Edition)
The document discusses data availability and integrity in Apache Hadoop. It explains that Hadoop uses replication and checksums to ensure data safety. While the NameNode is a single point of failure, Hadoop 1 provides cold failover capability using Linux HA clustering. Hadoop 2 will introduce live NameNode failover using ZooKeeper. The goal is full stack high availability to make the entire Hadoop system resilient to planned and unplanned outages.
Inside hadoop-dev
An overview of the development of the Apache Hadoop software stack, including some of the barriers to participation -and how and why to overcome them. It closes with some open discussion points/ideas of how the existing process can be improved.
More Related Content
What's hot
TEZ-8 UI Walkthrough
This document discusses the Tez UI and debugging tools. It provides an overview of the Tez UI, including how to view DAGs, track progress, view task logs and counters, and diagnose failed tasks. It also describes helper tools like swimlanes and a Tez log parser that can be used for post-hoc analysis of Tez applications.
Scaling graphite to handle a zerg rush
This document discusses scaling issues with Graphite and solutions implemented at Similarweb to handle high volumes of metrics. Key points:
1) Graphite struggled with high IOPS and a single-threaded carbon-cache. Replacing carbon-cache with the multi-threaded go-carbon and using SSDs helped address IOPS bottlenecks.
2) carbon-relay was replaced with the faster C implementation carbon-c-relay to load balance metrics among go-carbon instances.
3) statsd was replaced with the C implementation statsite for better performance and capabilities like quantiles.
4) The final setup consisted of statsite sending to multiple carbon-c-relay and go-carbon instances, handling
MapReduce basic
MapReduce is a programming model for processing large datasets in parallel. It works by breaking the dataset into independent chunks which are processed by the map function, and then grouping the output of the maps into partitions to be processed by the reduce function. Hadoop uses MapReduce to provide fault tolerance by restarting failed tasks and monitoring the JobTracker and TaskTrackers. MapReduce programs can be written in languages other than Java using Hadoop Streaming.
From Oracle to the Web - Automating Spatial Data Updates
The document discusses how EPA automated the monthly refresh of spatial data in its Facility Registry Service (FRS) map service. It used FME to extract data from Oracle Spatial into a file geodatabase, then updated the underlying data for an ArcGIS Server map service. A Python script in FME packaged and deployed the new geodatabase, refreshed the map service, and notified other workflows. This automated a previously manual process and improved performance by moving from Oracle to a file-based system. EPA plans to trigger the FME workflow directly from Oracle and integrate it into other refresh flows.
Map Reduce
The document provides an overview of MapReduce, including:
1) MapReduce is a programming model and implementation that allows for large-scale data processing across clusters of computers. It handles parallelization, distribution, and reliability.
2) The programming model involves mapping input data to intermediate key-value pairs and then reducing by key to output results.
3) Example uses of MapReduce include word counting and distributed searching of text.
Qubole Overview at the Fifth Elephant Conference
The document discusses some of the challenges of managing Hadoop clusters in the cloud, including setting up infrastructure components like the Hive metastore and determining optimal cluster sizing. It then presents some solutions offered by Qubole's data platform, like auto-scaling clusters and running periodic jobs. The document also covers techniques for improving query performance, such as using HDFS as a cache layer and storing data in columnar format for faster access compared to JSON or CSV files stored in S3.
Introduction to Yarn
YARN allows applications to share resources in a cluster by separating resource management from job scheduling and monitoring. It introduces the ResourceManager for resource management and per-application ApplicationMasters for negotiating resources and monitoring applications. The NodeManager runs on slave nodes and manages containers which are allocated by the ResourceManager and used by applications to execute tasks. YARN supports high availability, rolling upgrades, multi-tenancy and fault tolerance.
Map reduce presentation
MapReduce is a programming model for processing large datasets in a distributed system. It involves a map step that performs filtering and sorting, and a reduce step that performs summary operations. Hadoop is an open-source framework that supports MapReduce. It orchestrates tasks across distributed servers, manages communications and fault tolerance. Main steps include mapping of input data, shuffling of data between nodes, and reducing of shuffled data.
Hive
The document discusses Hive, a data warehouse infrastructure built on top of Hadoop. It provides the following key points:
- Hive provides a mechanism for analysts to query large datasets using a SQL-like language called HiveQL instead of using Java programs. This allows analysts who do not know Java to work with Hadoop.
- Hive includes components like a shell, compiler, execution engine, and metastore to allow SQL-like queries over data stored in Hadoop files. It uses a metastore to manage metadata about tables and their physical storage.
- Hive represents data as tables partitioned into buckets or ranges. It uses a data model similar to relational databases to organize data and query it using SQL-
Analysing of big data using map reduce
This document discusses efficient analysis of big data using the MapReduce framework. It introduces the challenges of analyzing large and complex datasets, and describes how MapReduce addresses these challenges through its map and reduce functions. MapReduce allows distributed processing of big data across clusters of computers using a simple programming model.
Introduction To Map Reduce
This document provides an overview of MapReduce, a programming model developed by Google for processing and generating large datasets in a distributed computing environment. It describes how MapReduce abstracts away the complexities of parallelization, fault tolerance, and load balancing to allow developers to focus on the problem logic. Examples are given showing how MapReduce can be used for tasks like word counting in documents and joining datasets. Implementation details and usage statistics from Google demonstrate how MapReduce has scaled to process exabytes of data across thousands of machines.
Tuning up with Apache Tez
This document discusses tuning a pipeline with Apache Tez. It provides an overview of Tez, including that it aims to enhance scenarios not well served by MapReduce by supporting Hive and Pig. It details lessons learned from using Tez, such as some Pig tasks not compiling and occasional freezing, lack of DistributedCache support for S3, and poor Amazon support. The document concludes by stating that Tez is good for early adopters of Pig and Hive and those with bounded workloads.
Hadoop Summit 2014 - recap
Hadoop Summit 2014 covered developments in YARN, Tez, Spark, BlinkDB, Summingbird, Storm, and machine learning. YARN now supports additional workloads beyond MapReduce. Tez is a new execution engine that provides performance gains of 2-3x for Hive and Pig queries. Spark is an emerging platform for interactive queries and streaming data. BlinkDB enables interactive queries on very large datasets by using sampling. Summingbird and Storm allow stream processing on Hadoop. Machine learning libraries are focusing on sparse data representation and deep learning techniques.
The Meta of Hadoop - COMAD 2012
What do you talk about to a hall full of database gurus? Instead of science - my talk focused on the art. What made Hadoop successful? What can we learn from it? What principles work well in building software for large scale services? What are some interesting unsolved problems in a world overrun by open-source (and VC investments :-))
Hadoop eco system-first class
This document provides an introduction to distributed programming and Apache Hadoop. It discusses sequential, asynchronous, concurrent, and distributed programming. It then describes how Hadoop works as an open source framework for distributed applications, using a cluster of nodes to scale linearly and process large amounts of data reliably in a simple way. Key concepts covered include MapReduce programming, Hadoop installation and usage, and working with files in the Hadoop Distributed File System.
The Future of Sharding
The document discusses different approaches to scaling databases, including sharding and partitioning. It provides examples of sharding data across multiple tables based on a hashed key. It also demonstrates range partitioning a single table into multiple sub-tables based on time intervals, using PostgreSQL's native partitioning. The document inserts sample data and runs queries to showcase these different scaling techniques.
Hadoop - Introduction to map reduce programming - Reunião 12/04/2014
Aaron Myers introduces MapReduce and Hadoop. MapReduce is a distributed programming paradigm that allows processing of large datasets across clusters. It works by splitting data, distributing it across nodes, processing it in parallel using map and reduce functions, and collecting the results. Hadoop is an open source software framework for distributed storage and processing of big data using MapReduce. It includes HDFS for storage and Hadoop MapReduce for distributed computing. Developers write MapReduce jobs in Java by implementing map and reduce functions.
Facebook Retrospective - Big data-world-europe-2012
This document provides a retrospective on data infrastructure at Facebook from 2007-2011 written by the ex-Facebook data infrastructure lead. It summarizes the goals of building a universal data logging and computing platform, the state and growth of the Hadoop cluster from 10TB to 50PB, and key components like Hive, Scribe, and reporting tools that helped various teams access and analyze data. It also discusses challenges around query performance, unnecessary duplication, and a lack of APIs that were missed opportunities. The overall message is that building useful services around the software was more important than the software itself.
Scrap Your MapReduce - Apache Spark
Remember the last time you tried to write a MapReduce job (obviously something non trivial than a word count)? It sure did the work, but has lot of pain points from getting an idea to implement it in terms of map reduce. Did you wonder how life will be much simple if you had to code like doing collection operations and hence being transparent* to its distributed nature? Did you want/hope for more performant/low latency jobs? Well, seems like you are in luck.
In this talk, we will be covering a different way to do MapReduce kind of operations without being just limited to map and reduce, yes, we will be talking about Apache Spark. We will compare and contrast Spark programming model with Map Reduce. We will see where it shines, and why to use it, how to use it. We’ll be covering aspects like testability, maintainability, conciseness of the code, and some features like iterative processing, optional in-memory caching and others. We will see how Spark, being just a cluster computing engine, abstracts the underlying distributed storage, and cluster management aspects, giving us a uniform interface to consume/process/query the data. We will explore the basic abstraction of RDD which gives us so many awesome features making Apache Spark a very good choice for your big data applications. We will see this through some non trivial code examples.
Session at the IndicThreads.com Confence held in Pune, India on 27-28 Feb 2015
http://www.indicthreads.com
http://pune15.indicthreads.com
Cloud Optimized Big Data
What makes a big-data platform 'cloud-optimized'. Here's our (Qubole's) shot at it. @Cloud-Asia 2014.
What's hot (20)
From Oracle to the Web - Automating Spatial Data Updates
From Oracle to the Web - Automating Spatial Data Updates
Hadoop - Introduction to map reduce programming - Reunião 12/04/2014
Hadoop - Introduction to map reduce programming - Reunião 12/04/2014
Facebook Retrospective - Big data-world-europe-2012
Facebook Retrospective - Big data-world-europe-2012
Viewers also liked
Availability and Integrity in hadoop (Strata EU Edition)
The document discusses data availability and integrity in Apache Hadoop. It explains that Hadoop uses replication and checksums to ensure data safety. While the NameNode is a single point of failure, Hadoop 1 provides cold failover capability using Linux HA clustering. Hadoop 2 will introduce live NameNode failover using ZooKeeper. The goal is full stack high availability to make the entire Hadoop system resilient to planned and unplanned outages.
Inside hadoop-dev
An overview of the development of the Apache Hadoop software stack, including some of the barriers to participation -and how and why to overcome them. It closes with some open discussion points/ideas of how the existing process can be improved.
Datamining Location
The document discusses how cities can analyze aggregate anonymized data collected from people's movements and wireless devices to better understand how cities function. This data could help identify patterns in traffic, social groups, and population flows. It raises concerns that this valuable data is being given away to large tech companies without compensation or consideration of who owns the aggregate and individual data traces.
Hadoop And Universities
The document discusses using Hadoop in universities for computer science education and research. It suggests teaching students MapReduce programming and hosting Hadoop on university computing grids for large-scale data analysis across various disciplines. University researchers could also develop new algorithms for Hadoop and use Hadoop for data-intensive scientific computing and research into Hadoop's scheduling and resource management.
Digital Pebble Behemoth
This document introduces Behemoth, an open source framework for large scale document processing using Apache Hadoop. Behemoth allows users to deploy natural language processing pipelines built with UIMA or GATE on Hadoop clusters. It provides common adapters for document inputs and encourages code reuse. The typical workflow involves loading documents into HDFS, converting them to a common format, running NLP pipelines in parallel using MapReduce, and allowing for post-processing of annotations across documents at scale.
Community Engagement
Sander W G van der Waal from OSS Watch discussing community engagement between UK University/Higher Education users and open source projects
Lessons from building large clusters
Small production clusters can be built and run by a few skilled people and are a natural extension of conventional IT, while large clusters require more pioneers and a large development staff to deal with the challenges of scale. For those wanting to start small but scale large, it is important to select the right server configuration and network setup, build the cluster consistently in a factory setting if possible, and treat core nodes as more important than data nodes when monitoring cluster health and operations. Not all aspects of IT scale as well as Hadoop, so the operational model must be designed to support large clusters.
2014 01-02-patching-workflow
workflow for patchng hadoop. uses the font P22 Typewriter which is why it renders oddly for everyone who doesn't have that installed
HDFS Issues
HDFS provides a filesystem that can scale to tens of petabytes, with high IO bandwidth and data replication across machines. Locality is important for performance in Hadoop. There are ongoing issues with data loss, handling full disks, underreplication of data, and limits to scaling the namenode. Failover handling and security also need improvement before HDFS is ready for broad production use.
High availability hadoop november 2010
This document discusses using the HP IBRIX file system as an alternative to HDFS in Hadoop to provide high availability. IBRIX is a segmented, fault tolerant file system that runs on top of RAIDed storage. It avoids single points of failure by distributing metadata across multiple segment servers rather than using a single NameNode as in HDFS. The document outlines how Hadoop could integrate with IBRIX, provides details on IBRIX architecture and configuration, and presents early performance results showing comparable performance to HDFS.
Hadoop and Kerberos: the Madness Beyond the Gate
Apachecon Big Data EU 2105 talk on Hadoop and Kerberos; needs to viewed with slide animation to make sense
High Availability Hadoop
This document discusses making Hadoop highly available by using the HP IBRIX file system instead of HDFS. It provides an overview of Hadoop, describing how it operates with HDFS and its failure response mechanisms. It then introduces IBRIX as an alternative file system that is fault tolerant with no single point of failure through its segmented design. The document suggests IBRIX could improve Hadoop availability by allowing it to take advantage of IBRIX's location awareness and failure recovery capabilities.
Slider: Applications on YARN
This document discusses Apache Slider, which allows applications to be deployed and managed on Apache Hadoop YARN. Slider uses an Application Master, agents, and scripts to deploy applications defined in an XML package. The Application Master keeps applications in a desired state across YARN containers and handles lifecycle commands like start, stop, and scaling. Slider integrates with Apache Ambari for graphical management and configuration of applications on YARN.
My other computer is a datacentre
Talk on the engineering aspects of building a datacentre, then a look at running MapReduce over the machines
HDFS
HDFS is a distributed file system used for large data sets in Hadoop. It scales well and can support thousands of nodes storing petabytes of data. Several large companies use HDFS in production including Yahoo, Facebook, and Last.fm. HDFS works well for batch jobs but may have issues for real-time logging or serving many small files to a website due to performance and high availability concerns. Improvements are being made to address issues with appends, availability, and reducing disk usage. Alternative solutions exist for low latency use cases.
HDFS: Hadoop Distributed Filesystem
Presentation on 2013-06-27, Workshop on the future of Big Data management, discussing hadoop for a science audience that are either HPC/grid users or people suddenly discovering that their data is accruing towards PB.
The other talks were on GPFS, LustreFS and Ceph, so rather than just do beauty-contest slides, I decided to raise the question of "what is a filesystem?", whether the constraints imposed by the Unix metaphor and API are becoming limits on scale and parallelism (both technically and, for GPFS and Lustre Enterprise in cost).
Then: HDFS as the foundation for the Hadoop stack.
All the other FS talks did emphasise their Hadoop integration, with the Intel talk doing the most to assert performance improvements of LustreFS over HDFSv1 in dfsIO and Terasort (no gridmix?), which showed something important: Hadoop is the application that add DFS developers have to have a story for
YARN Services
This document discusses YARN services in Hadoop, which allow long-lived applications to run within a Hadoop cluster. YARN (Yet Another Resource Negotiator) provides an operating system-like platform for data processing by allowing various applications to share cluster resources. The document outlines features for long-lived services in YARN, including log aggregation, Kerberos token renewal, and service registration/discovery. It also discusses how Hadoop 2.6 and later versions implement these features to enable long-running applications that can withstand failures.
Hadoop, Hive, Spark and Object Stores
Cloud deployments of Apache Hadoop are becoming more commonplace. Yet Hadoop and it's applications don't integrate that well —something which starts right down at the file IO operations. This talk looks at how to make use of cloud object stores in Hadoop applications, including Hive and Spark. It will go from the foundational "what's an object store?" to the practical "what should I avoid" and the timely "what's new in Hadoop?" — the latter covering the improved S3 support in Hadoop 2.8+. I'll explore the details of benchmarking and improving object store IO in Hive and Spark, showing what developers can do in order to gain performance improvements in their own code —and equally, what they must avoid. Finally, I'll look at ongoing work, especially "S3Guard" and what its fast and consistent file metadata operations promise.
Hadoop as data refinery
Explores the notion of "Hadoop as a Data Refinery" within an organisation, be it one with an existing Business Intelligence system or none - looks at 'agile data' as a a benefit of using Hadoop as the store for historical, unstructured and very-large-scale datasets.
The final slides look at the challenge of an organisation becoming "data driven"
Viewers also liked (19)
Availability and Integrity in hadoop (Strata EU Edition)
Availability and Integrity in hadoop (Strata EU Edition)
Similar to Hadoop: Beyond MapReduce
Hadoop past, present and future
Ever wonder what Hadoop might look like in 12 months or 24 months or longer? Apache Hadoop MapReduce has undergone a complete re-haul to emerge as Apache Hadoop YARN, a generic compute fabric to support MapReduce and other application paradigms. As a result, Hadoop looks very different from itself 12 months ago. This talk will take you through some ideas for YARN itself and the many myriad ways it is really moving the needle for MapReduce, Pig, Hive, Cascading and other data-processing tools in the Hadoop ecosystem.
Yarnthug2014
YARN is a resource management framework for Hadoop that allows multiple data processing engines such as MapReduce, Spark, and Storm to run on the same cluster. It introduces a global ResourceManager and per-node NodeManagers to allocate and manage resources across applications. YARN supports multi-tenant clusters with queues that provide resource guarantees and isolation between users and workloads. A demo showed preemption and multi-tenant queues handling different workloads hitting the cluster.
Hoya for Code Review
An iteration of the Hoya slides put up as part of a review of the code with others writing YARN services; looks at what Hoya offers -what we need from Apps to be able to deploy them this way, and what we need from YARN
Overview of slider project
The document discusses Hortonworks' Slider project, which aims to simplify deploying and managing distributed applications on YARN. Slider provides a packaging format for applications, launches application components as YARN containers via an Application Master, and handles service registration and configuration management. It addresses limitations of earlier frameworks by supporting dynamic configurations, embedded usage, and integration with service discovery in Zookeeper.
Tez big datacamp-la-bikas_saha
This document discusses Apache Tez, a framework for accelerating Hadoop query processing. Tez is designed to express query computations as dataflow graphs and execute them efficiently on YARN. It addresses limitations of MapReduce by allowing for custom dataflows and optimizations. Tez provides APIs for defining DAGs of tasks and customizing inputs/outputs/processors. This allows applications to focus on business logic while Tez handles distributed execution, fault tolerance, and resource management for Hadoop clusters.
Interactive query in hadoop
This document discusses interactive querying in Hadoop. It describes how Hive facilitates SQL querying over data stored in HDFS. Hive performance is improved through optimizations like using Tez as the execution engine instead of MapReduce, vectorized queries, and ORC file format. Tez is a dataflow framework that allows expressing queries as directed acyclic graphs (DAGs) of vertices and edges, avoiding the multi-step MapReduce approach and improving latency. The document provides examples of expressing Hive queries in Tez and demonstrates its capabilities.
Apache Hadoop YARN - The Future of Data Processing with Hadoop
A dive into new data processing capabilities in Hadoop 2.0 with Apache Hadoop YARN, Apache Tez and Apache Hive
2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0
The document discusses Hadoop 2.2.0 and new features in YARN and MapReduce. Key points include: YARN introduces a new application framework and resource management system that replaces the jobtracker, allowing multiple data processing engines besides MapReduce; MapReduce is now a library that runs on YARN; Tez is introduced as a new data processing framework to improve performance beyond MapReduce.
Apache Tez -- A modern processing engine
The document discusses Apache Tez, a framework for accelerating Hadoop query processing. Some key points:
- Tez is a dataflow framework that expresses computations as directed acyclic graphs (DAGs) of tasks. This allows optimizations like container reuse and locality-aware scheduling.
- It is built on YARN and provides a customizable execution engine as well as runtime and DAG APIs for applications to define computations.
- Compared to MapReduce, Tez can provide better performance, predictability, and resource utilization through its DAG execution model and optimizations like reducing intermediate data writes.
- It has been used to improve performance for workloads like Hive, Pig, and large TPC-DS queries
Hortonworks Yarn Code Walk Through January 2014
This slide deck accompanies the Webinar recording YARN Code Walk through on Jan. 22, 2014, on Hortonworks.com/webinars under Past Webinars, or
https://hortonworks.webex.com/hortonworks/lsr.php?AT=pb&SP=EC&rID=129468197&rKey=b645044305775657
YARN Ready: Integrating to YARN with Tez
YARN Ready webinar series helps developers integrate their applications to YARN. Tez is one vehicle to do that. We take a deep dive including code review to help you get started.
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez is a framework for accelerating Hadoop query processing. It is based on expressing a computation as a dataflow graph and executing it in a highly customizable way. Tez is built on top of YARN and provides benefits like better performance, predictability, and utilization of cluster resources compared to traditional MapReduce. It allows applications to focus on business logic rather than Hadoop internals.
Apache Tez - Accelerating Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing. Talk at Hadoop Summit, San Jose. 2014 By Bikas Saha and Hitesh Shah.
Apache Tez is a modern data processing engine designed for YARN on Hadoop 2. Tez aims to provide high performance and efficiency out of the box, across the spectrum of low latency queries and heavy-weight batch processing.
Apache Tez: Accelerating Hadoop Query Processing
This document discusses Apache Tez, a framework for accelerating Hadoop query processing. Some key points:
- Tez is a dataflow framework that expresses computations as directed acyclic graphs (DAGs) of tasks, allowing for optimizations like container reuse and locality-aware scheduling.
- It is built on YARN and provides a customizable execution engine as well as APIs for applications like Hive and Pig.
- By expressing jobs as DAGs, Tez can reduce overheads, queueing delays, and better utilize cluster resources compared to the traditional MapReduce framework.
- The document provides examples of how Tez can improve performance for operations like joins, aggregations, and handling of multiple outputs
Apache Tez : Accelerating Hadoop Query Processing
Apache Tez is the new data processing framework in the Hadoop ecosystem. It runs on top of YARN - the new compute platform for Hadoop 2. Learn how Tez is built from the ground up to tackle a broad spectrum of data processing scenarios in Hadoop/BigData - ranging from interactive query processing to complex batch processing. With a high degree of automation built-in, and support for extensive customization, Tez aims to work out of the box for good performance and efficiency. Apache Hive and Pig are already adopting Tez as their platform of choice for query execution.
MHUG - YARN
YARN (Yet Another Resource Negotiator) is a distributed operating system for large scale data processing. It improves on MapReduce by allowing multiple data processing engines and frameworks to share common distributed compute resources and data storage on large Hadoop clusters. YARN introduces a resource management layer separate from job scheduling and processing logic. This allows Hadoop to support diverse workloads including batch processing, interactive queries, real-time streams and more. YARN also enables multi-tenant clusters to share resources among multiple users and applications in a secure manner through queues and containers.
Apache Tez: Accelerating Hadoop Query Processing
Tez is the next generation Hadoop Query Processing framework written on top of YARN. Computation topologies in higher level languages like Pig/Hive can be naturally expressed in the new graph dataflow model exposed by Tez. Multi-stage queries can be expressed as a single Tez job resulting in lower latency for short queries and improved throughput for large scale queries. MapReduce has been the workhorse for Hadoop but its monolithic structure had made innovation slower. YARN separates resource management from application logic and thus enables the creation of Tez, a more flexible and generic new framework for data processing for the benefit of the entire Hadoop query ecosystem.
Get Started Building YARN Applications
Hortonworks Get Started Building YARN Applications Dec. 2013. We cover YARN basics, benefits, getting started and roadmap. Actian shares their experience and recommendations on building their real-world YARN application.
Hackathon bonn
Hortonworks' mission is to enable modern data architectures by delivering an enterprise-ready Apache Hadoop platform. They contribute the majority of code to Apache Hadoop and its related projects. Hortonworks develops the Hortonworks Data Platform (HDP), which provides core Hadoop services along with operational and data services to make Hadoop an enterprise data platform. Hortonworks aims to power data architectures by enabling Hadoop as a multi-purpose platform for batch, interactive, streaming and other workloads through projects like YARN, Tez, and improvements to Hive.
Bikas saha：the next generation of hadoop– hadoop 2 and yarn
The document discusses Apache YARN, which is the next-generation resource management platform for Apache Hadoop. YARN was designed to address limitations of the original Hadoop 1 architecture by supporting multiple data processing models (e.g. batch, interactive, streaming) and improving cluster utilization. YARN achieves this by separating resource management from application execution, allowing various data processing engines like MapReduce, HBase and Storm to run natively on Hadoop frames. This provides a flexible, efficient and shared platform for distributed applications.
Similar to Hadoop: Beyond MapReduce (20)
Apache Hadoop YARN - The Future of Data Processing with Hadoop
Apache Hadoop YARN - The Future of Data Processing with Hadoop
2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0
2013 Nov 20 Toronto Hadoop User Group (THUG) - Hadoop 2.2.0
Apache Tez - A New Chapter in Hadoop Data Processing
Apache Tez - A New Chapter in Hadoop Data Processing
Bikas saha：the next generation of hadoop– hadoop 2 and yarn
Bikas saha：the next generation of hadoop– hadoop 2 and yarn
More from Steve Loughran
Hadoop Vectored IO
Description of Hadoop vectored IO API with benchmark results showing 10-20% speedup in TPC-DS query times
The age of rename() is over
The document discusses how storage models need to evolve as the underlying technologies change. Object stores like S3 provide scale and high availability but lack semantics and performance of file systems. Non-volatile memory also challenges current models. The POSIX file system metaphor is ill-suited for object stores and NVM. SQL provides an alternative that abstracts away the underlying complexities, leaving just object-relational mapping and transaction isolation to address. The document examines renaming operations, asynchronous I/O, and persistent in-memory data structures as examples of areas where new models may be needed.
What does Rename Do: (detailed version)
August 2018 version of my "What does rename() do", includes the full details on what the Hadoop MapReduce and Spark commit protocols are, so the audience will really understand why rename really, really matters
Put is the new rename: San Jose Summit Edition
This is the June 2018 variant of the "Put is the new Rename Talk", looking at Hadoop stack integration with object stores, including S3, Azure storage and GCS.
@Dissidentbot: dissent will be automated!
This document outlines the development history of the Dissident bot from its creation in January 2017 to June 2018. It discusses improvements made over time including adding conversation mode, a TODO item to develop a Chomsky-Type-1 Grammar AI, and fixing a bug where conversation mode would spam the bot's username. It also provides details on the bot's configuration settings and methods used to detect spam, bots, and politicans spreading misinformation.
PUT is the new rename()
A review of the state of cloud store integration with the Hadoop stack in 2018; including S3Guard, the new S3A committers and S3 Select.
Presented at Dataworks Summit Berlin 2018, where the demos were live.
Extreme Programming Deployed
This document discusses the principles and practices of Extreme Programming (XP), an agile software development process. It describes XP as an intense, test-centric programming process focused on projects with high rates of change. Key practices include pair programming, test-driven development, planning with user stories and tasks, doing the simplest thing that could work, and refactoring code aggressively. Problems may include short-term "hill-climbing" solutions and risks of fundamental design errors. The document provides additional resources on XP and notes that the day's session will involve practicing XP techniques through pair programming.
Testing
2005 talk on testing: I know your code doesn't work, because I know my own is broken. Covers CruiseControl & Apache Gump
I hate mocking
Steve Loughran expresses dislike for mocking in tests because mock code reflects assumptions rather than reality. Any changes to the real code can break the tests, leading to false positives. Test failures are often "fixed" by editing the test or mock code, which could hide real problems. He proposes avoiding mock tests and instead adding functional tests against real infrastructure with fault injection for integration testing.
What does rename() do?
Berlin Buzzwords 2017 talk: A look at what our storage models, metaphors and APIs are, showing how we need to rethink the Posix APIs to work with object stores, while looking at different alternatives for local NVM.
This is the unabridged talk; the BBuzz talk was 20 minutes including demo and questions, so had ~half as many slides
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
A talk looking at the intricate details of working with an object store from Hadoop, Hive, Spark, etc, why the "filesystem" metaphor falls down, and what work myself and others have been up to to try and fix things
Apache Spark and Object Stores —for London Spark User Group
The March 2017 version of the "Apache Spark and Object Stores", includes coverage of the Staging Committer. If you'd been at the talk you'd have seen the projector fail just before the demo. It worked earlier! Honest!
Spark Summit East 2017: Apache spark and object stores
Spark Summit East 2017 talk on Apache Spark and object store integration, specifically AWS S3 and Azure WASB
Apache Spark and Object Stores
This document discusses using Apache Spark with object stores like Amazon S3 and Microsoft Azure Blob Storage. It covers challenges around classpath configuration, credentials, code examples, and performance commitments when using these storage systems. Key points include using Hadoop connectors like S3A and WASB, configuring credentials through properties or environment variables, and tuning Spark for object store performance and consistency.
Household INFOSEC in a Post-Sony Era
This document discusses household information security risks in the post-Sony era. It identifies key risks like data integrity, privacy, and availability issues. It provides examples of vulnerabilities across different devices and platforms like LG TVs, iPads, iPhones, and PS4s. It also discusses vulnerabilities in software like Firefox, Chrome, Internet Explorer, Flash, and SparkContext. It recommends approaches to address these risks like using containers for isolation, validating packages with PGP to ensure authentication, and enabling audit logs.
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
An update of the "Hadoop and Kerberos: the Madness Beyond the Gate" talk, covering recent work "the Fix Kerberos" JIRA and its first deliverable: KDiag
Datacentre stack
This document appears to be a list of terms related to computer science and data systems including HDFS, YARN, Kernighan, Cerf, Lamport, Codd, Knuth and SQL. It references people, technologies and concepts but provides no additional context or explanation.
Help! My Hadoop doesn't work!
This document provides guidance on reporting bugs in the Apache Hadoop project. It explains that the JIRA issue tracker is used to report bugs and feature requests, and outlines best practices for submitting high-quality issue reports. These include searching for existing issues and solutions first, providing detailed steps to replicate the problem, and attaching relevant logs and stack traces. The document discourages "help!" emails and stresses that the best way to get a bug fixed is often for the reporter to propose a patch with tests.
2013 11-19-hoya-status
This document discusses HBase on YARN (Hoya), which allows dynamic HBase clusters to be launched on a YARN cluster. Hoya is a Java tool that uses a JSON specification to deploy HBase clusters. The HBase master and region servers run as YARN containers while Zookeeper handles coordination. Hoya's Application Master interfaces with YARN to deploy, manage, and handle failures of the HBase cluster. This allows efficient sharing of resources and elastic scaling of HBase clusters based on workload.
HA Hadoop -ApacheCon talk
Hadoop provides high availability through replication of data across multiple nodes. Replication handles data integrity through checksums and automatic re-replication of corrupt blocks. Rack failures are reduced by dual networking and more replication bandwidth. NameNode failures are rare but cause downtime, so Hadoop 1 adds cold failover of Namenodes using VMware HA or RedHat HA. Hadoop 2 introduces live failover of Namenodes using a quorum journal manager to eliminate single points of failure. Full stack high availability adds monitoring and restart of all services.
More from Steve Loughran (20)
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and Hive
Apache Spark and Object Stores —for London Spark User Group
Apache Spark and Object Stores —for London Spark User Group
Spark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object stores
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Recently uploaded
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Available 24/7
GNSS spoofing via SDR (Criptored Talks 2024)
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
What is an RPA CoE? Session 2 – CoE Roles
In this session, we will review the players involved in the CoE and how each role impacts opportunities.
Topics covered:
• What roles are essential?
• What place in the automation journey does each role play?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Day 2 - Intro to UiPath Studio Fundamentals
In our second session, we shall learn all about the main features and fundamentals of UiPath Studio that enable us to use the building blocks for any automation project.
📕 Detailed agenda:
Variables and Datatypes
Workflow Layouts
Arguments
Control Flows and Loops
Conditional Statements
💻 Extra training through UiPath Academy:
Variables, Constants, and Arguments in Studio
Control Flow in Studio
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
Під час доповіді відповімо на питання, навіщо потрібно підвищувати продуктивність аплікації і які є найефективніші способи для цього. А також поговоримо про те, що таке кеш, які його види бувають та, основне — як знайти performance bottleneck?
Відео та деталі заходу: https://bit.ly/45tILxj
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
What is an RPA CoE? Session 1 – CoE Vision
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
This presentation, titled "MySQL - InnoDB" and delivered by Mayank Prasad at the Mydbops Open Source Database Meetup 16 on June 8th, 2024, covers dynamic configuration of REDO logs and instant ADD/DROP columns in InnoDB.
This presentation dives deep into the world of InnoDB, exploring two ground-breaking features introduced in MySQL 8.0:
• Dynamic Configuration of REDO Logs: Enhance your database's performance and flexibility with on-the-fly adjustments to REDO log capacity. Unleash the power of the snake metaphor to visualize how InnoDB manages REDO log files.
• Instant ADD/DROP Columns: Say goodbye to costly table rebuilds! This presentation unveils how InnoDB now enables seamless addition and removal of columns without compromising data integrity or incurring downtime.
Key Learnings:
• Grasp the concept of REDO logs and their significance in InnoDB's transaction management.
• Discover the advantages of dynamic REDO log configuration and how to leverage it for optimal performance.
• Understand the inner workings of instant ADD/DROP columns and their impact on database operations.
• Gain valuable insights into the row versioning mechanism that empowers instant column modifications.
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
ScyllaDB monitoring provides a lot of useful information. But sometimes it’s not easy to find the root of the problem if something is wrong or even estimate the remaining capacity by the load on the cluster. This talk shares our team's practical tips on: 1) How to find the root of the problem by metrics if ScyllaDB is slow 2) How to interpret the load and plan capacity for the future 3) Compaction strategies and how to choose the right one 4) Important metrics which aren’t available in the default monitoring setup.
Recently uploaded (20)
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Hadoop: Beyond MapReduce
- 1. © Hortonworks Inc. 2013 Hadoop: Beyond MapReduce Steve Loughran, Hortonworks stevel@hortonworks.com @steveloughran Big Data workshop, June 2013
- 2. © Hortonworks Inc. Hadoop MapReduce 1. Map: events <k,v>* pairs 2. Reduce: <k,[v1, v2,.. vn]> <k,v'> •Map trivially parallelisable on blocks in a file •Reduce parallelise on keys •MapReduce engine can execute Map and Reduce sequences against data •HDFS provides data location for work placement Page 2
- 3. © Hortonworks Inc. MapReduce democratised big data •Conceptual model easy to grasp •Can write and test locally, superlinear scaleup •Tools and stack Page 3 You don't need to understand parallel coding to run apps across 1000 machines
- 4. © Hortonworks Inc. 2012 The stack is key to use Page 4 Kafka
- 5. © Hortonworks Inc. 2012 Example: Pig Page 5 generated = LOAD '$src/$srcfile' USING PigStorage(',' , '-noschema') AS (line: int, gaussian: double, b: boolean, c:chararray ); sorted = ORDER generated BY c ASC; result = FILTER sorted BY gaussian >= 0;
- 6. © Hortonworks Inc. Example: Apache Giraph •Graph nodes in RAM •exchange data with peers at barriers •use cases: PageRank, Friend-of-Friend •But also: modelling cells in a heart Bulk-Synchronous-Parallel -read Pregel paper Page 6
- 7. © Hortonworks Inc. But there is a lot more we can do Page 7
- 8. © Hortonworks Inc. New Algorithms and runtimes •Giraph for graph work •Stream processing: Storm •Iterative and chained processing: Dryad-style •Long-lived processes Page 8
- 9. © Hortonworks Inc. Production-side issues •Scale to 10K nodes •Eliminate SPOFs & Bottlenecks •Improve versioning by moving MR engine user-side •Avoid having dedicated servers for other roles Page 9
- 10. © Hortonworks Inc. 2012 YARN: Yet Another Resource Negotiator Resource Manager MapReduce Status Job Submission Client Node Manager Node Manager Container Node Manager App Mstr Node Status Resource Request App Master manages the app AM can request containers and run code in them
- 11. © Hortonworks Inc. YARN vs Other Resource Negotiators •MapReduce #1 initial use case •Failures: AM handles worker failures, YARN handles AM failures •Scheduling Locality: sources of data, destinations. AM gets provides location requests along with (CPU, RAM Page 11
- 12. © Hortonworks Inc. Pig/Hive-MR versus Pig/Hive-Tez Page 12 I/O Synchronization Barrier I/O Pipelining Pig/Hive - Tez Pig/Hive - Tez SELECT a.state, COUNT(*) FROM a JOIN b ON (a.id = b.id) GROUP BY a.state
- 13. © Hortonworks Inc. FastQuery: Beyond Batch with YARN Page 13 Tez Generalizes Map-Reduce Simplified execution plans process data more efficiently Always-On Tez Service Low latency processing for all Hadoop data processing
- 14. © Hortonworks Inc. You too can write a distributed execution framework -if you need to Page 14
- 15. © Hortonworks Inc. Start the work in progress •Hamster: MPI •Storm-YARN from Yahoo! •Hoya: HBase on YARN me And start with other people's code •Continuuity Weave -looks best place to start Page 15
- 16. © Hortonworks Inc. What are the services and algorithms we are going to need? Page 16
- 17. © Hortonworks Inc http://hortonworks.com/careers/ Page 17 P.S: we are hiring