Large-scale data processing with Hadoop in the Cloud

•Download as KEY, PDF•

1 like•420 views

A lightning talk that I gave at CloudCamp San Diego on how Hadoop can be used in the cloud to process more than 100,000,000 records

Technology Business

Large-scale data processing
with Hadoop in the Cloud

by Patrick Salami
Senior Software Engineer

Hadoop

Distributed computing framework

Runs on commodity hardware

Splits up large data sets

Easy to deploy in AWS with Elastic
MapReduce

Java Code
Raw Product
Normalize
Data
Filter

Categorize
S3
Shipping

Coupons
Hadoop Cluster
Group

Post-Process

Solr
S3 WWW
Index

x32 nodes
compute time $0.17/h $5.44/h
EMR $0.03/h $0.96/h

Total $0.20/h $6.40/h

Source: http://aws.amazon.com/ec2/#pricing

twitter.com/psalami
linkedin.com/in/psalami

Viewers also liked

Hadoop on the Cloud

Amazon Web Services

Hadoop in the Cloud - The what, why and how from the experts

DataWorks Summit/Hadoop Summit

Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...

Abhiraj Butala

BlueTalon-Isilon-Validation

Boni Bruno

The purpose of these slides is to give a high-level overview of Big Data concepts and techniques, as well as its related tools and technologies, focusing on Microsoft Azure. It starts by defining what Big Data is, as well as why Big Data platforms are needed. Fundamental components of a Big Data Platform are discussed, followed by a little bit of theory about Distributed Processing & CAP Theorem, and its relevance to how Big Data Solutions compare to Traditional RDBMS. Use case of how Big Data fits in Enterprise Data Platforms are shown. The Hadoop Ecosystem is briefly reviewed before Big Data on Microsoft Azure is discussed. Then some directions of How to get started with Big Data.

Intorducing Big Data and Microsoft Azure

Khalid Salama

Moving towards enterprise ready Hadoop clusters on the cloud

DataWorks Summit/Hadoop Summit

Hadoop in the Cloud – The What, Why and How from the Experts

DataWorks Summit/Hadoop Summit

Big Data in Azure

DataWorks Summit/Hadoop Summit

Microsoft Azure Big Data Analytics

Mark Kromer

Introducing DocumentDB

James Serra

Maintainable cloud architecture_of_hadoop

Kai Sasaki

This talk given at the Hadoop Summit in San Jose on June 28, 2016, analyzes a few major trends in Big Data analytics. These are a few takeaways from this talk: - Adopt Apache Beam for easier development and portability between Big Data Execution Engines. - Adopt stream analytics for faster time to insight, competitive advantages and operational efficiency. - Accelerate your Big Data applications with In-Memory open source tools. - Adopt Rapid Application Development of Big Data applications: APIs, Notebooks, GUIs, Microservices… - Have Machine Learning part of your strategy or passively watch your industry completely transformed! - How to advance your strategy for hybrid integration between cloud and on-premise deployments?

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Slim Baltagi

Compare Clouds: Aws vs Azure vs Google vs SoftLayer

RightScale

The Future of Apache Hadoop an Enterprise Architecture View

DataWorks Summit/Hadoop Summit

cloud computing ppt

himanshuawasthi2109

Viewers also liked (15)

Hadoop on the Cloud

Hadoop in the Cloud - The what, why and how from the experts

Hadoop Security in Big-Data-as-a-Service Deployments - Presented at Hadoop Su...

BlueTalon-Isilon-Validation

Intorducing Big Data and Microsoft Azure

Moving towards enterprise ready Hadoop clusters on the cloud

Hadoop in the Cloud – The What, Why and How from the Experts

Big Data in Azure

Microsoft Azure Big Data Analytics

Introducing DocumentDB

Maintainable cloud architecture_of_hadoop

Analysis-of-Major-Trends-in-big-data-analytics-slim-baltagi-hadoop-summit

Compare Clouds: Aws vs Azure vs Google vs SoftLayer

The Future of Apache Hadoop an Enterprise Architecture View

cloud computing ppt

Recently uploaded

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Architecting Cloud Native Applications

WSO2

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

DBX First Quarter 2024 Investor Presentation

Dropbox

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Recently uploaded (20)

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Exploring the Future Potential of AI-Enabled Smartphone Processors

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Artificial Intelligence Chap.5 : Uncertainty

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Architecting Cloud Native Applications

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - The value of a flexible API Management solution for O...

Why Teams call analytics are critical to your entire business

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Boost Fertility New Invention Ups Success Rates.pdf

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

presentation ICT roal in 21st century education

MINDCTI Revenue Release Quarter One 2024

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

MS Copilot expands with MS Graph connectors