SlideShare a Scribd company logo

Building big data applications on AWS by Ran Tessler

•Download as PPTX, PDF•
•
I
Idan Tohami

Part of Big Data Month, foucs on Big Data Solution on AWS

Technology
1 of 17
Ran Tessler, Manager, Solutions Architecture Building Big Data Applications on AWS
A Modern Take on Alchemy Turning Data into Actionable Insights
What to Expect from this Session Big Data architectural principles Reference Lambda Architecture Live demo
Architectural Principles • Decoupled “data bus” Data → Store → Process → Answers • Use the right tool for the job Latency, throughput, access patterns • Apply Lambda architecture ideas Immutable (append-only) log, batch/speed/serving layer • Leverage AWS managed services No/low admin • Be cost conscious Big data ≠ big cost
Simplify Big Data Processing Ingest / collect store process / analyze consume / visualize Time to Answer (data freshness) Throughput
Demo http://aws.amazon.com/big-data/use-cases/
AccessLog - Common Log Format (CLF) 75.35.230.210 - - [20/Jul/2009:22:22:42 -0700] "GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236
Did NASA’s STS-69 Mission Land … … On the right homepage?
Your First Big Data Application on AWS PROCESS STORE ANALYZE & VISUALIZE COLLECT
Your First Big Data Application on AWS PROCESS STORE COLLECT: Amazon Kinesis Firehose ANALYZE & VISUALIZE
Your First Big Data Application on AWS STORE COLLECT: Amazon Kinesis Firehose ANALYZE & VISUALIZE PROCESS: Amazon EMR with Spark & Hive
Your First Big Data Application on AWS PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight COLLECT: Amazon Kinesis Firehose
Reference Lambda Architecture
process store Apps Batch Layer Amazon Kinesis S3 Connector Amazon S3 Amazon Redshift Amazon EMR Presto Hive Pig Spark Lambda Architecture Serving Layer Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES Amazon Kinesis Speed Layer KCL AWS Lambda Spark Streaming Storm Amazon MLdata
Back to our demo… PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight COLLECT: Amazon Kinesis Firehose
DIY Download all steps: http://bit.ly/29fhcwu http://aws.amazon.com/big-data/use-cases/
tesslerr@amazon.com

Recommended

IronSource Atom - Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons LearnedIronSource Atom - Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons LearnedIdan Tohami
 
Building a data warehouse with AWS Redshift, Matillion and Yellowfin
Building a data warehouse with AWS Redshift, Matillion and YellowfinBuilding a data warehouse with AWS Redshift, Matillion and Yellowfin
Building a data warehouse with AWS Redshift, Matillion and YellowfinLynn Langit
 
Big problems Big Data, simple solutions
Big problems Big Data, simple solutionsBig problems Big Data, simple solutions
Big problems Big Data, simple solutionsClaudio Pontili
 
SQL Server on Google Cloud Platform
SQL Server on Google Cloud PlatformSQL Server on Google Cloud Platform
SQL Server on Google Cloud PlatformLynn Langit
 
Big data on AWS
Big data on AWSBig data on AWS
Big data on AWSStylight
 
Beyond Relational
Beyond RelationalBeyond Relational
Beyond RelationalLynn Langit
 
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with EaseBenchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with EaseLynn Langit
 
The Fermilab HEPCloud Facility
The Fermilab HEPCloud FacilityThe Fermilab HEPCloud Facility
The Fermilab HEPCloud FacilityClaudio Pontili
 

Recommended

IronSource Atom - Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons LearnedIronSource Atom - Redshift - Lessons Learned
IronSource Atom - Redshift - Lessons LearnedIdan Tohami
 
Building a data warehouse with AWS Redshift, Matillion and Yellowfin
Building a data warehouse with AWS Redshift, Matillion and YellowfinBuilding a data warehouse with AWS Redshift, Matillion and Yellowfin
Building a data warehouse with AWS Redshift, Matillion and YellowfinLynn Langit
 
Big problems Big Data, simple solutions
Big problems Big Data, simple solutionsBig problems Big Data, simple solutions
Big problems Big Data, simple solutionsClaudio Pontili
 
SQL Server on Google Cloud Platform
SQL Server on Google Cloud PlatformSQL Server on Google Cloud Platform
SQL Server on Google Cloud PlatformLynn Langit
 
Big data on AWS
Big data on AWSBig data on AWS
Big data on AWSStylight
 
Beyond Relational
Beyond RelationalBeyond Relational
Beyond RelationalLynn Langit
 
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with EaseBenchmarking Aerospike on the Google Cloud - NoSQL Speed with Ease
Benchmarking Aerospike on the Google Cloud - NoSQL Speed with EaseLynn Langit
 
The Fermilab HEPCloud Facility
The Fermilab HEPCloud FacilityThe Fermilab HEPCloud Facility
The Fermilab HEPCloud FacilityClaudio Pontili
 
Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3Gareth Rogers
 
Managing application & instance state on AWS
Managing application & instance state on AWSManaging application & instance state on AWS
Managing application & instance state on AWSDavid Mat
 
Real-Time Vote Platform Benchmark
Real-Time Vote Platform BenchmarkReal-Time Vote Platform Benchmark
Real-Time Vote Platform BenchmarkLahav Savir
 
Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019Petr Zapletal
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
Apache Cassandra in the Cloud
Apache Cassandra in the CloudApache Cassandra in the Cloud
Apache Cassandra in the CloudInstaclustr
 
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Coburn Watson
 
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudGCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudSamuel Chow
 
PloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safetyPloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safetyChris Shenton
 
Cloud Overview
Cloud OverviewCloud Overview
Cloud Overviewiasaglobal
 
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson
 
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...HostedbyConfluent
 
Spotify's journey to GCP
Spotify's journey to GCPSpotify's journey to GCP
Spotify's journey to GCPAlexey Lapitsky
 
Intro to the Google Cloud for Developers
Intro to the Google Cloud for DevelopersIntro to the Google Cloud for Developers
Intro to the Google Cloud for DevelopersLynn Langit
 
Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, Milwaukee Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, MilwaukeeAWS Chicago
 
Big Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the CloudBig Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the CloudGeorge Ang
 
Presto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - LyftPresto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - Lyftkbajda
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)Amazon Web Services
 
Introduction to aws data pipeline services
Introduction to aws data pipeline servicesIntroduction to aws data pipeline services
Introduction to aws data pipeline servicesArcBlock
 
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, AthenaSLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, AthenaTimothy Collinson
 
Building Big Data Applications on AWS
Building Big Data Applications on AWSBuilding Big Data Applications on AWS
Building Big Data Applications on AWSAmazon Web Services
 
Data Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudData Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudIan Massingham
 

More Related Content

What's hot

Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3Gareth Rogers
 
Managing application & instance state on AWS
Managing application & instance state on AWSManaging application & instance state on AWS
Managing application & instance state on AWSDavid Mat
 
Real-Time Vote Platform Benchmark
Real-Time Vote Platform BenchmarkReal-Time Vote Platform Benchmark
Real-Time Vote Platform BenchmarkLahav Savir
 
Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019Petr Zapletal
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
Apache Cassandra in the Cloud
Apache Cassandra in the CloudApache Cassandra in the Cloud
Apache Cassandra in the CloudInstaclustr
 
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Coburn Watson
 
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudGCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudSamuel Chow
 
PloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safetyPloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safetyChris Shenton
 
Cloud Overview
Cloud OverviewCloud Overview
Cloud Overviewiasaglobal
 
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson
 
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...HostedbyConfluent
 
Spotify's journey to GCP
Spotify's journey to GCPSpotify's journey to GCP
Spotify's journey to GCPAlexey Lapitsky
 
Intro to the Google Cloud for Developers
Intro to the Google Cloud for DevelopersIntro to the Google Cloud for Developers
Intro to the Google Cloud for DevelopersLynn Langit
 
Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, Milwaukee Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, MilwaukeeAWS Chicago
 
Big Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the CloudBig Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the CloudGeorge Ang
 
Presto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - LyftPresto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - Lyftkbajda
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)Amazon Web Services
 
Introduction to aws data pipeline services
Introduction to aws data pipeline servicesIntroduction to aws data pipeline services
Introduction to aws data pipeline servicesArcBlock
 
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, AthenaSLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, AthenaTimothy Collinson
 

What's hot (20)

Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3Metail at Cambridge AWS User Group Main Meetup #3
Metail at Cambridge AWS User Group Main Meetup #3
 
Managing application & instance state on AWS
Managing application & instance state on AWSManaging application & instance state on AWS
Managing application & instance state on AWS
 
Real-Time Vote Platform Benchmark
Real-Time Vote Platform BenchmarkReal-Time Vote Platform Benchmark
Real-Time Vote Platform Benchmark
 
Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019Change Data Capture - Scale by the Bay 2019
Change Data Capture - Scale by the Bay 2019
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
 
Apache Cassandra in the Cloud
Apache Cassandra in the CloudApache Cassandra in the Cloud
Apache Cassandra in the Cloud
 
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016
 
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudGCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the Cloud
 
PloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safetyPloneConf2017: serverless python for astronaut safety
PloneConf2017: serverless python for astronaut safety
 
Cloud Overview
Cloud OverviewCloud Overview
Cloud Overview
 
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
 
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...
 
Spotify's journey to GCP
Spotify's journey to GCPSpotify's journey to GCP
Spotify's journey to GCP
 
Intro to the Google Cloud for Developers
Intro to the Google Cloud for DevelopersIntro to the Google Cloud for Developers
Intro to the Google Cloud for Developers
 
Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, Milwaukee Why Isn't the Cloud Cheaper - John Merline, Milwaukee
Why Isn't the Cloud Cheaper - John Merline, Milwaukee
 
Big Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the CloudBig Data on EC2: Mashing Technology in the Cloud
Big Data on EC2: Mashing Technology in the Cloud
 
Presto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - LyftPresto Summit 2018 - 07 - Lyft
Presto Summit 2018 - 07 - Lyft
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
 
Introduction to aws data pipeline services
Introduction to aws data pipeline servicesIntroduction to aws data pipeline services
Introduction to aws data pipeline services
 
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, AthenaSLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
SLC .Net User Group -- .Net, Kinesis Firehose, Glue, Athena
 

Similar to Building big data applications on AWS by Ran Tessler

Building Big Data Applications on AWS
Building Big Data Applications on AWSBuilding Big Data Applications on AWS
Building Big Data Applications on AWSAmazon Web Services
 
Data Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudData Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudIan Massingham
 
Journey Through the Cloud - Data Analysis
Journey Through the Cloud - Data AnalysisJourney Through the Cloud - Data Analysis
Journey Through the Cloud - Data AnalysisAmazon Web Services
 
AWS Webcast - Attunity Couchsurfing
AWS Webcast - Attunity CouchsurfingAWS Webcast - Attunity Couchsurfing
AWS Webcast - Attunity CouchsurfingAmazon Web Services
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your StartupAmazon Web Services
 
Migrating Enterprise Applications to AWS
Migrating Enterprise Applications to AWSMigrating Enterprise Applications to AWS
Migrating Enterprise Applications to AWSTom Laszewski
 
Being Well Architected in the Cloud (Updated)
Being Well Architected in the Cloud (Updated)Being Well Architected in the Cloud (Updated)
Being Well Architected in the Cloud (Updated)Adrian Hornsby
 
Being Well-Architected in the Cloud
Being Well-Architected in the CloudBeing Well-Architected in the Cloud
Being Well-Architected in the CloudAmazon Web Services
 
Big Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSBig Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSAmazon Web Services
 
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...Amazon Web Services
 
Scale 02 king_summit_aws
Scale 02 king_summit_awsScale 02 king_summit_aws
Scale 02 king_summit_awsAtlassian
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your StartupAmazon Web Services
 
How Glidewell Moves Data to Amazon Redshift
How Glidewell Moves Data to Amazon RedshiftHow Glidewell Moves Data to Amazon Redshift
How Glidewell Moves Data to Amazon RedshiftAttunity
 
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWS
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWSAWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWS
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWSAmazon Web Services
 
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloud
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloudA1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloud
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloudDr. Wilfred Lin (Ph.D.)
 
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013Understanding AWS Database Options (DAT201) | AWS re:Invent 2013
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013Amazon Web Services
 

Similar to Building big data applications on AWS by Ran Tessler (20)

Building Big Data Applications on AWS
Building Big Data Applications on AWSBuilding Big Data Applications on AWS
Building Big Data Applications on AWS
 
Data Analysis - Journey Through the Cloud
Data Analysis - Journey Through the CloudData Analysis - Journey Through the Cloud
Data Analysis - Journey Through the Cloud
 
Journey Through the Cloud - Data Analysis
Journey Through the Cloud - Data AnalysisJourney Through the Cloud - Data Analysis
Journey Through the Cloud - Data Analysis
 
AWS Webcast - Attunity Couchsurfing
AWS Webcast - Attunity CouchsurfingAWS Webcast - Attunity Couchsurfing
AWS Webcast - Attunity Couchsurfing
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your Startup
 
Migrating Enterprise Applications to AWS
Migrating Enterprise Applications to AWSMigrating Enterprise Applications to AWS
Migrating Enterprise Applications to AWS
 
Being Well Architected in the Cloud (Updated)
Being Well Architected in the Cloud (Updated)Being Well Architected in the Cloud (Updated)
Being Well Architected in the Cloud (Updated)
 
Being Well-Architected in the Cloud
Being Well-Architected in the CloudBeing Well-Architected in the Cloud
Being Well-Architected in the Cloud
 
Big Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWSBig Data Architectural Patterns and Best Practices on AWS
Big Data Architectural Patterns and Best Practices on AWS
 
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...
Migrating Enterprise Applications to AWS: Best Practices & Techniques (ENT303...
 
Scale 02 king_summit_aws
Scale 02 king_summit_awsScale 02 king_summit_aws
Scale 02 king_summit_aws
 
Scaling the Platform for Your Startup
Scaling the Platform for Your StartupScaling the Platform for Your Startup
Scaling the Platform for Your Startup
 
Using Data Lakes
Using Data LakesUsing Data Lakes
Using Data Lakes
 
How Glidewell Moves Data to Amazon Redshift
How Glidewell Moves Data to Amazon RedshiftHow Glidewell Moves Data to Amazon Redshift
How Glidewell Moves Data to Amazon Redshift
 
Big Data on AWS
Big Data on AWSBig Data on AWS
Big Data on AWS
 
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWS
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWSAWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWS
AWS Summit Stockholm 2014 – B2 – Migrating enterprise applications to AWS
 
AWS glue technical enablement training
AWS glue technical enablement trainingAWS glue technical enablement training
AWS glue technical enablement training
 
Create cloud service on AWS
Create cloud service on AWSCreate cloud service on AWS
Create cloud service on AWS
 
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloud
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloudA1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloud
A1 keynote oracle_infrastructure_as_a_service_move_any_workload_to_the_cloud
 
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013Understanding AWS Database Options (DAT201) | AWS re:Invent 2013
Understanding AWS Database Options (DAT201) | AWS re:Invent 2013
 

More from Idan Tohami

Simplify Your Security with Cybowall
Simplify Your Security with CybowallSimplify Your Security with Cybowall
Simplify Your Security with CybowallIdan Tohami
 
AML Transaction Monitoring Tuning Webinar
AML Transaction Monitoring Tuning WebinarAML Transaction Monitoring Tuning Webinar
AML Transaction Monitoring Tuning WebinarIdan Tohami
 
Robotic Process Automation (RPA) Webinar - By Matrix-IFS
Robotic Process Automation (RPA) Webinar - By Matrix-IFSRobotic Process Automation (RPA) Webinar - By Matrix-IFS
Robotic Process Automation (RPA) Webinar - By Matrix-IFSIdan Tohami
 
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...Idan Tohami
 
Robotic Automation Process (RPA) Webinar - By Matrix-IFS
Robotic Automation Process (RPA) Webinar - By Matrix-IFSRobotic Automation Process (RPA) Webinar - By Matrix-IFS
Robotic Automation Process (RPA) Webinar - By Matrix-IFSIdan Tohami
 
Robotic Automation Process (RPA) Brochure - By Matrix-IFS
Robotic Automation Process (RPA) Brochure - By Matrix-IFSRobotic Automation Process (RPA) Brochure - By Matrix-IFS
Robotic Automation Process (RPA) Brochure - By Matrix-IFSIdan Tohami
 
The Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi CloudThe Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi CloudIdan Tohami
 
Introdction to Cloud Regulation for Enterprise by 2Bsecure
Introdction to Cloud Regulation for Enterprise by 2BsecureIntrodction to Cloud Regulation for Enterprise by 2Bsecure
Introdction to Cloud Regulation for Enterprise by 2BsecureIdan Tohami
 
Enterprise Journey to the Cloud - Opening Remarks
Enterprise Journey to the Cloud - Opening RemarksEnterprise Journey to the Cloud - Opening Remarks
Enterprise Journey to the Cloud - Opening RemarksIdan Tohami
 
Ready.Set.Cloud - Enterprise Cloud Migration Framework
Ready.Set.Cloud - Enterprise Cloud Migration FrameworkReady.Set.Cloud - Enterprise Cloud Migration Framework
Ready.Set.Cloud - Enterprise Cloud Migration FrameworkIdan Tohami
 
Journey to the Public Cloud
Journey to the Public CloudJourney to the Public Cloud
Journey to the Public CloudIdan Tohami
 
Google Cloud Fundamentals by CloudZone
Google Cloud Fundamentals by CloudZoneGoogle Cloud Fundamentals by CloudZone
Google Cloud Fundamentals by CloudZoneIdan Tohami
 
HDinsight Workshop - Prerequisite Activity
HDinsight Workshop - Prerequisite ActivityHDinsight Workshop - Prerequisite Activity
HDinsight Workshop - Prerequisite ActivityIdan Tohami
 
Cloud Regulations and Security Standards by Ran Adler
Cloud Regulations and Security Standards by Ran AdlerCloud Regulations and Security Standards by Ran Adler
Cloud Regulations and Security Standards by Ran AdlerIdan Tohami
 
Azure Logic Apps by Gil Gross, CloudZone
Azure Logic Apps by Gil Gross, CloudZoneAzure Logic Apps by Gil Gross, CloudZone
Azure Logic Apps by Gil Gross, CloudZoneIdan Tohami
 
AWS Fundamentals @Back2School by CloudZone
AWS Fundamentals @Back2School by CloudZoneAWS Fundamentals @Back2School by CloudZone
AWS Fundamentals @Back2School by CloudZoneIdan Tohami
 
Cloud Accelerated Genomics
Cloud Accelerated GenomicsCloud Accelerated Genomics
Cloud Accelerated GenomicsIdan Tohami
 
Achieving HIPAA on GCP
Achieving HIPAA on GCPAchieving HIPAA on GCP
Achieving HIPAA on GCPIdan Tohami
 

More from Idan Tohami (20)

Simplify Your Security with Cybowall
Simplify Your Security with CybowallSimplify Your Security with Cybowall
Simplify Your Security with Cybowall
 
AML Transaction Monitoring Tuning Webinar
AML Transaction Monitoring Tuning WebinarAML Transaction Monitoring Tuning Webinar
AML Transaction Monitoring Tuning Webinar
 
Robotic Process Automation (RPA) Webinar - By Matrix-IFS
Robotic Process Automation (RPA) Webinar - By Matrix-IFSRobotic Process Automation (RPA) Webinar - By Matrix-IFS
Robotic Process Automation (RPA) Webinar - By Matrix-IFS
 
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...
Open Banking / PSD2 & GDPR Regulations and How They Are Changing Fraud & Fina...
 
Robotic Automation Process (RPA) Webinar - By Matrix-IFS
Robotic Automation Process (RPA) Webinar - By Matrix-IFSRobotic Automation Process (RPA) Webinar - By Matrix-IFS
Robotic Automation Process (RPA) Webinar - By Matrix-IFS
 
Robotic Automation Process (RPA) Brochure - By Matrix-IFS
Robotic Automation Process (RPA) Brochure - By Matrix-IFSRobotic Automation Process (RPA) Brochure - By Matrix-IFS
Robotic Automation Process (RPA) Brochure - By Matrix-IFS
 
The Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi CloudThe Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi Cloud
 
Introdction to Cloud Regulation for Enterprise by 2Bsecure
Introdction to Cloud Regulation for Enterprise by 2BsecureIntrodction to Cloud Regulation for Enterprise by 2Bsecure
Introdction to Cloud Regulation for Enterprise by 2Bsecure
 
Enterprise Journey to the Cloud - Opening Remarks
Enterprise Journey to the Cloud - Opening RemarksEnterprise Journey to the Cloud - Opening Remarks
Enterprise Journey to the Cloud - Opening Remarks
 
Vmware on aws
Vmware on awsVmware on aws
Vmware on aws
 
Ready.Set.Cloud - Enterprise Cloud Migration Framework
Ready.Set.Cloud - Enterprise Cloud Migration FrameworkReady.Set.Cloud - Enterprise Cloud Migration Framework
Ready.Set.Cloud - Enterprise Cloud Migration Framework
 
Journey to the Public Cloud
Journey to the Public CloudJourney to the Public Cloud
Journey to the Public Cloud
 
Google Cloud Fundamentals by CloudZone
Google Cloud Fundamentals by CloudZoneGoogle Cloud Fundamentals by CloudZone
Google Cloud Fundamentals by CloudZone
 
HDinsight Workshop - Prerequisite Activity
HDinsight Workshop - Prerequisite ActivityHDinsight Workshop - Prerequisite Activity
HDinsight Workshop - Prerequisite Activity
 
Cloud Regulations and Security Standards by Ran Adler
Cloud Regulations and Security Standards by Ran AdlerCloud Regulations and Security Standards by Ran Adler
Cloud Regulations and Security Standards by Ran Adler
 
Azure Logic Apps by Gil Gross, CloudZone
Azure Logic Apps by Gil Gross, CloudZoneAzure Logic Apps by Gil Gross, CloudZone
Azure Logic Apps by Gil Gross, CloudZone
 
AWS Fundamentals @Back2School by CloudZone
AWS Fundamentals @Back2School by CloudZoneAWS Fundamentals @Back2School by CloudZone
AWS Fundamentals @Back2School by CloudZone
 
Cloud Accelerated Genomics
Cloud Accelerated GenomicsCloud Accelerated Genomics
Cloud Accelerated Genomics
 
Achieving HIPAA on GCP
Achieving HIPAA on GCPAchieving HIPAA on GCP
Achieving HIPAA on GCP
 
Couchbase Day
Couchbase DayCouchbase Day
Couchbase Day
 

Recently uploaded

08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Hyundai Motor Group
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard
 

Recently uploaded (20)

08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
The transition to renewables in India.pdf
The transition to renewables in India.pdfThe transition to renewables in India.pdf
The transition to renewables in India.pdf
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptxVulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptx
 

Building big data applications on AWS by Ran Tessler

  • 1. Ran Tessler, Manager, Solutions Architecture Building Big Data Applications on AWS
  • 2. A Modern Take on Alchemy Turning Data into Actionable Insights
  • 3. What to Expect from this Session Big Data architectural principles Reference Lambda Architecture Live demo
  • 4. Architectural Principles • Decoupled “data bus” Data → Store → Process → Answers • Use the right tool for the job Latency, throughput, access patterns • Apply Lambda architecture ideas Immutable (append-only) log, batch/speed/serving layer • Leverage AWS managed services No/low admin • Be cost conscious Big data ≠ big cost
  • 5. Simplify Big Data Processing Ingest / collect store process / analyze consume / visualize Time to Answer (data freshness) Throughput
  • 7. AccessLog - Common Log Format (CLF) 75.35.230.210 - - [20/Jul/2009:22:22:42 -0700] "GET /images/pigtrihawk.jpg HTTP/1.1" 200 29236
  • 8. Did NASA’s STS-69 Mission Land … … On the right homepage?
  • 9. Your First Big Data Application on AWS PROCESS STORE ANALYZE & VISUALIZE COLLECT
  • 10. Your First Big Data Application on AWS PROCESS STORE COLLECT: Amazon Kinesis Firehose ANALYZE & VISUALIZE
  • 11. Your First Big Data Application on AWS STORE COLLECT: Amazon Kinesis Firehose ANALYZE & VISUALIZE PROCESS: Amazon EMR with Spark & Hive
  • 12. Your First Big Data Application on AWS PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight COLLECT: Amazon Kinesis Firehose
  • 14. process store Apps Batch Layer Amazon Kinesis S3 Connector Amazon S3 Amazon Redshift Amazon EMR Presto Hive Pig Spark Lambda Architecture Serving Layer Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES Amazon Kinesis Speed Layer KCL AWS Lambda Spark Streaming Storm Amazon MLdata
  • 15. Back to our demo… PROCESS: Amazon EMR with Spark & Hive STORE ANALYZE & VISUALIZE: Amazon Redshift and Amazon QuickSight COLLECT: Amazon Kinesis Firehose
  • 16. DIY Download all steps: http://bit.ly/29fhcwu http://aws.amazon.com/big-data/use-cases/