Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2016 Utah Cloud Summit: Big Data Architectural Patterns and Best Practices on AWS

1,320 views

Published on

In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.

Published in: Technology
  • Be the first to comment

2016 Utah Cloud Summit: Big Data Architectural Patterns and Best Practices on AWS

  1. 1. Martin Schade, Solutions Architect, Amazon Web Services January 2016 Big Data Architectural Patterns and Best Practices on AWS
  2. 2. What to Expect from the Session Big data challenges How to simplify big data processing What technologies should you use? • Why? • How? Reference architecture Design patterns
  3. 3. Ever Increasing Big Data Volume Velocity Variety
  4. 4. Big Data Evolution Batch Report Real-time Alerts Prediction Forecast
  5. 5. Plethora of Tools Amazon Glacier S3 DynamoDB RDS EMR Amazon Redshift Data PipelineAmazon Kinesis Cassandra CloudSearch Kinesis- enabled app Lambda ML SQS ElastiCache DynamoDB Streams
  6. 6. Is there a reference architecture ? What tools should I use ? How ? Why ?
  7. 7. Architectural Principles • Decoupled “data bus” • Data → Store → Process → Answers • Use the right tool for the job • Data structure, latency, throughput, access patterns, cost • Use Lambda architecture ideas • Immutable (append-only) log, batch/speed/serving layer • Leverage AWS managed services • No/low admin • Big data ≠ big cost
  8. 8. Simplify Big Data Processing ingest / collect store process / analyze consume / visualize Time to Answer (Latency) Throughput Cost
  9. 9. Collect / Ingest
  10. 10. Types of Data • Transactional • Database reads & writes (OLTP) • Cache • Search • Logs • Streams • File • Log files (/var/log) • Log collectors & frameworks • Stream • Log records • Sensors & IoT data Database File Storage Stream Storage A iOS Android Web Apps Logstash LoggingIoTApplications Transactional Data File Data Stream Data Mobile Apps Search Data Search Collect Store LoggingIoT
  11. 11. Store
  12. 12. Stream Storage A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Database File Storage Search Collect Store LoggingIoTApplications 
  13. 13. Which stream storage should I use? Amazon Kinesis DynamoDB Streams Amazon SQS Amazon SNS Kafka Managed Yes Yes Yes No Ordering Yes Yes No Yes Delivery at-least-once exactly-once at-least-once at-least-once Lifetime 7 days 24 hours 14 days Configurable Replication 3 AZ 3 AZ 3 AZ Configurable Throughput No Limit No Limit No Limit ~ Nodes Parallel Clients Yes Yes No (SQS) Yes MapReduce Yes Yes No Yes Record size 1MB 400KB 256KB Configurable Cost Low Higher(table cost) Low-Medium Low (+admin)
  14. 14. File Storage A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Database Search Collect Store LoggingIoTApplications 
  15. 15. Why Is Amazon S3 Good for Big Data? • Natively supported by big data frameworks (Spark, Hive, Presto, etc.) • No need to run compute clusters for storage (unlike HDFS) • Can run transient Hadoop clusters & Amazon EC2 Spot instances • Multiple distinct (Spark, Hive, Presto) clusters can use the same data • Unlimited number of objects • Very high bandwidth – no aggregate throughput limit • Highly available – can tolerate AZ failure • Designed for 99.999999999% durability • Tired-storage (Standard, IA, Amazon Glacier) via life-cycle policy • Secure – SSL, client/server-side encryption at rest • Low cost
  16. 16. What about HDFS & Amazon Glacier? • Use HDFS for very frequently accessed (hot) data • Use Amazon S3 Standard for frequently accessed data • Use Amazon S3 Standard – IA for infrequently accessed data • Use Amazon Glacier for archiving cold data
  17. 17. Database + Search Tier A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Collect Store 
  18. 18. Database + Search Tier Anti-pattern Database + Search Tier
  19. 19. Best Practice — Use the Right Tool for the Job Data Tier Search Amazon Elasticsearch Service Amazon CloudSearch Cache Redis Memcached SQL Amazon Aurora MySQL PostgreSQL Oracle SQL Server NoSQL Cassandra Amazon DynamoDB HBase MongoDB Database + Search Tier
  20. 20. What Data Store Should I Use? • Data structure → Fixed schema, JSON, key-value • Access patterns → Store data in the format you will access it • Data / access characteristics → Hot, warm, cold • Cost → Right cost
  21. 21. Data Structure and Access Patterns Access Patterns What to use? Put/Get (Key, Value) Cache, NoSQL Simple relationships → 1:N, M:N NoSQL Cross table joins, transaction, SQL SQL Faceting, Search Search Data Structure What to use? Fixed schema SQL, NoSQL Schema-free (JSON) NoSQL, Search (Key, Value) Cache, NoSQL
  22. 22. Process / Analyze
  23. 23. AnalyzeA iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Hot Cold Warm Hot Hot ML Transactional Data File Data Stream Data Mobile Apps Search Data Collect Store Analyze  
  24. 24. Process / Analyze Analysis of data is a process of inspecting, cleaning, transforming, and modeling data with the goal of discovering useful information, suggesting conclusions, and supporting decision-making. Examples • Interactive dashboards → Interactive analytics • Daily/weekly/monthly reports → Batch analytics • Billing/fraud alerts, 1 minute metrics → Real-time analytics • Sentiment analysis, prediction models → Machine learning
  25. 25. Analysis Tools and Frameworks Machine Learning • Mahout, Spark ML, Amazon ML Interactive Analytics • Amazon Redshift, Presto, Impala, Spark Batch Processing • MapReduce, Hive, Pig, Spark Stream Processing • Micro-batch: Spark Streaming, KCL, Hive, Pig • Real-time: Storm, AWS Lambda, KCL Amazon Redshift Impala Pig Amazon Machine Learning Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce StreamProcessingBatchInteractiveML Analyze
  26. 26. What Stream Processing Technology Should I Use? Spark Streaming Apache Storm Amazon Kinesis Client Library AWS Lambda Amazon EMR (Hive, Pig) Scale / Throughput ~ Nodes ~ Nodes ~ Nodes Automatic ~ Nodes Batch or Real- time Real-time Real-time Real-time Real-time Batch Manageability Yes (Amazon EMR) Do it yourself Amazon EC2 + Auto Scaling AWS managed Yes (Amazon EMR) Fault Tolerance Single AZ Configurable Multi-AZ Multi-AZ Single AZ Programming languages Java, Python, Scala Any language via Thrift Java, via MultiLangDaemon ( .Net, Python, Ruby, Node.js) Node.js, Java Hive, Pig, Streaming languages High
  27. 27. What Data Processing Technology Should I Use? Amazon Redshift Impala Presto Spark Hive Query Latency Low Low Low Low Medium (Tez) – High (MapReduce) Durability High High High High High Data Volume 2 PB Max ~Nodes ~Nodes ~Nodes ~Nodes Managed Yes Yes (EMR) Yes (EMR) Yes (EMR) Yes (EMR) Storage Native HDFS / S3A* HDFS / S3 HDFS / S3 HDFS / S3 SQL Compatibility High Medium High Low (SparkSQL) Medium (HQL) HighMedium
  28. 28. What about ETL? Store Analyze https://aws.amazon.com/big-data/partner-solutions/ ETL
  29. 29. Consume / Visualize
  30. 30. Collect Store Analyze Consume A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Analysis&Visualization Hot Cold Warm Hot Slow Hot ML Fast Fast Transactional Data File Data Stream Data Notebooks Predictions Apps & APIs Mobile Apps IDE Search Data ETL Amazon QuickSight
  31. 31. Consume • Predictions • Analysis and Visualization • Notebooks • IDE • Applications & API Consume Analysis&Visualization Amazon QuickSight Notebooks Predictions Apps & APIs IDE Store Analyze ConsumeETL Business users Data Scientist, Developers
  32. 32. Putting It All Together
  33. 33. Collect Store Analyze Consume A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Analysis&Visualization Hot Cold Warm Hot Slow Hot ML Fast Fast Amazon QuickSight Transactional Data File Data Stream Data Notebooks Predictions Apps & APIs Mobile Apps IDE Search Data ETL Reference Architecture
  34. 34. Thank you! Have Fun!
  35. 35. Design Patterns
  36. 36. Multi-Stage Decoupled “Data Bus” • Multiple stages • Storage decoupled from processing Store Process Store Process process store
  37. 37. Multiple Processing Applications (or Connectors) Can Read from or Write to Multiple Data Stores Amazon Kinesis AWS Lambda Amazon DynamoDB Amazon Kinesis S3 Connector Amazon S3 process store
  38. 38. Processing Frameworks (KCL, Storm, Hive, Spark, etc.) Could Read from Multiple Data Stores Amazon Kinesis AWS Lambda Amazon S3 Amazon DynamoDB Hive SparkStorm Amazon Kinesis S3 Connector process store
  39. 39. Real-time Analytics Producer Apache Kafka KCL AWS Lambda Spark Streaming Apache Storm Amazon SNS Amazon ML Notifications Amazon ElastiCache (Redis) Amazon DynamoDB Amazon RDS Amazon ES Alert App state Real-time Prediction KPI process store DynamoDB Streams Amazon Kinesis
  40. 40. Interactive & Batch Analytics Producer Amazon S3 Amazon EMR Hive Pig Spark Amazon ML process store Consume Amazon Redshift Amazon EMR Presto Impala Spark Batch Interactive Batch Prediction Real-time Prediction
  41. 41. Batch Layer Amazon Kinesis data process store Lambda Architecture Amazon Kinesis S3 Connector Amazon S3 A p p l i c a t i o n s Amazon Redshift Amazon EMR Presto Hive Pig Spark answer Speed Layer answer Serving Layer Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES answer Amazon ML KCL AWS Lambda Spark Streaming Storm
  42. 42. Summary • Build decoupled “data bus” • Data → Store ↔ Process → Answers • Use the right tool for the job • Latency, throughput, access patterns • Use Lambda architecture ideas • Immutable (append-only) log, batch/speed/serving layer • Leverage AWS managed services • No/low admin • Be cost conscious • Big data ≠ big cost
  43. 43. Remember to complete your evaluations!
  44. 44. Thank you!

×