Downloaded 15 times
Uploaded byDanilo Poccia
Get Value From Your Data
The document provides an overview of various AWS services related to data analytics, including Amazon EMR for big data processing, Amazon Redshift for data warehousing, and Amazon Kinesis for real-time streaming data. It highlights the importance of choosing right data architectures and technologies to optimize costs, performance, and scalability. The document emphasizes the benefits of Amazon's services, such as pay-per-use pricing, ease of integration, and managing data effectively using cloud solutions.
More Related Content
![[よくわかるAmazon Redshift]Amazon Redshift最新情報と導入事例のご紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20140219redshiftupdatesv1tokyo-140224010117-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[よくわかるAmazon Redshift in 大阪]Amazon Redshift最新情報と導入事例のご紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20140221redshiftupdatesv2osaka-140224010309-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Get Value From Your Data
- 1. Get Value FromYour Data Danilo Poccia AWS Technical Evangelist @danilop danilop
- 6.
- 7. Instead of $20+MILLIONS in infrastructure
- 8. ON A SINGLEINSTANCE COMPUTE TIME: 4h COST: 4h x $2.1 = $8.4
- 9. ON MULTIPLE INSTANCES COMPUTETIME: 1h COST: 1h x 4 x $2.1 = $8.4
- 10.
- 11. Data Analytics Value >Costs Storage and Analysis Costs are Going Down Making New Use Cases Possible
- 12. + Elastic andHighly Scalable + No Upfront Capital Expense + Only Pay for What You Use + Available On-Demand = Remove Constraints
- 13. Structured Vs Unstructured Data High Degree of Organization DataModel Free Text Multimedia Social Media
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Unstructured Data Structured Data Resilient Distributed Datasets(RDDs) Memory Fast Processing Large Quantity of Data Disk Hadoop MapReduce Spark ?
- 20.
- 21.
- 22. Amazon Elastic MapReduce (Amazon EMR)Managed clusters For Hadoop, Spark, Presto or any other applications in the Apache / Hadoop stackWhat is Amazon EMR?
- 23. Amazon Elastic MapReduce (Amazon EMR) Overviewof Amazon EMR Architecture Storage HDFS EMRFS Local File System Data Processing Frameworks Hadoop Spark … Applications and Programs Hive Pig … ClusterResourceManagement YARNAgent…
- 24. Amazon Elastic MapReduce (Amazon EMR) Overviewof Amazon EMR Architecture Master Instance Group Core Instance Group Task Instance Group EC2 Spot Instances
- 25. Separate Compute andStorage Resize and shut down Amazon EMR clusters with no data loss Point multiple Amazon EMR clusters at the same data in Amazon S3 Easily evolve your analytic infrastructure as technology evolves Leverage Amazon S3 with EMR File System (EMRFS) S3 Bucket Cluster EMR Cluster Cluster EMR Cluster Amazon Elastic MapReduce (Amazon EMR)
- 26. Read-after-write consistency Very fastlist operations (thanks to Amazon DynamoDB) Transparent to applications as s3://… S3 Bucket Cluster EMR Cluster DynamoDB Table Amazon Elastic MapReduce (Amazon EMR) EMRFS makes it easier to use Amazon S3
- 27. CREATE EXTERNAL TABLEserde_regex( host STRING, referer STRING, agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' ) LOCATION ‘some/path/input/' S3 Bucket Cluster EMR Cluster DynamoDB Table Amazon Elastic MapReduce (Amazon EMR) Going from HDFS …
- 28. S3 Bucket Cluster EMR Cluster DynamoDBTable Amazon Elastic MapReduce (Amazon EMR) Going from HDFS to Amazon S3 CREATE EXTERNAL TABLE serde_regex( host STRING, referer STRING, agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' ) LOCATION 's3://bucket/path/input/'
- 29. Amazon Elastic MapReduce (Amazon EMR) EMRFSclient-side encryption S3 Bucket Cluster EMR Cluster Cluster EMR Cluster AWS KMS or your custom key vendor AmazonS3encryptionclients EMRFSenabledfor AmazonS3client-sideencryption
- 30. Amazon S3 isyour Data Lake S3 Bucket Cluster Hive, Pig Cluster Presto Cluster Spark Cluster Ad Hoc Cluster Cascading Logical Separation of Jobs
- 31. CASE STUDY: SPOTIFY ADDS20,000 TRACKS/DAY TO ITS CATALOGUE
- 32.
- 33.
- 34. A managed servicethat makes it easy to deploy, operate, and scale Elasticsearch in the AWS Cloud High availability, patch management, failure detection and node replacement, backups, and monitoring Integrated with Logstash and Kibana Scale up and scale down your cluster to deliver optimum performance as data and usage patterns change, paying only for the resources you actually consume Control access to the Elasticsearch APIs using AWS Identity and Access Management (IAM) policies What is Amazon ES? Amazon Elasticsearch Service (Amazon ES)
- 35. Amazon ES Architecture Amazon Elasticsearch Service (AmazonES) Elasticsearch Kibana Amazon CloudWatch AWS CloudTrail Elastic Load Balancing Amazon Route 53 Elasticsearch APIs AWS Credentials (AWS IAM)
- 36.
- 37.
- 38.
- 39. Relational Data Warehouse alot faster a lot simpler a lot cheaper Massively parallel + Petabyte scale Fully managed HDD and SSD Platforms $1,000/TB/Year; starts at $0.25/hour What is Amazon Redshift? Amazon Redshift
- 40. Amazon Redshift Architecture Amazon Redshift Compute Node Compute Node Compute Node Leader Node SQL Clients/ BI Tools Amazon S3 / Amazon DynamoDB / SSH 10GbE Ingestion/Backup JDBC / ODBC
- 41. Dramatically less I/O Columnstorage Data compression Zone maps Direct-attached storage Large data block sizes Amazon Redshift Performance Amazon Redshift analyze compression listing; Table | Column | Encoding ---------+----------------+---------- listing | listid | delta listing | sellerid | delta32k listing | eventid | delta32k listing | dateid | bytedict listing | numtickets | bytedict listing | priceperticket | delta32k listing | totalprice | mostly32 listing | listtime | raw 10 | 13 | 14 | 26 |… … | 100 | 245 | 324 375 | 393 | 417… … 512 | 549 | 623 637 | 712 | 809 … … | 834 | 921 | 959 10 324 375 623 637 959
- 42. Sort Keys and Zone Maps Amazon Redshift SELECTCOUNT(*) FROM LOGS WHERE DATE = ‘09-JUNE-2013’ Unsorted Sorted by Date MIN: 01-JUNE-2013 MAX: 20-JUNE-2013 MIN: 08-JUNE-2013 MAX: 30-JUNE-2013 MIN: 12-JUNE-2013 MAX: 20-JUNE-2013 MIN: 02-JUNE-2013 MAX: 25-JUNE-2013 MIN: 01-JUNE-2013 MAX: 06-JUNE-2013 MIN: 07-JUNE-2013 MAX: 12-JUNE-2013 MIN: 13-JUNE-2013 MAX: 18-JUNE-2013 MIN: 19-JUNE-2013 MAX: 24-JUNE-2013
- 43. Parallel and Distributed Amazon Redshift Compute Node Compute Node Compute Node Leader Node SQL Clients /BI Tools Amazon S3 / Amazon DynamoDB / SSH Query Load / Export / Backup / Restore
- 44. Parallel and Distributed Amazon Redshift Compute Node Compute Node Compute Node Leader Node SQL Clients /BI Tools Amazon S3 / Amazon DynamoDB / SSH Compute Node Query Load / Export / Backup / Restore Resize
- 45. Amazon Redshift Innovation Amazon Redshift Service Launch(2/14) PDX (4/2) Temp Credentials (4/11) DUB (4/25) SOC1/2/3 (5/8) Unload Encrypted Files NRT (6/5) JDBC Fetch Size (6/27) Unload logs (7/5) SHA1 Builtin (7/15) 4 byte UTF-8 (7/18) Sharing snapshots (7/18) Statement Timeout (7/22) Timezone, Epoch, Autoformat (7/25) WLM Timeout/Wildcards (8/1) CRC32 Builtin, CSV, Restore Progress (8/9) Resource Level IAM (8/9) PCI (8/22) UTF-8 Substitution (8/29) JSON, Regex, Cursors (9/10) Split_part, Audit tables (10/3) SIN/SYD (10/8) HSM Support (11/11) Kinesis EMR/HDFS/SSH copy, Distributed Tables, Audit Logging/CloudTrail, Concurrency, Resize Perf., Approximate Count Distinct, SNS Alerts, Cross Region Backup (11/13) Distributed Tables, Single Node Cursor Support, Maximum Connections to 500 (12/13) EIP Support for VPC Clusters (12/28) New query monitoring system tables and diststyle all (1/13) Redshift on DW2 (SSD) Nodes (1/23) Compression for COPY from SSH, Fetch size support for single node clusters, new system tables with commit stats, row_number(), strotol() and query termination (2/13) Resize progress indicator & Cluster Version (3/21) Regex_Substr, COPY from JSON (3/25) 50 slots, COPY from EMR, ECDHE ciphers (4/22) 3 new regex features, Unload to single file, FedRAMP(5/6) Rename Cluster (6/2) Copy from multiple regions, percentile_cont, percentile_disc (6/30) Free Trial (7/1) pg_last_unload_count (9/15) AES-128 S3 encryption (9/29) UTF-16 support (9/29) Well over 100 new features added since launch Release every two weeks Automatic patching
- 46. Amazon Redshift Features Amazon Redshift Approximate functions Userdefined functions Machine Learning Data Science Amazon ML
- 47. Amazon Redshift Ecosystem Amazon Redshift Data IntegrationSystems IntegratorsBusiness Intelligence
- 50. 30 MINUTES DOWNTO 12 SECONDS
- 51.
- 52.
- 53.
- 54.
- 55. A Platform forStreaming Data on AWS What is Amazon Kinesis? Amazon Kinesis Amazon Kinesis Streams Amazon Kinesis Firehose Amazon Kinesis Analytics
- 56. Amazon Kinesis Streams Amazon Kinesis Build your owncustom applications that process or analyze streaming data
- 57. Amazon Kinesis Streams Amazon Kinesis Use the KinesisClient Library (KCL) to consume data from Kinesys Streams
- 58. Amazon Kinesis Streams Amazon Kinesis AWS Lambda Functions Use AWSLambda for a serverless architecture
- 59. Amazon Kinesis Streams Amazon Kinesis Low latency I/O Configurableretention period from 1 to 7 days The maximum size of a data blob is up to 1 MB Each shard can support: up to 1,000 records / second and up to 1 MB / second for writes up to 5 transactions / second and up to 2 MB / second for reads
- 60. Amazon Kinesis Firehose Amazon Kinesis Easily load massivevolumes of streaming data into AWS
- 61. Amazon Kinesis Analytics Amazon Kinesis Easily analyze streamingdata with standard SQL (Coming Soon)
- 62.
- 63.
- 64.
- 65.
- 66. Machine learning isthe technology that automatically finds patterns in your data and uses them to make predictions for new data points as they become available Your Data + Machine Learning = Smart Applications What is Machine Learning? Amazon Machine Learning (Amazon ML)
- 67. Designed for Developers NoMachine Learning skills are required Batch prediction Real-time predictions Can be used by other applications via APIs What can you do? Amazon Machine Learning (Amazon ML)
- 68.
- 71.
- 72. B EST PR A C TIC ES USE ALL AVAILABLEDATA Your company has more data on your users than what you think…
- 73. Quizz What percentage ofdata do firms use for analytics? A: 12% C: 52% B: 34% D: 68%
- 74. Quizz What percentage ofdata do firms use for analytics? A: 12% C: 52% B: 34% D: 68%
- 75. B EST PR A C TIC ES ENRICH DATA BASED ONSOCIAL NETWORKS User’s friends are valuable sources of information
- 76. 75% of usersselect movies based on recommendations
- 78.
- 79.
- 80. Data Orchestration canbe a Task by Itself S3 Bucket Cluster EMR Cluster DynamoDB Table Redshift DB RDS Instance S3 Bucket On Premises
- 81. Helps you reliablyprocess and move data between different AWS compute and storage services, as well as on-premise data sources, at specified intervals What is AWS Data Pipeline? AWS Data Pipeline
- 82. Access your datawhere it’s stored, transform and process it at scale, and efficiently transfer the results to other AWS services What is AWS Data Pipeline? AWS Data Pipeline
- 83. Helps you migratedatabases to AWS easily and securely: the source database remains fully operational during the migration, minimizing downtime to applications that rely on the database What is AWS Database Migration Service? AWS Database Migration Service Customer Premises Application Users AWS Internet VPN AWS Database Migration Service
- 84. Migrate off Oracleand SQL Server Move your tables, views, stored procedures and DML to MySQL, MariaDB, and Amazon Aurora AWS Schema Conversion Tool AWS Database Migration Service
- 85. Know exactly wheremanual edits are needed AWS Schema Conversion Tool AWS Database Migration Service
- 86.
- 87.
- 88.
- 89.
- 90. A very fast,cloud-powered business intelligence (BI) service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from their data What is Amazon QuickSight? Amazon QuickSight
- 91. First analysis in about 60seconds Amazon QuickSight Business user Sign-in
- 92. Amazon QuickSight Architecture Amazon QuickSight Business User QuickSightAPI Data Prep Metadata SuggestionsConnectors SPICE Business User QuickSight UI Mobile Devices Web Browsers Partner BI products Amazon S3 Amazon Kinesis Amazon DynamoDB Amazon EMR Amazon Redshift Amazon RDSFiles Third-party
- 93. Point to aData Source
- 94.
- 95.
- 96.
- 97. Get Answers Fast Amazon QuickSight Amazon QuickSightuses SPICE – a Super-fast, Parallel, In-memory optimized Calculation Engine built from the ground up to generate answers on large datasets
- 98. Use AWS Partner BISolutions with Amazon QuickSight Amazon QuickSight Amazon QuickSight provides partners a simple SQL-like interface to query the data stored in SPICE, so that customers can continue using their existing BI tools while benefiting from the faster performance delivered by SPICE
- 99.
- 100. Collect Store Analyze AWSDirect Connect AWS Import/Export Disk AWS Import/Export Snowball Amazon Kinesis Streams Amazon VPC VPN Connection AWS Database Migration Service AWS Data Pipeline Amazon Kinesis Firehose Amazon Kinesis Analytics AWS Storage Gateway Amazon S3 Amazon Glacier Amazon RDS Amazon Redshift Amazon Elastisearch Service Amazon DynamoDB Amazon EMR Amazon EC2 Amazon EC2 Container Service Amazon ML Amazon QuickSight
- 101. Start Simple Amazon S3+ Amazon EMR or Amazon S3 + Amazon Redshift
- 102. Grow As YouNeed
- 103. Pay Only ForWhat You Use
- 104.
- 105. Get Value FromYour Data Danilo Poccia AWS Technical Evangelist @danilop danilop