Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multi-Source, Multi-Speed Analytics on AWS Webinar

2,103 views

Published on

Data is becoming more varied in type and in speed. We will explain and demonstrate how to consume data from IoT devices and API’s together with traditional data stores to gain insights at scale by combining real-time data feeds with traditional data sources to produce immediate insights.

Speaker: Paul Macey, Big Data Specialist Solutions Architect, AWS

  • Be the first to comment

Multi-Source, Multi-Speed Analytics on AWS Webinar

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Paul Macey Specialist Solution Architect, Big Data and Analytics AWS Public Sector November 2019 Multi Source, Multi Speed Analytics on AWS Deep dive webinar
  2. 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Sources of data Organisational Goldmines Well architected data pipelines Combining multispeed data sources Wrap up
  3. 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Geospatial Operational Data Store Data ScientistBespoke datasets Other data stores Analyst Operations Sources of data IoT
  4. 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Organisational goldmines csv json xls Databases Batch Streaming IoT SFTP
  5. 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Queensland Department of Transport and Main Road’s (TMR) Connected and Autonomous Vehicle Initiative (CAVI) https://www.itnews.com.au/news/queensland-debuts-most-advanced-driverless-car-in-oz-529501
  6. 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics and Insights ProcessInitiation Data Lake Storage Metadata / Search Big Data Querying, ETL & Insights Database / BI Amazon Athena Amazon QuickSight AWS Glue Amazon S3
  7. 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Combining multispeed data sources
  8. 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. External Data Pattern Weather Amazon EventBridge AWS Lambda Amazon S3
  9. 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo External data
  10. 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming
  11. 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Generating streaming data Kinesis Data Generator https://awslabs.github.io/amazon-kinesis-data-generator/ For the streaming demonstration the generator will be producing simulated smart city camera data
  12. 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming Data Pattern Smart City Camera camera_stream datetime sensorId locationId currentTemperature battery objectDetected status camera_stream Create SQL Schema Write SQL to query stream Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon Kinesis Data Analytics Amazon Kinesis Data Analytics
  13. 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo Streaming data
  14. 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SFTP Pattern – Reference Data + No end-user disruption + Fully managed servers + Simple to use + Pay as you use + Native cloud integrations AWS SFTP Amazon S3 AWS Transfer for SFTP
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo Reference data
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo Combining streaming, external & reference data
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. csv json xls Databases Batch Streaming IoT AWS has services to bring together each of YOUR organisational data goldmines Geospatial Operational Data Store Operations External Requests Data Scientist Analyst Sales SFTP
  18. 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Available today @ GitHub https://github.com/aws-samples/accelerated-data-lake Includes Data lake pipeline (CloudFormation) Instructions Data configuration, security and metadata templates Delivery Professional services AWS partners Accelerated Data Lake
  19. 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. References AWS Accelerated Data Lake (Git) https://github.com/aws-samples/accelerated-data-lake AWS Accelerated Data Lake Blog (part 1 & 2) https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-together https://aws.amazon.com/blogs/publicsector/securing-your-data-by-knowing-your-data Our data lake story: How Woot.com built a serverless data lake on AWS https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws Kinesis Data Generator https://awslabs.github.io/amazon-kinesis-data-generator/
  20. 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. References Amazon Kinesis https://aws.amazon.com/kinesis/ Amazon Kinesis Analytics https://aws.amazon.com/kinesis/data- analytics
  21. 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Webinar question resources – Data validation https://json-schema.org/ https://frictionlessdata.io/docs/csv/ https://frictionlessdata.io/docs/csv/#libraries https://pypi.org/project/CsvSchema/
  22. 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Webinar question resources - CDC https://aws.amazon.com/blogs/database/load-cdc-data-from-relational-databases-to-amazon-kinesis-using-database- migration-service/ https://aws.amazon.com/blogs/database/use-the-aws-database-migration-service-to-stream-change-data-to-amazon- kinesis-data-streams/ https://aws.amazon.com/blogs/database/automating-database-migration-and-refreshing-activities-with-aws-dms/ https://aws.amazon.com/blogs/database/stream-changes-from-amazon-rds-for-postgresql-using-amazon-kinesis-data- streams-and-aws-lambda/ https://docs.aws.amazon.com/en_pv/streams/latest/dev/kinesis-record-processor-ddb.html https://aws.amazon.com/blogs/big-data/loading-ongoing-data-lake-changes-with-aws-dms-and-aws-glue/
  23. 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Webinar question resources – AWS Glue & JSON https://aws.amazon.com/blogs/big-data/simplify-querying-nested-json-with-the-aws-glue-relationalize-transform/ https://docs.aws.amazon.com/en_pv/glue/latest/dg/add-classifier.html https://docs.aws.amazon.com/en_pv/glue/latest/dg/aws-glue-programming-python-samples-legislators.html

×