Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Immersion Day - IoT, Big Data, and Machine Learning Specialist session

0 views

Published on

Take your next steps with AWS and learn in this deep-dive session how data is generated, cost effectively analysed, and then developed to predicted actionable outcomes.

  • Be the first to comment

AWS Immersion Day - IoT, Big Data, and Machine Learning Specialist session

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark AWS | Immersion Day
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Immersion Day - IoT & Transport IoT, Big Data & Machine Learning Specialist Session Craig Lawton Smart Cities and IoT Solutions Architecture, A/NZ
  3. 3. Today’s Journey: IoT, Big Data and ML pipeline Connected vehicles with AWS Greengrass Core Staged data Raw data IoT Big Data ML
  4. 4. Digital Transformation means trillions of connected devices making data, decisions, and giving directions
  5. 5. To securely connect devices to the AWS cloud & other devices at scale To fully integrate with other AWS services to reason on top of the data (Analytics, Databases, AI, etc.) To route, process, and act upon data from connected devices AWS IoT Core is a managed service that lets connected devices easily and securely interact with cloud applications and other devices. To enable applications to interact with devices even when they are offline AWS IoT Core Control services
  6. 6. Driverless Car
  7. 7. Some things must be done locally to the car Image recognition Threat detection Road management Collision control Time sensitive analysis Processing that can’t go offline
  8. 8. Some things are best handled in the cloud Turn by turn instructions Map updates and road construction Traffic/congestion management Car efficiency management (data analysis for best effect) Refueling, maintenance needs Logging usage/managing usage payments Management and upgrades
  9. 9. Edge computing is:
  10. 10. Edge Cloud
  11. 11. Edge Cloud Law of Economics Law of Physics Law of the Land AWS IoT Greengrass AWS IoT Greengrass extends AWS IoT onto your devices, so that they can act locally on the data they generate, while still taking advantage of the cloud. Device software
  12. 12. AWS IoT Greengrass: Why did we build it?
  13. 13. Inference Training Machine Learning at the Edge Local actions Edge Cloud Amazon SageMaker Optimize with Neo AWS IoT Greengrass Feedback
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS IoT Greengrass use-cases IIoT data collection and organization: AWS IoT SiteWise Machine Learning at the Edge • Predictive Maintenance - AWS IoT Analytics • Video Analytics: • AWS DeepLens, AWS DeepRacer • Warny iCETANA • ISVs AWS Snowball Edge
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS RoboMaker Development Environment SimulationCloud Extensions for ROS Fleet Management
  16. 16. AWS IoT architecture IoT Greengrass Amazon FreeRTOS AWS IoT Device SDK IoT Core IoT Device Management IoT Device Defender IoT Things GraphAWSIoTDevice Tester IoT Analytics IoT SiteWise IoT Events 2 1
  17. 17. Safety Impact Analysis Project
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS IoT Greengrass and Automotive Grade Linux
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS IoT Greengrass at the Mobile Edge
  20. 20. Vehicle-to-everything • 802.11p (DSRC) • 3GPP (C-V2X) • Forward collision warning • Lane change warning/blind spot warning • Emergency electric brake light warning • Intersection movement assist • Emergency vehicle approaching • Roadworks warning • Platooning
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS IoT Greengrass & Connected Vehicle
  22. 22. Fire up the simulator Connected vehicles with AWS Greengrass Core Staged data Raw data IoT Big Data ML
  23. 23. AWS IoT Greengrass release v1.7 • Simplified deployments • AWS IoT Greengrass Connectors • Enhanced security • AWS IoT Greengrass Secrets Manager • AWS IoT Greengrass Hardware Security Integration • Greater flexibility • AWS IoT Greengrass per-Lambda Isolation & Permission Configurations • Try it from the console • Connectors and documentation available now • Find qualified hardware • Search the Amazon Partner Network Device Qualification Portal for Greengrass HSI [https://devices.amazonaws.com/search?kw=HSI&page=1] • Use our Docker file • Access a Greengrass Docker file here • See documentation about pullling the Greengrass Docker image from AWS ECR here
  24. 24. AWS IoT Greengrass Connectors • More to come in 2019 • What are connectors? • Connectivity to AWS services, industrial protocols, local- and cloud-based applications • Code-free configuration and installation • 11 connectors available today Amazon Kinesis Data Firehose Amazon Simple Notification Service AWS IoT Device Defender Amazon CloudWatch Modbus RTU Protocol Adapter Raspberry Pi GPIO Serial Stream ML Inference
  25. 25. AWS IoT Greengrass Secrets Manager • Extends AWS Secrets Manager to the Greengrass cores for secure management of keys, passwords, credentials, endpoints, and configurations • Usable by connectors or AWS Lambda functions
  26. 26. AWS IoT Greengrass hardware security integration Private key stored in file system PKCS#11API Interface
  27. 27. Isolation and permission configurations Customers want to test AWS IoT Greengrass in a diverse set of environments Run AWS IoT Greengrass with fewer dependencies and no kernel-level changes Access more local resources like Bluetooth Low Energy or USB devices Run AWS IoT Greengrass in a Docker container
  28. 28. Add AWS IoT Greengrass to existing architecture that uses Docker containers for isolation Existing applications running in separate Docker containers Application A Run AWS IoT Greengrass in a Docker container Application B On-Premise Device
  29. 29. Download AWS IoT Device Tester from AWS IoT Greengrass and Amazon FreeRTOS product pages AWS IoT Device Tester is a test automation tool that lets you test Amazon FreeRTOS or AWS IoT Greengrass on your choice of devices. AWS IoT Device Tester for Amazon FreeRTOS Tests if the Amazon FreeRTOS cloud connectivity, OTA, and security libraries function correctly on top of microcontroller board device drivers AWS IoT Device Tester for AWS IoT Greengrass Tests if the combination of device’s CPU architecture, Linux kernel configuration, and drivers work with AWS IoT Greengrass AWS IoT Device Tester
  30. 30. AWS IoT & Hardware partners https://devices.amazonaws.com/
  31. 31. IoT Well Architected and iotatlas.net http://iotatlas.net/ https://d1.awsstatic.com/whitepapers/architecture/AWS-IoT-Lens.pdf
  32. 32. AWS IoT Partner Network Edge Silicon OEM ODM/CM Connectivity Gateway Network/Carrier Solution ISV Regional SI Global SI We build IoT solutions through our partnersWe build IoT solutions through our partners https://aws.amazon.com/iot/partner-solutions/
  33. 33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark ISVs • Parking • Traffic • Logistics
  34. 34. IoT, Big Data and ML pipeline Connected vehicles with AWS Greengrass Core Staged data Raw data IoT Big Data ML
  35. 35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Paul Macey Specialist Solution Architect, Big Data & Analytics AWS Public Sector Immersion Day – Big Data IoT, Big Data & Machine Learning Specialist Session
  36. 36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Key topics Organisational data challenges Accelerated data lake (3 x 3 x 3) Demonstration Analytics and Insights Demonstration
  37. 37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “…building contemporary capabilities, including the public service winning the war for talent in critical areas like data, cloud, and engineering. WHAT DO YOU SEE AS THE SINGLE BIGGEST TECHNOLOGY CHALLENGE FOR AUSTRALIA’S PUBLIC SERVICE IN 2030? “Collaboration across departmental silos is a big challenge and is made more complicated by the compression in budgets.” Maile Carnegie – Dec 2018 Australian Public Sector Review Panel Member https://www.apsreview.gov.au/news/deep-sense-purpose
  38. 38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IoT to Data lake integration Connected vehicles with AWS Greengrass Core IoT Staged data Big Data connectedcar conncar_2019 month day minute connected car json data file dev-iot-bd-ml-staging
  39. 39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IoT, Big Data and ML pipeline Connected vehicles with AWS Greengrass Core Staged data Raw data IoT Big Data ML
  40. 40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Organisational Data Pain Silos Governance ? ScalabilitySecurity
  41. 41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Accelerated Data Lake (3 x 3 x 3) Security Day 0 Data governance & metadata Data centralised & scalable SQL & BI ready Analytical & Data Science foundation Repeatable & Extensible
  42. 42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data was in • Multiple databases • API’s • IoT feeds Across multiple departments Previously couldn’t bring data together Worked with our professional services team to deploy the AWS accelerated data lake and run workshops Built in 3 weeks Real use case - Connected Vehicle Customer
  43. 43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Education Health Utilities Essential services Accelerating deployments to other sectors
  44. 44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Available today @ GitHub https://github.com/aws-samples/accelerated-data-lake Includes Data lake pipeline (CloudFormation) Instructions Security and metadata templates Delivery 3x3x3 delivery method Professional services or AWS partners Accelerated Data Lake
  45. 45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The solution focusses on : • Securing data • Cataloging data • Staging data • Establishing a repeatable data onboarding pattern Accelerated Data Lake Overview The data lake solution is a subset of an entire data pipeline Process Security Metadata Data Lake Storage Initiation Database / BI Big Data Querying ETL & ML Data Lake Analytics & Insights
  46. 46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Accelerated Data Lake and Analytics High Level Data Flow Lambda Functions • Import • Validation • Apply Security • Attach Metadata • File movement • Alerts Time based or Event Driven S3 buckets • Staging • Raw • Curated • Gold • Data discovery • Logs ProcessInitiation Data Lake Storage Metadata / Search Big Data, Querying ETL & ML Database / BI Data Lake Analytics & Insights
  47. 47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demonstration – Data Lake
  48. 48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics and Insights ProcessInitiation Data Lake Storage Metadata / Search Big Data Querying, ETL & Insights Database / BI Raw data
  49. 49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demonstration – Analytics and BI
  50. 50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Bringing it all together Security Day 0 Data governance & metadata Data centralised & scalable SQL & BI ready Analytical & Data Science foundation Repeatable & Extensible The accelerated data lake solution Can enable your data Support data security and data governance Can grow and scale in harmony with your organisation Can be granted access to AWS’s analytics, ML and AI ecosystem
  51. 51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. References Amazon S3 security https://aws.amazon.com/s3/faqs/#Security https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html AWS Accelerated Data Lake (Git) https://github.com/aws-samples/accelerated-data-lake AWS Accelerated Data Lake Blog (part 1) https://aws.amazon.com/blogs/publicsector/from-data-silos-to-data-domains-bringing-common-data-together Our data lake story: How Woot.com built a serverless data lake on AWS https://aws.amazon.com/blogs/big-data/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws
  52. 52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark AWS | Machine Learning
  53. 53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark 1. Use case: Reduce severe road accidents by using transport data • Overview of how transport organisations can predict accidents • Use Amazon SageMaker to create a custom model to predict a car’s speed 2. Overview of AI and machine learning on AWS
  54. 54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
  55. 55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Determine patterns from data D ata Machi ne Learni ng P redi cti ons Connected Cars IoT Mobile phone logs Taxi logs Incidents Travel time Demand modelling
  56. 56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Live Alert System • Weather (BoM) • Car speed • Public transport events • Known events • Location (Waze, HERE.com) • Emergency Vehicle Locations • Speed limits • Police alerts Machine Learning + Data Sources WHEN and WHERE are the predicted dangers? Results
  57. 57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark 1 2 3
  58. 58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Amazon SageMaker Fast & accurate data labeling Built-in, high performance algorithms & notebooks B U I LD 1 One-click training and tuning TRAI N Model optimization 2 Fully managed hosting with auto-scaling and elastic inference One-click deployment D E P LOY 3
  59. 59. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Connected vehicles with AWS Greengrass Core Staged data Raw data IoT Big Data ML
  60. 60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark • Contain live code, equations, visualizations and explanatory text • Amazon SageMaker does the heavy lifting by hosting Jupyter Notebooks • Control: 1. Build 2. Train 3. Deploy
  61. 61. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
  62. 62. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
  63. 63. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Scale out Scale up
  64. 64. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
  65. 65. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Scale out Scale up
  66. 66. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Live Alert System • Weather (BoM) • Car speed • Public transport events • Known events • Location (Waze, HERE.com) • Emergency Vehicle Locations • Speed limits • Police alerts Machine Learning + Data Sources WHEN and WHERE are the predicted dangers? Results
  67. 67. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark 1. Use case: Reduce severe road accidents by using transport data • Overview of how transport organisations can predict accidents • Use Amazon SageMaker to create a custom model to predict a car’s speed 2. Overview of AI and machine learning on AWS
  68. 68. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ML @ AWS OUR MISSION Put machine learning in the hands of every developer
  69. 69. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Our unique approach Customer-focused 90%+ of our ML roadmap is defined by customers Breadth & depth More AI and ML services in production than any other provider Embedded R&D Customer-centric approach to advancing the state of the art Security & analytics Deepest set of security and encryption features, with robust analytics capabilities Multi-framework Support for the most popular frameworks Pace of innovation 200+ new ML launches in the last year
  70. 70. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
  71. 71. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark M L F R A M E W O R K S & I N F R A S T R U C T U R E A I S E R V I C E S R E K O G N I T I O N I M A G E P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N V I D E O Vision Speech Language Chatbots A M A Z O N S A G E M A K E R B U I L D T R A I N F O R E C A S T Forecasting T E X T R A C T P E R S O N A L I Z E Recommendations D E P L O Y Pre-built algorithms & notebooks Data labeling (G R O U N D T R U T H ) One-click model training & tuning Optimization (N E O ) One-click deployment & hosting M L S E R V I C E S F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e E C 2 P 3 & P 3 N E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C I N F E R E N C E Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E F O R M A C H I N E L E A R N I N G )
  72. 72. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark M L F R A M E W O R K S & I N F R A S T R U C T U R E A I S E R V I C E S R E K O G N I T I O N I M A G E P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N V I D E O Vision Speech Language Chatbots A M A Z O N S A G E M A K E R B U I L D T R A I N F O R E C A S T Forecasting T E X T R A C T P E R S O N A L I Z E Recommendations D E P L O Y Pre-built algorithms & notebooks Data labeling (G R O U N D T R U T H ) One-click model training & tuning Optimization (N E O ) One-click deployment & hosting M L S E R V I C E S F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e E C 2 P 3 & P 3 N E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C I N F E R E N C E Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E F O R M A C H I N E L E A R N I N G )
  73. 73. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark A I S E R V I C E S R E K O G N I T I O N I M A G E P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N V I D E O Vision Speech Language Chatbots F O R E C A S T Forecasting T E X T R A C T P E R S O N A L I Z E Recommendations
  74. 74. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Enhancing the fan experience One week of NFL games now creates 3TB of data. NFL uses Amazon SageMaker to analyze telemetry data to predict plays. Computations that could take months to refine now take only weeks or days. WATCH VIDEO >>
  75. 75. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark • Build machine learning models in Amazon SageMaker • Train, test, and iterate on the track using the AWS DeepRacer 3D racing simulator • Compete in the world’s first global autonomous racing league, to race for prizes and a chance to advance to win the coveted AWS DeepRacer Cup A fully autonomous 1/18th-scale race car designed to help you learn about reinforcement learning through autonomous driving
  76. 76. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark • Purpose-built for ML-skills development • Fully programmable & customizable • Build custom Amazon SageMaker models • 10-minutes to your first deep learning project The world’s first deep learning-enabled video camera for developers

×