SlideShare a Scribd company logo
1 of 59
Download to read offline
NORDICS
Event-driven and serverless
in the world of IoT
JIMMY DAHLQVIST | 2023-04-20
@jimmydahlqvist
Todays talk
• Introduction to AWS IoT
• Sample cloud architecture
• Going event-driven and serverless
• Takeaways and thought
@jimmydahlqvist
Hello, I'm
JIMMY DAHLQVIST
Father of two girls
Serverless enthusiast
AWS Ambassador
AWS Community Builder
Head of AWS Technologies at Sigma Technology Cloud
IoT
@jimmydahlqvist
Concept
• A network of physical device
• Collection of data from connected devices
• Wide range of use cases
• Rapidly growing, more and more devices being connected
• IIoT – Industry 4.0
@jimmydahlqvist
Scenario
• Thousands to millions of devices
• Unpredicatble traffic and varying load
• High speed real time data
• Asynchronous data processing
AWS IoT
@jimmydahlqvist
AWS IoT Core
• Managed service to easily connect, register, and manage devices
• Robust security, authentication, encryption, and access control
• Scales to millions of devices
• Support standard protocol, MQTT and HTTP
• Event-based architecture
@jimmydahlqvist
IoT Core and X.509 Certificates
• Used to authenticate and secure connections
• Certificate base mutual authentication
• Unique certificate per device
• Policy based authorization, based on identities
• Built in Certificate Authority (CA)
• Support custom CA
@jimmydahlqvist
MQTT Broker and Topics
• Integrates with several AWS services
• Powerful rules engine
• Topics are used to route messages
• Hierarchical
• sensors/{sensor_id}/temperature
First Second Third
@jimmydahlqvist
IoT Things
• Representation of physical or virtual device
• Can be assigned a Thing Type
• Publish / Subscribe to MQTT Topics
@jimmydahlqvist
Thing registration
• Device certificates bundle during manufacturing
• Provision a thing Just-In-Time
@jimmydahlqvist
Just-In-Time Provisioning
• Require custom CA
• Attach provisioning template to CA cert
• Use pre-provisioning hooks
@jimmydahlqvist
Just-in-time Provisioning
@jimmydahlqvist
IoT Policy
• Define permissions and access
• Attached to Things, Thing Types, or AWS IoT resources
• Support variables from Thing and Certificate
@jimmydahlqvist
IoT Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iot:Publish",
"Resource":
"arn:aws:iot:<region>:<account>:topic/${aws:iot:ThingName}"
}
]
}
@jimmydahlqvist
IoT Iam credentials endpoint
• HTTP based
• Exchange x.509 certificate for IAM credentials
• Direct access to AWS Services
@jimmydahlqvist
Sending Data
• Device send data on topic
• Ingress/{Thing-Name}/data
• Topic-based routing
@jimmydahlqvist
Reciving Data
• Topic-based
• Egress/{Thing-Name}/data
• Device subcribe for data
@jimmydahlqvist
MQTT based API
AWS Step Functions workflow
Cloud Architecture
@jimmydahlqvist
Cloud architecture
@jimmydahlqvist
Lesson learned
• IoT Rules as router
• Small objects in S3
• Hard to change data format
• Hard to maintain and extend
Going Event-driven
@jimmydahlqvist
Theory
• Remove IoT Rules as event router
• Improve extensibility
• Introduce Amazon EventBridge
@jimmydahlqvist
Event
A signal to indicate that
something has
changed or happened.
• Immutable
• Observable
• Facts
• JSON
@jimmydahlqvist
Amazon EventBridge
• Fully managed and serverless
• 15+ AWS services as target
• API destinations
• EventBridge to EventBridge
• Easy to build event-driven architecture
• Low price
@jimmydahlqvist
Cloud architecture
@jimmydahlqvist
Data storage
@jimmydahlqvist
Data storage
Service C
Service B
Service A
Data Transformation Service
Analytics Service
Service C
Service B
Service A
Data Transformation Service
Analytics Service
@jimmydahlqvist
Data Transformation Service
• Data transform pattern
• Decouple data format from device
Service C
Service B
Service A
Data Transformation Service
Analytics Service
@jimmydahlqvist
Serverless and event-driven
Loosely coupled
Scale and fail independently
Cost effective
Extensibility
Highly available
Architecture Pattern
@jimmydahlqvist
• Reliably capture data
• Managed services
• Powerful
Storage first
Service C
Service B
Service A
Data Transformation Service
Analytics Service
Takeaways and thoughts
@jimmydahlqvist
AWS IoT Core Basic Ingest
• Uses reserved topic
• $aws/rules/rule_name
• No additional message cost
• $1.00 (per million messages)
@jimmydahlqvist
Transform don’t Transport
Use AWS Lambda to transform data, not transport.
@jimmydahlqvist
Be kind
• Don’t overwhelm downstream services
• Use patterns like Circuit Breaker
@jimmydahlqvist
AWS StepFunctions
@jimmydahlqvist
SDK integrations
• 200+ SDK integrations
• Call a wide range of services using 9000+ API actions.
• Low code / no code
@jimmydahlqvist
Express workflow
• Short running – 5 minutes
• 100K starts per second
• Pay for duration
@jimmydahlqvist
Express workflows
Express Workflows employ an at-least-once model for asynchronous invocation
@jimmydahlqvist
Amazon EventBridge
@jimmydahlqvist
Create subscriptions
Creates coupling on the filter
Complex to add more targets
Changing the filter affect all
Coupling is on the event
Easy to add more targets
Filters can change independently
@jimmydahlqvist
Default Event Bus
One per region and AWS account
Avoid using the default bus for custom applications!
@jimmydahlqvist
Event bus patterns
Single Bus (Centralized)
Multi Bus (Decentralized)
@jimmydahlqvist
Centralized
Advantages
• Centralized routing and resource management
• Easy to integrate new application
Disadvantages
• Single point of failure
• Leaking business logic
@jimmydahlqvist
Decentralized (Distributed)
Advantages
• No single point of failure
• Enables autonomy
Disadvantages
• Challenge to design
• More resources to manage
@jimmydahlqvist
Single bus pattern
• Good place to start
@jimmydahlqvist
Data storage
• Use multiple data stores
• Timestream is great for analytic queries
• DynamoDB is great for quick dashboards
Has it been battle tested?
@jimmydahlqvist
I would say so!
• Several production environments
• Thousands and thousands of devices
• Hundreds of messages per second
• Vast amount of data analysed
@jimmydahlqvist
And……..
@jimmydahlqvist
Great BBQ!
NORDICS
Thank you!

More Related Content

Similar to Event-driven and serverless in the world of IoT

AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"
AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"
AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"Chris Munns
 
Getting Started with Windows Workloads on Amazon EC2 - Toronto
 Getting Started with Windows Workloads on Amazon EC2 - Toronto Getting Started with Windows Workloads on Amazon EC2 - Toronto
Getting Started with Windows Workloads on Amazon EC2 - TorontoAmazon Web Services
 
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스Amazon Web Services Korea
 
Slide share device to iot solution – a blueprint
Slide share   device to iot solution – a blueprintSlide share   device to iot solution – a blueprint
Slide share device to iot solution – a blueprintGuy Vinograd ☁
 
Virtualization and cloud computing
Virtualization and cloud computingVirtualization and cloud computing
Virtualization and cloud computingDeep Gupta
 
XCloudLabs- AWS Overview
XCloudLabs- AWS Overview XCloudLabs- AWS Overview
XCloudLabs- AWS Overview sangam biradar
 
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...Amazon Web Services
 
Big data journey to the cloud 5.30.18 asher bartch
Big data journey to the cloud 5.30.18   asher bartchBig data journey to the cloud 5.30.18   asher bartch
Big data journey to the cloud 5.30.18 asher bartchCloudera, Inc.
 
Full Stack Application Monitoring for AWS Powered by AI
Full Stack Application Monitoring for AWS Powered by AIFull Stack Application Monitoring for AWS Powered by AI
Full Stack Application Monitoring for AWS Powered by AIDynatrace
 
AWS August Webinar Series - Services Overview
AWS August Webinar Series - Services Overview AWS August Webinar Series - Services Overview
AWS August Webinar Series - Services Overview Amazon Web Services
 
Microservices and serverless for MegaStartups - DLD TLV 2017
Microservices and serverless for MegaStartups - DLD TLV 2017Microservices and serverless for MegaStartups - DLD TLV 2017
Microservices and serverless for MegaStartups - DLD TLV 2017Boaz Ziniman
 
Amazon AWS vs Azure Cloud vs Kubernetes
Amazon AWS vs Azure Cloud vs KubernetesAmazon AWS vs Azure Cloud vs Kubernetes
Amazon AWS vs Azure Cloud vs KubernetesStridely Solutions
 
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...SnapLogic
 
Trust No-One Architecture For Services And Data
Trust No-One Architecture For Services And DataTrust No-One Architecture For Services And Data
Trust No-One Architecture For Services And DataAidan Finn
 
AWS IoT: colmare il divario tra il mondo fisico e quello digitale
AWS IoT: colmare il divario tra il mondo fisico e quello digitaleAWS IoT: colmare il divario tra il mondo fisico e quello digitale
AWS IoT: colmare il divario tra il mondo fisico e quello digitaleAmazon Web Services
 
Hybrid IT with Amazon Web Services: Best of Both Worlds
Hybrid IT with Amazon Web Services: Best of Both WorldsHybrid IT with Amazon Web Services: Best of Both Worlds
Hybrid IT with Amazon Web Services: Best of Both WorldsAmazon Web Services
 
AWS Finland User Group Meetup 2017-05-23
AWS Finland User Group Meetup 2017-05-23AWS Finland User Group Meetup 2017-05-23
AWS Finland User Group Meetup 2017-05-23Rolf Koski
 
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸Amazon Web Services
 

Similar to Event-driven and serverless in the world of IoT (20)

AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"
AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"
AWS NYC Meetup - May 2017 - "AWS IoT and Greengrass"
 
Getting Started with Windows Workloads on Amazon EC2 - Toronto
 Getting Started with Windows Workloads on Amazon EC2 - Toronto Getting Started with Windows Workloads on Amazon EC2 - Toronto
Getting Started with Windows Workloads on Amazon EC2 - Toronto
 
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스
[AWS에서의 미디어 및 엔터테인먼트] 클라우드에서의 브로드캐스팅 서비스
 
Slide share device to iot solution – a blueprint
Slide share   device to iot solution – a blueprintSlide share   device to iot solution – a blueprint
Slide share device to iot solution – a blueprint
 
Virtualization and cloud computing
Virtualization and cloud computingVirtualization and cloud computing
Virtualization and cloud computing
 
XCloudLabs- AWS Overview
XCloudLabs- AWS Overview XCloudLabs- AWS Overview
XCloudLabs- AWS Overview
 
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...
AWS re:Invent 2016: Understanding IoT Data: How to Leverage Amazon Kinesis in...
 
Big data journey to the cloud 5.30.18 asher bartch
Big data journey to the cloud 5.30.18   asher bartchBig data journey to the cloud 5.30.18   asher bartch
Big data journey to the cloud 5.30.18 asher bartch
 
Full Stack Application Monitoring for AWS Powered by AI
Full Stack Application Monitoring for AWS Powered by AIFull Stack Application Monitoring for AWS Powered by AI
Full Stack Application Monitoring for AWS Powered by AI
 
AWS August Webinar Series - Services Overview
AWS August Webinar Series - Services Overview AWS August Webinar Series - Services Overview
AWS August Webinar Series - Services Overview
 
cloud_ch1.pptx
cloud_ch1.pptxcloud_ch1.pptx
cloud_ch1.pptx
 
Microservices and serverless for MegaStartups - DLD TLV 2017
Microservices and serverless for MegaStartups - DLD TLV 2017Microservices and serverless for MegaStartups - DLD TLV 2017
Microservices and serverless for MegaStartups - DLD TLV 2017
 
Amazon AWS vs Azure Cloud vs Kubernetes
Amazon AWS vs Azure Cloud vs KubernetesAmazon AWS vs Azure Cloud vs Kubernetes
Amazon AWS vs Azure Cloud vs Kubernetes
 
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...
 
Trust No-One Architecture For Services And Data
Trust No-One Architecture For Services And DataTrust No-One Architecture For Services And Data
Trust No-One Architecture For Services And Data
 
AWS IoT: colmare il divario tra il mondo fisico e quello digitale
AWS IoT: colmare il divario tra il mondo fisico e quello digitaleAWS IoT: colmare il divario tra il mondo fisico e quello digitale
AWS IoT: colmare il divario tra il mondo fisico e quello digitale
 
Hybrid IT with Amazon Web Services: Best of Both Worlds
Hybrid IT with Amazon Web Services: Best of Both WorldsHybrid IT with Amazon Web Services: Best of Both Worlds
Hybrid IT with Amazon Web Services: Best of Both Worlds
 
AWS Finland User Group Meetup 2017-05-23
AWS Finland User Group Meetup 2017-05-23AWS Finland User Group Meetup 2017-05-23
AWS Finland User Group Meetup 2017-05-23
 
AWS_AIoT_Blockchain_in_IoT
AWS_AIoT_Blockchain_in_IoTAWS_AIoT_Blockchain_in_IoT
AWS_AIoT_Blockchain_in_IoT
 
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸
透過Amazon CloudFront 和AWS WAF來執行安全的內容傳輸
 

More from Jimmy Dahlqvist

Building a serverless AI powered translation service
Building a serverless AI powered translation serviceBuilding a serverless AI powered translation service
Building a serverless AI powered translation serviceJimmy Dahlqvist
 
Serverless cloud architecture patterns
Serverless cloud architecture patternsServerless cloud architecture patterns
Serverless cloud architecture patternsJimmy Dahlqvist
 
AI Powered event-driven translation bot
AI Powered event-driven translation botAI Powered event-driven translation bot
AI Powered event-driven translation botJimmy Dahlqvist
 
Building a serverless event driven Slack Bot
Building a serverless event driven Slack BotBuilding a serverless event driven Slack Bot
Building a serverless event driven Slack BotJimmy Dahlqvist
 
IoT enable smoker for great BBQ
IoT enable smoker  for great BBQIoT enable smoker  for great BBQ
IoT enable smoker for great BBQJimmy Dahlqvist
 
Autoscaled Github Runners using StepFunctions
Autoscaled Github Runners using StepFunctionsAutoscaled Github Runners using StepFunctions
Autoscaled Github Runners using StepFunctionsJimmy Dahlqvist
 
EventBridge Patterns and real world use case
EventBridge Patterns and real world use caseEventBridge Patterns and real world use case
EventBridge Patterns and real world use caseJimmy Dahlqvist
 
re:Invent Recap Breakfast
re:Invent Recap Breakfastre:Invent Recap Breakfast
re:Invent Recap BreakfastJimmy Dahlqvist
 
CI/CD As first and last line of defence
CI/CD As first and last line of defenceCI/CD As first and last line of defence
CI/CD As first and last line of defenceJimmy Dahlqvist
 
Introduction to testing in Cloud / AWS
Introduction to testing in Cloud / AWSIntroduction to testing in Cloud / AWS
Introduction to testing in Cloud / AWSJimmy Dahlqvist
 
CHAOS ENGINEERING – OR LET'S SHAKE THE TREE
CHAOS ENGINEERING – OR LET'S SHAKE THE TREECHAOS ENGINEERING – OR LET'S SHAKE THE TREE
CHAOS ENGINEERING – OR LET'S SHAKE THE TREEJimmy Dahlqvist
 
Road to an asynchronous device registration API
Road to an asynchronous device registration APIRoad to an asynchronous device registration API
Road to an asynchronous device registration APIJimmy Dahlqvist
 
GitOps in action, powered by Alexa.
GitOps in action, powered by Alexa.GitOps in action, powered by Alexa.
GitOps in action, powered by Alexa.Jimmy Dahlqvist
 

More from Jimmy Dahlqvist (13)

Building a serverless AI powered translation service
Building a serverless AI powered translation serviceBuilding a serverless AI powered translation service
Building a serverless AI powered translation service
 
Serverless cloud architecture patterns
Serverless cloud architecture patternsServerless cloud architecture patterns
Serverless cloud architecture patterns
 
AI Powered event-driven translation bot
AI Powered event-driven translation botAI Powered event-driven translation bot
AI Powered event-driven translation bot
 
Building a serverless event driven Slack Bot
Building a serverless event driven Slack BotBuilding a serverless event driven Slack Bot
Building a serverless event driven Slack Bot
 
IoT enable smoker for great BBQ
IoT enable smoker  for great BBQIoT enable smoker  for great BBQ
IoT enable smoker for great BBQ
 
Autoscaled Github Runners using StepFunctions
Autoscaled Github Runners using StepFunctionsAutoscaled Github Runners using StepFunctions
Autoscaled Github Runners using StepFunctions
 
EventBridge Patterns and real world use case
EventBridge Patterns and real world use caseEventBridge Patterns and real world use case
EventBridge Patterns and real world use case
 
re:Invent Recap Breakfast
re:Invent Recap Breakfastre:Invent Recap Breakfast
re:Invent Recap Breakfast
 
CI/CD As first and last line of defence
CI/CD As first and last line of defenceCI/CD As first and last line of defence
CI/CD As first and last line of defence
 
Introduction to testing in Cloud / AWS
Introduction to testing in Cloud / AWSIntroduction to testing in Cloud / AWS
Introduction to testing in Cloud / AWS
 
CHAOS ENGINEERING – OR LET'S SHAKE THE TREE
CHAOS ENGINEERING – OR LET'S SHAKE THE TREECHAOS ENGINEERING – OR LET'S SHAKE THE TREE
CHAOS ENGINEERING – OR LET'S SHAKE THE TREE
 
Road to an asynchronous device registration API
Road to an asynchronous device registration APIRoad to an asynchronous device registration API
Road to an asynchronous device registration API
 
GitOps in action, powered by Alexa.
GitOps in action, powered by Alexa.GitOps in action, powered by Alexa.
GitOps in action, powered by Alexa.
 

Recently uploaded

Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonApplitools
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingShane Coughlan
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfMarharyta Nedzelska
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Natan Silnitsky
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
SoftTeco - Software Development Company Profile
SoftTeco - Software Development Company ProfileSoftTeco - Software Development Company Profile
SoftTeco - Software Development Company Profileakrivarotava
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxRTS corp
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesAmazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesKrzysztofKkol1
 

Recently uploaded (20)

Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + KobitonLeveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
Leveraging AI for Mobile App Testing on Real Devices | Applitools + Kobiton
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full RecordingOpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
OpenChain Education Work Group Monthly Meeting - 2024-04-10 - Full Recording
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
A healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdfA healthy diet for your Java application Devoxx France.pdf
A healthy diet for your Java application Devoxx France.pdf
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
SoftTeco - Software Development Company Profile
SoftTeco - Software Development Company ProfileSoftTeco - Software Development Company Profile
SoftTeco - Software Development Company Profile
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptxThe Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
The Role of IoT and Sensor Technology in Cargo Cloud Solutions.pptx
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilitiesAmazon Bedrock in Action - presentation of the Bedrock's capabilities
Amazon Bedrock in Action - presentation of the Bedrock's capabilities
 

Event-driven and serverless in the world of IoT

Editor's Notes

  1. Smart homes, manufacturing, smart buildings, door bells, smart cities? ....
  2. Lightbulbs, Locks, Sensors for temp and humidity, Security cameras, connected Doors, Water usage sensors, Gas usage The list continues 100-rds of devices, 1000, 10.0000, Millions The scale is unlimited Iot Systems come in all shapes and forms and our architecture must be able to meet the demand when is come to scalability and cost. The number of IoT systems grows for every day and the IoT space is one of the most growing areas at the moment. Today we will take a look at Iot and one of several cloud architectures I have built over the year. The challenges we phased and what changes we did.
  3. Today we will look into 4 areas. Starting of with an introduction to IoT and AWS IoT services We’ll take a look at a sample architecture, the challenges we phased What changes was made when we decided to go event-driven and fully serverless And finally I will leae you with some takeaways and thoughts.
  4. Before we start, who am I? My name is Jimmy Dahlqvist!! Father of two daughter, afraid of nothing!! IoT, Serverless, event-driven fantast Serverless since 2016 – lambda Old AWS Ambassador + Community Builder (cheer ??) Day Job – Head of sigma – (cheer) collogues in audience
  5. IoT…. Internet of things. IIOT…. Industrial internet of things, also called industry 4.0 What is it? What is a thing? How do we connect them? What are the purpose of them all? How do we build a efficient cloud architecture it? Hopefully I can answer some of those questions today.
  6. network of physical devices, vehicles, home appliances, and other items embedded with electronics, software, sensors, and connectivity, which enables these objects to connect and exchange data. IoT enables businesses and organizations to collect data from a vast network of connected devices, analyse that data, and use the insights gained to optimize operations, reduce costs, and improve customer experiences. IoT technology is rapidly growing, with more and more devices being connected to the internet every day. IoT has a wide range of use cases, from simple home automation to complex industrial automation systems. IIOT…. Industrial internet of things, industry 4.0 IIoT enables machines, sensors, and other devices to collect and exchange data, and can be used to improve operational efficiency, reduce downtime, and enable predictive maintenance. IIoT can also help businesses to optimize their supply chain management, reduce waste, and improve product quality.
  7. Through out this talk this is the scanario we’ll keep in mind. This is what we’ll try to build our architecture and IoT system for.
  8. With that in mind, now let’s move over and talk about different components of AWS IoT. There are so many different services in this collection so our focus will be on AWS IoT Core and it’s components. Except core there are services like Greengrass that you run on your device, Device Defender for security and protection of devices, Fleetwise to manage and on-board vast amount of devices.
  9. managed cloud service that makes it easy to connect, register, and manage IoT devices at scale. organizations can easily connect a wide range of devices, including sensors, gateways, and appliances, to the cloud and securely transmit data to and from them. key features of AWS IoT Core is its robust security capabilities. It provides authentication, encryption, and access control to help ensure that only authorized devices and users can access the system and the data transmitted over it. AWS IoT Core is highly scalable and can support millions of devices, making it ideal for large-scale IoT deployments. AWS IoT Core supports standard protocols such as MQTT (Message Queuing Telemetry Transport) and HTTP. The event-based architecture of AWS IoT Core enables organizations to build complex IoT applications that respond to real-time data and events. This architecture supports the use of rules and actions to automate actions based on incoming data, which can help organizations to respond quickly to changing conditions and improve operational efficiency.
  10. AWS IoT Core uses X.509 certificates to authenticate and secure connections between IoT devices and the cloud. X.509 certificates are digital certificates that use a standard format to encode public key information, identity information, and digital signatures. X.509 certificates are widely used for authentication and encryption in many security protocols, such as SSL/TLS, IPSec, and S/MIME. In AWS IoT Core, X.509 certificates are used for mutual authentication, which means that both the device and the cloud need to authenticate each other using a unique certificate. Each device is assigned a unique X.509 certificate, which is used to authenticate and authorize its connections to the cloud. The device's certificate is signed by a built-in Certificate Authority (CA) in IoT Core, which ensures the authenticity and integrity of the certificate. IoT Core also supports custom CAs, which enable customers to use their own certificate infrastructure to issue and manage device certificates. In addition to authentication, IoT Core uses X.509 certificates for authorization, which means that access to resources and services is based on the policies associated with the device's identity. Policies in IoT Core define the permissions and restrictions for each device, based on its identity and attributes. Policies can be based on the device's certificate, its Thing Type, its attributes, or other contextual information. X.509 certificates and policies enable a secure and scalable IoT ecosystem, where devices and cloud services can communicate with each other in a trusted and controlled way. In summary, AWS IoT Core uses X.509 certificates for mutual authentication and authorization of IoT devices. Each device has a unique certificate, which is signed by a built-in or custom CA. Policies define the permissions and restrictions for each device, based on its identity and attributes. X.509 certificates and policies enable a secure and scalable IoT ecosystem.
  11. MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol that is widely used in IoT applications MQTT broker is a server that acts as a central hub for message exchange between devices. The broker receives messages published by devices and routes them to the appropriate subscribers. key features of MQTT is its use of topics to route messages between devices MQTT topics are hierarchical, For example, a topic like "sensors/{sensor_id}/temperature" AWS IoT integrates with several AWS services, including Amazon S3, Amazon Kinesis, and Amazon DynamoDB AWS IoT also includes a powerful rules engine that allows organizations to define rules and actions based on incoming data from IoT devices. In summary, MQTT is a lightweight messaging protocol that enables efficient and reliable communication between devices and the cloud. The use of topics and a hierarchical structure enables devices to subscribe to specific messages based on their interests. AWS IoT integrates with several AWS services and includes a powerful rules engine that enables organizations to automate workflows and respond quickly to changing conditions.
  12. What is an IoT Thing? IoT Things represent physical or virtual devices that can communicate with other devices and cloud services over the internet. Things can be assigned a Thing Type, which defines their attributes and behaviors. Things can publish and subscribe to MQTT topics to send and receive data. IoT Things can also interact with other AWS services to enable storage, analytics, and processing of IoT data.
  13. Before we can use a Thing and receive data from it, it must be registered in IoT Core. The unique device certificates are provisioned and flashed onto the device during manufacturing. However we don’t want it to be registered directly. Instead we like to provision our devices as they connect, just-in-time.
  14. What do we need for JIT? It required the use of our own custom CA, we can’t use the AWS built in CA as these certs are created and feteched from IoT Core. For this CA we attach what is know as a provision template that defines attributes and actions that should happen during the provisioning process. As an extra security measure it’s good pratcice to use pr-provisioning hooks to validate the cert data and things.
  15. So a step by step illustration for the provisioning process. The thing connects to IoT Core. The CA is recognized and the template is used to start the prcess. We invoke our lambda hook that can validate our thing and return OK/ NOK back. We create our policy, register the certificate but doesn’t enable it, and create the Iot thing. Next the policy is attached to the cert and the cert is attached to the thing. Finally we enable the cert.
  16. An IoT Core policy is a set of rules and permissions that defines how IoT devices and other entities can interact with AWS IoT Core resources and services. Policies in IoT Core are based on identities, which are represented by X.509 certificates Policies can grant or deny access to MQTT topics Policies can be attached to things, thing types, principals, or groups. Policy language support variables from Thing and Certificate
  17. Here is an example policy that would allow a device to publish to a topic that end with it’s ThingName. By using simiiar conditions we can restrict topics that a device can subscribe to, this way we can enforce that devices can only subscribe or publish to topics that belomng to that device.
  18. Sometimes devices need to interact with AWS services directly, that could be a video camera sending video to Kinesis video stream or a device uploadning data to S3. The device can then use the Iam credentials endpoint and exchange the x.509 certificate for temporary iam credentials. This is similar to IAM Roles Anywhere, but this functionality has been around for longer time.
  19. When devices send data we use topic based routing. IoT rules are created and associated with an topic, the rule can the contain logic and conditions and if these are met the rule and associated targets are invoked.
  20. Receiving data work abou the same but there are no rules that are invoked. A device creates a subscription on a topic and when messages arrive on that topic the message is sent to the device
  21. With this send / subscribe functionality we can build an MQTT based API where devices send api actions on one topic and then expects an answer to be published to a specific topic. This is a very powerful way for devices to interact and perform actions on resources in the cloud. For example, a device could send an API action that is want to upload data to S3. Instead of using IAM credentials endpoint and implement logic in the device, an MQTT api could respond with an pre-signed url that the device can upload to.
  22. So that was the basics….. Now let’s jumo into and start building our Cloud architecture. We will start out with a fairly basic design, from a real world use case, and we’ll iterate over that design to improve it over time, where we finally arrive at out final version.
  23. First version of cloud looked like this… Device data -> IoT Core IoT Core -> Rules -> Storage + Athena + Dynamo IoT Core -> Rules -> Several SF business logic (thresholds, trends…) API GW RESTful…
  24. IoT rules as router –> on mqtt –> not on payload –> messages discarded in business logic Each event –> one OBJECT in s3 –> Glue/Athena not optimized for that Data written directly to storage (Storage First is good but…) –> Format dictated by the device –> need transform Hard to extend –> Several services did same thing –> notification –> or needed to implement API
  25. First improvement to this is to move to a more event-driven system. Even if Iot Core can be used to build out an event-driven system I often found it not to be optimal.
  26. The theory I had that was going to improve the architecture was: Remove IoT Rules as event router Improve extensibility Introduce Amazon EventBridge Rules Engine pricing Rules initiated: $0.15 (per million rules triggered / per million actions applied) Actions applied: $0.15 (per million rules triggered / per million actions applied)
  27. But before we do any changes, just let us look at what an event can be defined as
  28. In the Theory the plan was to use EB. So why use EB then and not IoT Core? Well it come with several good things and some differences Main difference is that EB-2-EB, services can subscibe and publish, not only devices., easy to extend
  29. So with that improvement we removed the Topic routing and instead send all messages to an SQS queue. The queue invokes a Lambda function that send the event / message to EB. All services can then listen for the event and react to that. BUT…..
  30. Still not solved the S3 part…. Needed a way for the raw untouched data to be stored for later use.
  31. So instead of SQS Kinesis was used as the service bewteen IoT and EB That way Firehose can be set as consumer on kinesis as well and send data in batches to S3.
  32. If we then expand on that simplified architecture to include more services for different purpose. We have expanded on the analytics service, not only does it store data in S3, it also send data to an OpenSearch cluster that can be used for easy visualization. QuickSight could be used with Athena to create advanced business dashboards I have also introduced one new key services….. CLICK!
  33. This data transformation service. Why? Becasuse this makes it easy to use different data formats from thde devices but all internal services use an different format. We can also augment the data and add more information in the event that goes to EB. That mean that this service pick upp all data events from IoT, transform to an internal format, repost it onto EB and all internal service uses this.
  34. Of course the service is using StepFunction and in that as many SDK actions as possible. Why write code if you don’t need to right ? In this small example we….. And if you have seen my BBQ talk before you might recognize this 
  35. After a couple of iteration this is the architecture I’m using…. It has a couple of advantages
  36. Reason You build an serverless and eventdriven architecture. Loosely coupled services Scale and fail independently Cost effective – pay for what you use Extensibility – easy and fast to extend HA – built in
  37. Let’s take a closer look at architecture pattern that I tend to use all the time. Actually even used it in my very first architecture in 2016. That time we integrates API Gateway directly with SQS: And that of course is: CLICK! -> Animate Create reliable way to capture data – prevent data loss Use managed services Very powerful when incoming data doesn’t require instant transformation
  38. That has been a lot of information… Now I like to leave you with some take-aways and thoughts
  39. MQTT and HTTP messaging pricing Up to 1 billion messages: $1.00 (per million messages) Rules Engine pricing Rules initiated: $0.15 (per million rules triggered / per million actions applied) Actions applied: $0.15 (per million rules triggered / per million actions applied)
  40. EventBridge Choreographs Four bounded contexts represented by each service (orchestrated) Stepfunctions The service has unique business logic that need to be implemented and happen in a certain order, when a event is invoked.
  41. Don’t use lambda to transport data, it should be used to transform data. Before there were no other option, unfortunately. But nw we have EventBridge pipe…… So the next change in this will be to remove the Lambda function and move over to EB Pipes
  42. Be kind to services around you. In an event-driven and serverless system you can scale out massivly. IF you call downstream services, event 3rd party, try and not overwhelm them. If they fail don’t retry over and over agin. Use exponantial backof and patterns like circuit breaker
  43. Some tjoughts on StepFunctions
  44. I use SDK integration as much as possble…. As I said, why write code. But choose carefully
  45. Be aware of the different workflow types, how they behave, and how you pay for them Selecting the wrong one can become expensive And also…. Remember that!
  46. StepFunctions Express Workflow has an invocation model of “At least once” which means that it’s possible that your workflow get invoked twice. In dev and test it’s unlikely that it will happen, even in my small project I never saw it. But if you run it in a large enough scale it will happen. So make sure your workload is idempotent and can handle it. We learned this the hard way….. With data being written twice. In the normal small case you will not see this, but when you run millions of events it will happen
  47. When building on EventBridge I would recommend that you create subscriptions. With that I mean that you create one event rule for one target. Even if EB support up to 5 targets per rule I still say this should be a 1-2-1 mapping. Why? If you create one rule with multiple targets you will create a coupling on the event filter. And if you hit the 5 target limit what are you suppose to do then? Create a second rule with the same filter and start adding new targets? What if you need to update the filter? You will impact several targets that might not be what you wanted to do in the first place, leaving you to start breaking things apart again. So instead we create subscriptions where we create the coupling on the event it self, and we set one target for one rule. It’s easy to add more target, just create a second rule and add the target to it. And the filter in every rule can change without affecting any other target. However…. This of course can lead to several rules having the same filter and that could create problems on it’s own. But in my opinion it’s still better to create subscriptions and deal with a rule explosion.
  48. there is a default EB bus in the region already existing that AWS services post events to. Should you use that as your bus in your application? My recommendation is NO. Leave that bus to be used by AWS services and create your own custom buses for you application. The reason for that is that in that case will become easier to extend and ass busses later. I have seen the default bus being used and the mess it was when then moving to a custom bus. So can’t repeat it enough… Leave the default bus alone! Create custom buses! It’s like using the default VPC…. We don’t do that either!
  49. Talk about 2 types of patterns, centralized and de-centralized (Single / multi Bus) Single and Multi Account All the patterns allow us to decouple the publisher from the subscriber. The service that publish doesn’t really need to know who is listening in the other end.
  50. Look at centralized advantages and disadvantages Advantages allow you to manage all routing, security and policies in one place, single deployment) All routing centralized, concentrating all communication to a single event bus ( Enables central management of resources Allows you to easy integrate applications with few changes. Disadvantages As number of integrations grow so does the complexity and resource utilization. Can become a Single point of failure. All routing is centralized…. Prevents autonomy Single point of failure
  51. In a decentralized approach routing is spread across multiple event buses and the publisher often becomes the logical owner of that bus. The service owns the mechanism to distribute the events. Even if more buses are more work from a operational approach it enables autonomy and doesn’t become a single point of failure On the other hand, designing distributed systems, managing all resources, can become a huge challenge if not done properly from the start. Applying this as an afterthought is almost impossible. So the time to get started might be longer, integration of new services and applications require more change and take more time.
  52. Characteristics……
  53. Not to forget!! It powers the most amazing system in the whole world!!
  54. My Connected Smoker for great BBQ!!