Recommended
More Related Content
Similar to Cloud-grilled delights a high-tech approach to perfect BBQ
Similar to Cloud-grilled delights a high-tech approach to perfect BBQ (20)
More from Jimmy Dahlqvist
More from Jimmy Dahlqvist (10)
Recently uploaded
Recently uploaded (20)
Cloud-grilled delights a high-tech approach to perfect BBQ
- 1. Cloud-grilled delights a high- tech approach to perfect BBQ JIMMY DAHLQVIST | Believe In Serverless 2024-05- 16
- 2. @jimmydahlqvist JIMMY DAHLQVIST Head of AWS @ Sigma Technology Cloud Founder of serverless-handbook.com AWS Community Builder | AWS Ambassador | User Group Leader § Hello, I'm
- 3. @jimmydahlqvist Agenda • Background / Problem • Architecture overview • Cloud deep dive • Summary
- 5. @jimmydahlqvist Problem Multi-probe thermometer Wifi Connectivity High / Low thresholds Temperature trends / Stall History
- 6. @jimmydahlqvist BBQ Low and slow • Low even temperature for a long time It’s an art form
- 7. @jimmydahlqvist Smoker types Ceramic – Kamado UDS – Ugly drum smoker Electric pellets Offset
- 9. IoT device – v0.1
- 12. @jimmydahlqvist IoT Device Software AWS IoT Greengrass 2.0 - Core Custom component Hardware Raspberry Pi 4 model B 2.5mm food probes MCP3008 – AD converter
- 13. @jimmydahlqvist IoT device software First iteration Python application Updated over SSH Connected directly to AWS IoT Core
- 16. @jimmydahlqvist Lesson learned • IoT Rules as router • Small objects in S3 • Hard to change data format • Extending was a challenge
- 17. @jimmydahlqvist Improvements Cloud architecture • Data transformation • Event driven architecture • Decouple services IoT Device • Software development lifecycle • Log handling • Configuration
- 18. IoT Device
- 19. @jimmydahlqvist AWS Greengrass Interact with AWS services AWS provided components • Log manager Custom components • Publish new versions of components Support for AWS Lambda
- 20. Cloud deep dive
- 21. @jimmydahlqvist Cloud architecture • Reliably capture data • Managed services • No direct EventBridge integration Data service Detection service Notification service Data augmentation service IoT Core and SQS
- 22. @jimmydahlqvist • Reliably capture data • Managed services • No direct EventBridge integration IoT Core and SQS
- 23. @jimmydahlqvist Cloud architecture Data service Detection service Notification service Data augmentation service
- 24. @jimmydahlqvist Data augmentation • Data transform pattern • Fetch data from DynamoDB
- 25. @jimmydahlqvist Cloud architecture Data service Detection service Notification service Data augmentation service
- 26. @jimmydahlqvist Detection service • Threshold breaches • Temperature trends • The dreadful stall
- 27. @jimmydahlqvist Cloud architecture Data service Detection service Notification service Data augmentation service
- 28. @jimmydahlqvist Serverless and event-driven Loosely coupled Scale and fail independently Cost effective Extensibility Highly available
- 29. Great BBQ?
- 30. I would say so
- 32. @jimmydahlqvist Choreography + Orchestration Data storage service Detection service Notification service Data transform service
- 33. @jimmydahlqvist AWS StepFunctions workflow types Standard workflow • Long running - 1 year • 2000 starts per second • Pay per state transition Express workflow • Short running - 5 minutes • 100k starts per second • Pay for duration
- 34. @jimmydahlqvist Express workflows Express Workflows employ an at-least-once model for asynchronous invocation
- 35. @jimmydahlqvist Create subscriptions Creates coupling on the filter Complex to add more targets Changing the filter affect all Coupling is on the event Easy to add more targets Filters can change independently
- 36. @jimmydahlqvist Default bus Avoid using the default bus for custom applications!
- 37. @jimmydahlqvist Transform don’t transport Use AWS Lambda to transform data, not transport.
- 39. @jimmydahlqvist Next step • Remove my own hardware (Inkbird IBT-6XS) • Make it SaaS!! • Blog posts and Open Source coming! • Add camera support • Train an ML model for detection • Alexa integration
- 40. @jimmydahlqvist Summary • Building an IoT system • Serverless and event-driven • Get great BBQ with help of technology
Editor's Notes
- Hi! I'm Jimmy! I have worked with AWS and severless since 2015, almost a decade now, and I have seen all kind of strange things. I’m a true serverless enthusiast, the very first solution I built on AWS was serverless and I have not looked back since. I have built serverless solutions for a variaty of companies, from startups to large enterpices. I'm the founder of serverless-handbook.com where you can find all kind of serverless things that i have built, ranging from workshops to small architecture patterns. And I have my blog on Jimmydqv.com As a day-time job, and yes, I do have a daytime job, I know people have been questioning that. I work as Head of AWS at Sigma Technology Cloud, we are an advanced services partner with AWS and do all kind of fun solutions. If you like to know more about us, visit our booth outside.... I'm AWS Ambassador, AWS Community Builder and one of user Group leader for the Scania user group.
- So the agenda for the day is: Background to the project Overview of the architecture and how it has evolved over time Cloud deep dive and – talk about changes I made and the benefits of that Summary – and final thoughts Before we dive into everything, who am I?
- Let’s jump into the background so we all have the same context.
- What is ? Low & Slow – Vs hot and fast I normally target ~120-125c – Reason for doing that is….. Tenderize, tissue breakdown Styles – Us (varies by state, Texas, Tenesee, Nort caroline, New york…), Jamaica, Austrailia, UK – Sweden most US styles Not cooking – Art! But doesn’t mean we can’t use Tech…..
- Audience Poll!! Kamado ? UDS? Electric ? Offset ?
- My offset smoker – Isn’t she a beuti! How does an offset work + IoT Device – ANIMATION IoT Device – Watch fire, help from technology, What does it do!!
- This is an overview of the IoT Device and the food probes I use, standard 2.5mm thermistor probes. What is a Thermistor? Negative Temperature Coefficient (NTC) Thermistors NTC thermistors exhibit a rapid decrease in resistance with an increase in temperature. NTC is used Positive Temperature Coefficient (PTC) Thermistors PTC thermistors exhibit a sharp increase in resistance with an increase in temperature
- Background and context done Let’s look at the components, and architecture
- Two parts – IoT device + cloud Example services for both CLICK!!! Let us take a closer look at the IoT Device
- Two parts – HW + SW HW: Rasp-Pi 4 2.5mm food probes – thermistor! MCP3008 (10bit 8 channel)– AD converter – read voltage SW: AWS IoT Greengrass 2.0 – Core Custom component – math voltage to temp
- Initialy simple Python app updated over SSH Problems – Logs and Hard to update
- Done – Looked at IoT device CLICK Let’s talk about Cloud – where we will spend most of our time
- First version of cloud looked like this… Happy little man to the left I guess is me… Device data -> IoT Core IoT Core -> Rules -> Storage + Athena + Dynamo IoT Core -> Rules -> Several SF business logic (thresholds, trends…) API GW RESTful…
- IoT rules as router –> on mqtt –> not on payload –> messages discarded in business logic Each event –> one OBJECT in s3 –> Glue/Athena not optimized for that Data written directly to storage (Storage First is good but…) –> Format dictated by the device –> need transform Hard to extend –> Several services did same thing –> notification –> or needed to implement API
- So I had a couple of areas that I wanted to improve. I wanted to add the possibility to do a proper ETL and data transformation. So it would be easy to change how data is stored and presented in the cloud withour having to change the device, Introduce a event driven architecture with EventBridge as the event router instead of relying to heavily on rules in IoT core. The rules are great but at this point they didn’t really fulfill what I wanted to accomplish. Lastly decouple the services. Break the mini-monolith... All changes was to create a more flexible system that was easy to extend and manage.
- Let’s start by looking at the changes made for the IoT device
- With the Initial problems I decided to test out Greengrass. Interact with AWS Services – s3 config etc AWS provided components – Log Manager Build SW as components - Easy to push and publish new versions AWS Lambda support
- Now we move over to the cloud part what was done there
- Second iteration several improvments. IoT Core no longer primary message router -> EventBridge introduced -> EventDriven architecture -° Rules / targets / subscription Business Logic -> Microservice pattern – with clear responsibility -> Communicating over EB and API Transformation service -> EB Transform / augemnt EB – PayLoad filtering
- Let’s take a closer look at some parts of the architecture….. And start with the ingress part…… CLICK! -> Animate Favorite pattern – Storage First Create reliable way to capture data – prevent data loss Use managed services Very powerful when incoming data doesn’t require instant transformation
- Lambda is probably not the best solution……
- The next part we should look at is the data augmentation CLICK This part became very important in the new design… CLICK
- ….. It allowed me to decouple cloud development from device development Data transform pattern Data augmentation -> Additional information fetched from DynamoDB. Data is transformed to an internal format -> Decouple from the IoT device Almost no code. StepFunctions integration to other services
- One of the most important services are the detection service CLICK This is where all the BBQ magic happens….. CLICK
- Threshold breach Trends Stall – Happens around 70c (160f). What is it?
- Next part to look at is actually the entire system. CLICK!! Everything is built on a serverless and event-driven approach.
- Reason You build an serverless and eventdriven architecture. Loosely coupled services Scale and fail independently Cost effective – pay for what you use Extensibility – easy and fast to extend HA – built in
- So have technology help me to become a better pitmaster and to get some great BBQ? Some may say it’s cheating but why not use tech to help?
- …. I let the result speak for it self!
- EventBridge Choreographs Four bounded contexts represented by each service (orchestrated) Stepfunctions The service has unique business logic that need to be implemented and happen in a certain order, when a event is invoked.
- StepFunctions Express Workflow has an invocation model of “At least once” which means that it’s possible that your workflow get invoked twice. In dev and test it’s unlikely that it will happen, even in my small project I never saw it. But if you run it in a large enough scale it will happen. So make sure your workload is idempotent and can handle it. This makes them ideal for orchestrating idempotent actions such as transforming input data and storing via PUT in Amazon DynamoDB.
- When building on eventbridge I would recommend that you create subscriptions. With that I mean that you create one event rule for one target. Even if EB support up to 5 targets per rule I still say this should be a 1-2-1 mapping. Why? If you create one rule with multiple targets you will create a coupling on the event filter. And if you hit the 5 target limit what are you suppose to do then? Create a second rule with the same filter and start adding new targets? What if you need to update the filter? You will impact several targets that might not be what you wanted to do in the first place, leaving you to start breaking things apart again. So instead we create subscriptions where we create the coupling on the event it self, and we set one target for one rule. It’s easy to add more target, just create a second rule and add the target to it. And the filter in every rule can change without affecting any other target. However…. This of course can lead to several rules having the same filter and that could create problems on it’s own. But in my opinion it’s still better to create subscriptions and deal with a rule explosion.
- there is a default EB bus in the region already existing that AWS services post events to. Should you use that as your bus in your application? My recommendation is NO. Leave that bus to be used by AWS services and create your own custom buses for you application. The reason for that is that in that case will become easier to extend and ass busses later. I have seen the default bus being used and the mess it was when then moving to a custom bus. So can’t repeat it enough… Leave the default bus alone! Create custom buses! It’s like using the default VPC…. We don’t do that either!
- So instead of using SQS + Lambda to transport. It would be possible to have IoT Core invoke a Express StepFunction with only one step which would be a SDK integration and that integration would just post the event to EB. No code, only 100% managed AWS services doing all of the work for me behind the scene. That is a really nice way to be able to use the SDK integration in StepFunctions. This wcould however break my storage first approach, but since there is no code involved, just AWS calling the SDK on my behalf I could totally live with that. It truly show the power and flexibility in a serverless approach and a service like AWS StepFunctions.
- So what is next in this project then?
- So to summarize the last 40 minutes. Building an IoT system Serverless and event-driven Get great BBQ with help of technology And with that I say! CLICK!!