"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

1. Fish Plays Pokemon
Using IoT and Kafka to Monitor an
Aquarium Environment and Gamify Fish Behaviour
Shritesh Jamulkar
Software Engineer @THG

2. What We built...
IoT based monitoring
Cloud Processing - identify fish
and monitor
Alerts owner
Fish positioning used to play
game

3. YouTube: https://www.youtube.com/watch?v=48-qOC4fCdk

4. Architecture

5. Hardware
ESP-EYE v2.1
ESP32
Power mb
2 kits
9V Battery
Power mb v2

6. Frames Processing:
Yolov5s model used.
Trained on several hundred images -
bounding boxes.
Created Weights file.
Hosted on Roboflow
Roboflow API called
ROBOFLOW
Lambda function triggered
1
2
3
AWS CLOUD

11. About me...
Software Engineer @Thehutgroup, Manchester UK 🇬🇧
Master's in Computer Science - Future Networked Systems from
Trinity College, Dublin, Ireland 🇮🇪.
Previously, Software Engineer @Samsung Research, Banglore,
India 🇮🇳
Hyperledger Mentorship Program 2021 - Mentee, Open-Source 🌐
Bachelor's in Computer Science from NIT Raipur, India 🇮🇳

12. THANK YOU