Biking on the edge - Jerome Mies - SRD23

1. Jerome Mies (Lecturer/ Researcher at Hogeschool van Amsterdam) 23 May, 2023 Biking on the edge Monitoring the conditions of streets in Amsterdam with AWS IoT Greengrass

2. Introduction video – http://vimeo.com/769075918/f2159baf0b

3. Overview • What are we working on? • What is edge computing? • How do we use edge computing? • How do we use AWS? • How can we integrate AWS? • Example models

4. Together improving the wellbeing in public spaces Hogeschool van Amsterdam Centre of expertise for applied artificial intelligence Smart Asset Management Lab SURF SURF is the collaborative organisation for IT in Dutch education and research Municipalities Velotech Solutions Sustainable object detections in the public space

5. Conditions of the assets in the city impact our safety and wellbeing Light posts Traffic lights Garbage Traffic signs Road markings Road surface Trees 1

6. Our ambition is to cover the complete city maintenance with LiDAR & video AI Video image recognition LiDAR mapping Light posts Traffic lights Garbage Traffic signs Road markings Road surface Trees 1

7. Provide an accurate & complete overview of deficiencies in the city through eco-friendly inspections 3 Detailed routes per area Image analysis using algorithms Registration in eform-app Bicycles can go where cars can’t Validation of the inspections Integration with maintenance subcontractor systems Key features of the inspections Inspections (periodically) of designated areas by bicycle/vehicle Number of identified objects, defects & exact GPS locations Reporting & dashboard that is linked with the existing workflows Purpose Improve social & traffic safety Minimize liability risks Preventing peak workload for maintenance parties in winter

8. Route planning Schedule riders Data gathering & mobile mapping Before & after inspections Data analysis Reporting municipality Reporting maintenance contractor Green and fast methods to get insights in the public space by using bikes & technology AI 2 Our approach:

10. Edge computing is real-time computing at the end devices

11. What are edge devices?

12. A cloud computing provider can help integrate applications with edge devices • Cloud providers • Amazon AWS • Microsoft Azure • Edge Impulse • IoT (Edge) devices • Jetson Nano • Raspberry Pi • Arduino Nano

13. Edge computing versus Cloud computing Advantages of edge computing • Real-time computing • Limited sending of information • Privacy

15. Light post detection at the edge 4 Inspections on a bicycle Real-time inference on images to detect defects Bicycle-mounted edge devices operate autonomously.

16. From light post detection to tilted street lights The angle of a light post detected in an image is determined in multiple steps. The angle of the light post is compared to the horizon. 5 Light posts in a training set are labelled using the Roboflow application and models are trained.

17. Modular edge device • NVIDIA Jetson Nano or NVIDIA Xavier NX • 32 GB SD card • 4G and WiFi connectivity • Battery pack • Camera • GPS sensor and gyroscope

18. Modular edge device Multi-vehicle use

20. • Deploy software to all remote devices at once from a central cloud location • Software versioning • CI/CD, Infrastructure as Code • Modular Greengrass components make it possible to design flexible architectures adapting to multiple use cases • Docker support • Dealing with unstable network connections reliably Managing remote devices with AWS Greengrass

21. • Every Greengrass component acts as a microservice in a data pipeline, e.g. taking pictures with a camera, doing the AI inference, removing sensitive data, uploading to the cloud, etc. • Orchestrate event-driven workflows at the device level, with components talking over inter-process communication • Trigger or schedule workflows remotely over MQTT from AWS IoT Core • Address individual edge devices using a dedicated MQTT topic structure Greengrass components and event-driven workflow

22. • Using publicly available models • Dedicated models to detect objects of interest, e.g. models from municipalities for garbage detection • Pre-trained models, such as YOLOv5 trained on a COCO dataset, to detect faces for further removal • Training custom models with transfer learning • Training data is collected from the bicycles and labelled manually. • Train models with Amazon SageMaker Studio • Track training experiments and model performance with MLflow Models and training

23. • Model optimization is crucial for battery-powered edge applications. • Limited to GPU capacity of the device (NVIDIA Jetson Nano vs NVIDIA Xavier NX) • Faster models (and more powerful devices) will allow us to move from image processing to video processing to LIDAR processing at the edge. Model optimization and limitations of the edge

24. • With remote device management, scaling towards multiple edge devices is easy. • Remote connectivity enables real-time updates and event-driven data processing. • Modular design allows us to accommodate multiple use cases. • Let’s make cities a safer place through cloud and edge technology and AI. • We have developed a blueprint for smart edge-computing applications for use in public sector and research. Cloud is the key to scalability at the edge

25. We can use AWS to integrate with our dashboard for simpler use Integration AWS with a dashboard

27. Examples of use cases for VeloTech • Road deterioration • Broken street lights • Tilted lanterns • Street signs pollution • Lidar scans

29. Road deterioration example - text in Dutch

30. Broken street lights example

31. Tilted street lights example

32. Street signs pollution example

33. Lidar scans – some ideas Tilted street lights Falling trees Collapsing quays (‘kades’) and bridges Mapping the city

34. Questions?

Biking on the edge - Jerome Mies - SRD23

Biking on the edge - Jerome Mies - SRD23

Recommended

More Related Content

Similar to Biking on the edge - Jerome Mies - SRD23

Similar to Biking on the edge - Jerome Mies - SRD23(20)

More from SURFevents

More from SURFevents(20)

Recently uploaded

Recently uploaded(20)

Biking on the edge - Jerome Mies - SRD23