1.
This project has received funding from
the European Union’s Horizon 20 20
research and innovation programme
under grant agreement No 688403
www.tulipp.eu
TULIPP
Title :
Place :
Date :
Towards Ubiquitous Low-power Image Processing Platform
London, 10th Intelligent Imaging Event
29th of April 2019
Flemming Christensen
Sundance Multiprocessor Technology Ltd

2.
• Everything started with a common need for
• High performance
• Real-time
• Low-Power
• Embedded
• Image processing applications
What is TULIPP?
Goal: Safer driving experience
Goal: Bring intelligence to the drones
Goal: Reduce Radiation Dose by 75%

3.
What is TULIPP?
Tulipp: Bringing energy efficiency from chip level to system level
Intensive Image Processing Embedded
Constraints
Time-to-market / Cost-
sensitive
Source: http://www.lnci.org.au

4.
The Reference Platform
Processor
IO
Memory
Component
tools
Operating
System
Toolchain
CPU
What is TULIPP?
(the concept)
Methodology to select the
best suitable components
for build power-efficient
image-processing platforms

5.
How did we proceed ?
WP7: Management, Coordination
LABEL : Marketing, Ecosystem and Pre-normalisation
WP6: IP protection, Dissemination, Communication, Advisory Board
and Exploitation preparation
WP1: Reference platform definition
(Interfaces & implementation Rules)
Instantiations
WP2:
Hardware
WP4:
Programming
Toolchain
WP3:
Runtime, API,
Libraries & OS
feedback WP5 : Usecases description
and Integration and platform
validation

6.
The Reference Platform in a book
Methodology captured
in a book
+
Guidelines to provide
expert hints on common
issues & valuable
implementation clues

7.
• A guideline is insights that occurred while working on the project.
It can be a result of:
• running experiments
• our expertise
• reading other documents
• teaching something to "a student"
Like:
• Do not use floating point computation on FPGA
• Avoid using heavy libraries while writing source code for
embedded systems
• …
https://github.com/tulipp-eu/tulipp-guidelines

8.
TULIPP Guidelines - more than 50 collected

9.
The TULIPP HW platform: Find a chip!

10.
The TULIPP HW platform: Select the chip

11.
TULIPP reference platform interfaces

12.
The TULIPP HW platform: Zynq CPU Module

13.
The TULIPP HW platform: I/O versatile

14.
The TULIPP HW platform: made for vision

15.
The TULIPP hardware solution

16.
The TULIPP HW platform: PC104 stackable

17.
The TULIPP Operating system: HIPPEROS Maestro
M
A
E
S
T
R
O
Maestro, an RTOS for Multi-cores

18.
The TULIPP Operating system: HIPPEROS Maestro
Maestro is designed for Real-Time applications
• Determinism & bounded guarantees
• Checks & controls deadlines of tasks with
Real-Time scheduling policies
• Resource usage is bounded and checked

19.
The TULIPP Operating system: HIPPEROS Maestro
Maestro is a new micro-kernel
• No legacy from any former mono-core OS like
“Linux”
• Designed to leverage the power of multi-core
architectures
• Unlike other RTOS, Maestro is Multi-core at its
very heart

20.
The TULIPP Operating system: HIPPEROS Maestro
Maestro is designed for embedded application
• small memory footprint
• embedded processors MultiCore ARM CPUs
• support FPGA technology

21.
The TULIPP Operating system: HIPPEROS Maestro
Maestro is a full featured RTOS
• MMU support
• resource sharing
• usual OS services (timers, etc...)

22.
The TULIPP Operating system: HIPPEROS Maestro
Click here to play on YouTube

23.
STHEM: The TULIPP Tool-chain
Support uTilities for Heterogeneous EMbedded image
processing (STHEM)
Insights:
• Significant effort has been invested into the
development of vendor tools
• STHEM fills the productivity gaps between existing tools
• Extensions and Compatible with Xilinx Tools

24.
STHEM: The TULIPP Tool-chain
• Supports development for all platform components
• Maps source files of the application to the
appropriate tool chain
• Retrieves OS configuration from the developer
Development and Mapping

25.
STHEM: The TULIPP Tool-chain
• Boots OS with selected configuration (if needed due to
changed configuration)
• Updates files (binaries, bitfiles, etc.)
• Initialises the reconfigurable logic (if needed)
• Starts the application with the requested instrumentation
Runner

26.
STHEM: The TULIPP Tool-chain
• Analyses performance results and presents
findings to the developer
Analyser

27.
The Tool-chain – Interactive GUI
https://github.com/tulipp-eu/sthem

28.
Embedded  Control the energy
• Heterogeneous platform  tasks mapping
• How to make sure we did the best mapping? (energy)
Intensive Image Processing Embedded
Constraints
Time-to-market / Cost-
sensitive
Source: http://www.lnci.org.au

29.
The Lynsyn PWM board : the probe

30.
The power consumption analyser

31.
Design Space Exploration tool

32.
How Lynsyn Works
Click here to play on YouTube

33.
The Generic Development Process
Connect and
abstract
STHEM =
Supporting
uTilities for
Heterogeneous
Embedded image
processing
platforms
Support for TULIPP
platform instances

34.
The Starter Kit: How does it works?
Goal-oriented
Advice
Instantiation
Recommended
Implementation
Methods
Project Applications
Platform Instance

35.
end product
dependent
Several Instances from same reference

36.
Advanced Driver Assistance Systems
• Pedestrian detection for collision avoidance
• Based on Viola-Jones object detection
• Operates on 640x480 24-bit images
Original
C/C++
code
Adapted
C/C++
code
Accelerated on the
Tulipp Platform
PC platform
150 W
10 s/frame
15 W
66 ms/frame

37.
Unmanned Aerial Vehicle
𝐼left, 𝐼right
Obstacle
avoidance
Disparity
estimation
𝐷
Original
C/C++
code
Adapted
C/C++
code
Accelerated on the
Tulipp Platform
15 W
29 frame/s
29ms latency

38.
Unmanned Aerial Vehicle
Click here to play on YouTube

39.
Medical Use Case
Original
C/C++
code
Adapted
C/C++
code
Accelerated on the
Tulipp Platform
15 W
29 frame/s
29ms latency
• X-ray video for surgery
• Embedded on the sensor
• Lower radiation doses by factor 4
• Image denoise & enhancement
• 1024x1024 24-bit images
sensor

40.
Medical UC
Noisy input
image
Denoised
image
Enhanced
image
+ rotation

41.
Tulipp Endorsement – Advisory Member

42.
Tulipp Endorsement - Advisory Member

43.
TULIPP Starter Kit – Next Project?
• HW instance based on Xilinx Zynq UltraScale+
• Power aware RT operating system
• Toolchain support for efficient implementation
• Sample Applications
• The TULIPP Handbook