Embedded video tends to be an increasing subject in embedded Linux developments. Even if ARM SOCs provide great resources for video treatment with dedicated IPU, GPU …, a dual approach based on FPGA + general purpose processor is an interesting alternative. In this presentation Christian Charreyre will provide a return of experience of a video acquisition and treatment application developed on a Zynq board, with focus on interactions between the FPGA part and Linux. The presentation will also describe the embedded Linux development choices, based on standard meta-xilinx Yocto layer instead of current solutions using PetaLinux or WindRiver Pulsar. The typical audience is people interested in embedded Linux solutions using a mixed architecture between a general purpose processor and a specialized co-processor. Attendees will receive informations on the relationships between the Linux environment and the FPGA about hardware description, boot mechanism, necessary customizations of Yocto recipes, and much more. Christian Charreyre, CIO Systèmes Embarqués

1. 11 Avenue Marigny 13014 Marseille www.ciose.fr christian.charreyre@ciose.fr
Developing an embedded video application on dual Linux + FPGA architecture 109/26/17
Developing an embedded videoDeveloping an embedded video
application on dual Linux + FPGAapplication on dual Linux + FPGA
architecturearchitecture
C. CharreyreC. Charreyre
christian.charreyre@ciose.frchristian.charreyre@ciose.fr
http://www.ciose.frhttp://www.ciose.fr
https://twitter.com/CIO_SysEmbhttps://twitter.com/CIO_SysEmb
http://fr.slideshare.net/charreyrehttp://fr.slideshare.net/charreyre

2. License
Developing an embedded video application on dual Linux + FPGA architecture 209/26/17
Attribution-Noncommercial-Share Alike 4.0 International
● You are free:
to Share - copy and redistribute the material in any medium or format
to Adapt - remix, transform, and build upon the material
The licensor cannot revoke these freedoms as long as you follow the license terms.
● Under the following conditions:
Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so
in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes.
ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license
as the original.
● No additional restrictions — You may not apply legal terms or technological measures that
legally restrict others from doing anything the license permits.
● License text : http://creativecommons.org/licenses/by-nc-sa/4.0/legalcode

3.  Small company dedicated to embedded solutions (16 persons)
 Electronics and software developments
 Embedded Linux expertise since 15 years
 Open Embedded and Yocto expertise since 7 years
 Headquarters in Saint-Etienne
 Agency in Marseilles
 Embedded Linux expert for Cap’tronic program
Developing an embedded video application on dual Linux + FPGA architecture 309/26/17
Presentation of CIO Systèmes Embarqués

4. Developing an embedded video application on dual Linux + FPGA architecture 4
 Responsible of Embedded Linux technologies in CIO Systèmes Embarqués
 Embedded Linux trainer
 30 years in embedded and Unix / Linux world
 Developer and maintainer of kernel, u-boot ports on Arm boards
 Developer and maintainer of Yocto layers
09/26/17
Presentation of the speaker

5. Developing an embedded video application on dual Linux + FPGA architecture 509/26/17
The embedded video application
● High performance video acquisition, real time treatments and
display
● Merge and synchronization of 2 images issued by 2 cameras
● Deployed in a device with safety constraints related to video
● Latency between real world and display < 200 ms
● Video must be available 7 s after Power On
● Device developed on an ARM architecture (CIO’s skills) →
embedded Linux software environment

6. Developing an embedded video application on dual Linux + FPGA architecture 609/26/17
Why use a FPGA ?
● Previous video applications developed on iMX.6 processor with gstreamer
to manage video pipeline
● This architecture can’t be used here :
– Safety certification problem on Linux
– Real time performances (latency, jitter) ???
● High performance IP for FPGA dedicated to video applications available
(commercial product)
● We decided to share the application between Embedded Linux on ARM
processor and FPGA
– Video processing done by the FPGA
– Configuration, errors monitoring and utilities done by Embedded Linux
distribution

7. Developing an embedded video application on dual Linux + FPGA architecture 709/26/17
The hardware platform
● The choosen hardware platform is a PicoZed board mounted on a custom base board developed
by CIO
● PicoZed = System On Module based on the Xilinx Zynq®-7000 All Programmable (AP) SoC
● Zynq 7000 offers a dual core ARM processor + FPGA in the same chip
● ARM processor is called Processing System (PS)
● FPGA is called Programmable Logic (PL)

8. Developing an embedded video application on dual Linux + FPGA architecture 809/26/17
Linux software environment
● PicoZed Linux environment proposed with 2 options :
– Petalinux is the Linux environment proposed by Xilinx
– Wind River Pulsar Linux, a Yocto based environment
● CIO has great experience of OpenEmbedded and Yocto tools :
– Petalinux quickly eliminated as Yocto is a kind of de facto
standard in Embedded Linux
– Wind River Pulsar Linux evaluated :
● Rejected due to the presence of containerization in the solution. We did
not want to loose time with this technology
– We finally selected meta-xilinx layer and assembled it with other
layers to build our Embedded Linux environment

9. Developing an embedded video application on dual Linux + FPGA architecture 909/26/17
Linux software environment
● All necessary layers selected from OpenEmbedded Layer Index
● Layer coherency assured by repo tool
● Finally we have made a Yocto based build system tailored for the
embedded distribution
● Embedded distribution built by customizing Yocto recipes and
creating new ones for in house applications
● A more complex Board Support Package due to interactions
between PS and PL (detailed later)

10. Developing an embedded video application on dual Linux + FPGA architecture 1009/26/17
FPGA developments
● FPGA developments made with Xilinx Vivado tool (Eclipse based)

11. Developing an embedded video application on dual Linux + FPGA architecture 1109/26/17
FPGA developments
● FPGA engineer designs its IPs inside Vivado
● When the design is finished, the final VHDL code for the FPGA is
generated
● But we don’t want to have 2 isolated parts : Embedded Linux
distribution & logic design. AXI interface allow us to communicate
between PS and PL.
– AXI stream : dedicated to stream data
– AXI ACP : allow copy data to cache memory (close to the processor)
– AXI light : allow exchange of some data
– AXI HP : allow access to the DDR and store data like images without
bandwidth issues.

12. Developing an embedded video application on dual Linux + FPGA architecture 1209/26/17
How can PS and PL interact ?
● It is possible to design hardware components in the PL that can be
seen by the PS
● Potential use cases :
– Add additional hardware resources for the embedded Linux
distribution, normally not available on PS → extend PS capabilities
● Create an additional serial line
– Use dedicated hardware to communicate between PL and PS
● Create GPIOs in the PL and make Embedded Linux use them to read / write
logic data from / to the VHDL application
– Allow dynamic customization of IPs
● Perform a specific crop on an image which can be changed from Linux
(position, size).

13. Developing an embedded video application on dual Linux + FPGA architecture 1309/26/17
Technically, how does it work ?
● We have seen
– The application constraints
– The selected hardware and software architecture
– The tools
● But of does it work ?

14. Developing an embedded video application on dual Linux + FPGA architecture 1409/26/17
Boot Mechanism
● PicoZed needs a First Stage Boot Loader (FSBL) before u-boot
● FSBL code is generated by Vivado according to the design
– This code can be compiled inside Vivado or externally with
and ARM toolchain
● FSBL then starts u-boot which in turn starts Linux
● The FPGA code (bitstream) must be loaded by FSBL or by u-boot
– The FPGA can’t start alone
– The VHDL code is reloaded at each boot → impact on boot
time

15. Developing an embedded video application on dual Linux + FPGA architecture 1509/26/17
Interactions between PS and PL
● The Xilinx Linux kernel (founder specific) has dedicated drivers for
hardware resources implemented by PL and shared with PS
● These resources are appended to the device tree, thus making
them available to Linux kernel
● Vivado automatically generates a pl.dtsi file that is included
in the board device tree
● pl.dtsi reflects selection and design of hardware resources of
the PL accessible to the PS

16. Developing an embedded video application on dual Linux + FPGA architecture 1609/26/17
Interactions between PS and PL
/*
* CAUTION: This file is automatically generated by Xilinx.
* Version: HSI 2016.4
* Today is: Fri Aug 4 16:29:54 2017
*/
/ {
amba_pl: amba_pl {
#address­cells = <1>;
#size­cells = <1>;
compatible = "simple­bus";
Ranges ;
RS485_PL: serial@42c00000 {
clock­names = "ref_clk";
clocks = <&clkc 0>;
compatible = "xlnx,xps­uartlite­1.00.a";
current­speed = <115200>;
device_type = "serial";
interrupt­parent = <&intc>;
interrupts = <0 35 1>;
port­number = <0>;
reg = <0x42c00000 0x10000>;
xlnx,baudrate = <0x2580>;
xlnx,data­bits = <0x8>;
xlnx,odd­parity = <0x0>;
xlnx,s­axi­aclk­freq­hz­d = "50.0";
xlnx,use­parity = <0x0>;
};
Additional serial line
Link with Xilinx driver

17. Developing an embedded video application on dual Linux + FPGA architecture 1709/26/17
Interactions between PS and PL
…
video_in_0_Status: gpio@41230000 {
#gpio­cells = <2>;
compatible = "xlnx,xps­gpio­1.00.a";
gpio­controller ;
reg = <0x41230000 0x10000>;
xlnx,all­inputs = <0x1>;
xlnx,all­inputs­2 = <0x0>;
xlnx,all­outputs = <0x0>;
xlnx,all­outputs­2 = <0x0>;
xlnx,dout­default = <0x00000000>;
xlnx,dout­default­2 = <0x00000000>;
xlnx,gpio­width = <0x5>;
xlnx,gpio2­width = <0x20>;
xlnx,interrupt­present = <0x0>;
xlnx,is­dual = <0x0>;
xlnx,tri­default = <0xFFFFFFFF>;
xlnx,tri­default­2 = <0xFFFFFFFF>;
};
};
};
Information towards PS
Link with Xilinx driver

18. Developing an embedded video application on dual Linux + FPGA architecture 1809/26/17
Device tree generation
● In fact, Vivado generates the whole device tree :
– PS part through zynq­7000.dtsi
– PL part through pl.dtsi
– And the final system.dts that includes PS and PL part, + nodes
customizations according to Vivado design
● Vivado covers PL but also parameters impacting PS
● Device tree must be rebuilt after each change in Vivado → automation
of the workflow welcome
– Development of dedicated Yocto recipes to manage generation and
dispatch of all these files in Board Support Package dedicated recipes
● U-boot and device tree recipes impacted

19. Developing an embedded video application on dual Linux + FPGA architecture 1909/26/17
Boot time optimization
● The requirement is to have the image completely stable at much 7 s
after Power ON
● Tasks necessary before image :
– FSBL
– U-boot
– Bitstream loading and FPGA programming
– Kernel initialization
– Userland startup (system V init)
– Cameras initialization through spi communication
– FPGA video IP configuration

20. Developing an embedded video application on dual Linux + FPGA architecture 2009/26/17
Boot time optimization
● Misc techniques used to minimize image arrival time :
– Activation of bootstage report feature of u-boot
– Use of bootchart to identify where boot time is consumed during Linux start-up
– U-boot optimization to reduce peripherals inits
– Linux kernel optimization to reduce kernel size (media access time) and
peripherals init
– System console removal
– Completely reorder init scripts to start video application as early as possible
– Use of read only root file system to avoid file system checks and corrections
(power off without halt)
– Maximize spi speed on spi bus with camera
● Finally image is here and stable 5.6 s after Power On.

21. Developing an embedded video application on dual Linux + FPGA architecture 2109/26/17
Consequences of safety requirements
● The design has been done so that the safety certification will impact
only the FPGA
– Linux developments not directly impacted by certification
process
● But the software must be updated through IP communication over
Ethernet
● As FPGA software is updated through Linux IP features, the
installation & upgrade process must be secured to avoid potential
hacking
● Installer and updater utilities use asymmetric cryptography
mechanism to authenticate all software elements before
installation / update

22. Developing an embedded video application on dual Linux + FPGA architecture 2209/26/17
Conclusion
● At the beginning of the project, 2 major decisions :
– Use a mixed design based on Embedded Linux + FPGA
– Use meta-xilinx layer and assemble layers by ourself instead
of standard solutions (Petalinux or Wind River Pulsar Linux)
● What are the results ?

23. Developing an embedded video application on dual Linux + FPGA architecture 2309/26/17
Conclusion
● The main concerns regarding the dual architecture were about :
– Splitted design
– Communication between PS and PL
– Kernel management of peripherals built in PL part
● Splitted design is a constraint but the advantages of using a FPGA for video
treatments are more important than the drawbacks
● AXI interface offers many solutions to implement communications between PS and
PL
● Drivers available in Xilinx kernel + automatic generation of device tree make the
peripherals designed in PL well supported by the kernel
● PL design make the final platform easily adaptive to new requirements
– Ex : debug GPIOs useful only during development phase synthesized in the PL
part then removed

24. Developing an embedded video application on dual Linux + FPGA architecture 2409/26/17
Conclusion
● Using meta-xilinx and assembling them by ourselves with other
layers from OE layers repository was a good choice
– Good quality of meta-xilinx layer
– We select exactly the layers we need
– Finally we have a very well adapted distribution
● This was also possible because we have good skills on Yocto build
tool and layers architecture and features
– Especially to understand the BSP logic

25. Developing an embedded video application on dual Linux + FPGA architecture 2509/26/17
Questions ?
● Thank you for your attention
● Time for questions