0
Persistent log with UBI
Matthieu CASTET - www.parrot.com

25 September 2013

Persistent log with UBI
Persistent log with UBI

ulog
Goal

Log must be persistent on product : Nand flash
Can be used to trace system updates

Has to be independent/hidden of a...
Which interface to use ?

Raw flash (MTD)
UBI
Nand filesystem (UBIFS, ...)

Persistent log with UBI

ulog
Flash device vs block device 1/2

Block device
Consists of sectors
Sectors are small (512, 1024 B)
Maintains 2 main operat...
Flash device vs block device 2/2

Block device
Bad sectors are re-mapped and
hidden by hardware (at least in
modern LBA ha...
MTD

MTD stands for "Memory Technology Devices"
Provides an abstraction of flash devices
Hides many aspects specific to part...
UBI

Stands for "Unsorted Block Images"
Provides an abstraction of "UBI volume"
Has kernel API and user space API (/dev/ub...
UBI volume vs MTD device
MTD device
Consists of physical
(PEB), typically 128 KB
Has 3 main operations

eraseblocks

UBI d...
Main idea behind UBI
Maps LEBs to PEBs
Any LEB may be mapped to any PEB
Eraseblock headers store mapping information and e...
Other
Handle bit-flips by moving data to a different PEB
Configurable wear-leveling threshold
Atomic LEB change
Volume updat...
UBIFS

Filesystem on top of UBI (2.6.27 2008-10)
Needs a minimal number of LEBs to work : 17
http://www.linuxmtd.infradead...
log over UBI

we have a UBI device (for linux kernel) with free space
UBIFS has too much overhead

Persistent log with UBI...
UBI user API
include/mtd/ubi-user.h
device attach
device detach
volume create
volume delete
volume resize
volume rename
vo...
ulog

A log entry is much smaller than a page size (512B-4KB)
A cache is needed (in flash)
ulog uses a dynamic volume to ha...
ulog

Persistent log with UBI

ulog
ulog : flush 1

Persistent log with UBI

ulog
ulog : flush 2

Persistent log with UBI

ulog
ulog : flush 2

Persistent log with UBI

ulog
ulog : flush 3

Persistent log with UBI

ulog
ulog : flush 4

Persistent log with UBI

ulog
Data flush

Copy data from level N to level N+1
Merge pages to try to fill pages from level N+1
if level 0 has 64 entries of...
ulog : rotate 1

Persistent log with UBI

ulog
ulog : rotate 2

Persistent log with UBI

ulog
ulog : rotate 3

Persistent log with UBI

ulog
ulog : rotate 4

Persistent log with UBI

ulog
ulog : rotate 5

Persistent log with UBI

ulog
Data rotate

Clean next LEB (if not empty)
Write data on it

Persistent log with UBI

ulog
Data format

Header (32 bits)
size (24 bits)
version (3 bits) (relevant for first page)

Log data

Persistent log with UBI
...
Scan

Parse header for all LEBs
LEB version
LEB index (which pages have data)

For last level (L2), find current LEB
All em...
Scan

Parse header for all LEBs
LEB version
LEB index (which pages have data)

For last level (L2), find current LEB
All em...
ulog API

ulog_init
ulog_destroy
ulog_read
ulog_printf
ulog_vprintf
ulog_flush

Persistent log with UBI

ulog
ulog demo

Persistent log with UBI

ulog
TODO

any remark?

Persistent log with UBI

ulog
TODO

any remark?
Need to correctly handle all power failure cases during a
flush
Can be complex in case of cascaded flush

...
Questions ?

Merci pour votre attention !
Thanks for your attention!
Questions?

Persistent log with UBI

ulog
Upcoming SlideShare
Loading in...5
×

Kernel Recipes 2013 - Persistent logs using UBI

611

Published on

A short presentation about a library for persistent log based on the UBI layer. The goal is to enable an embedded software component to log persistent messages (even after a full update of the system) without requiring the use of a file system.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
611
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Kernel Recipes 2013 - Persistent logs using UBI"

  1. 1. Persistent log with UBI Matthieu CASTET - www.parrot.com 25 September 2013 Persistent log with UBI
  2. 2. Persistent log with UBI ulog
  3. 3. Goal Log must be persistent on product : Nand flash Can be used to trace system updates Has to be independent/hidden of application filesystem not on / filesystem Has to be usable on shipped products difficult to resize all partitions Persistent log with UBI ulog
  4. 4. Which interface to use ? Raw flash (MTD) UBI Nand filesystem (UBIFS, ...) Persistent log with UBI ulog
  5. 5. Flash device vs block device 1/2 Block device Consists of sectors Sectors are small (512, 1024 B) Maintains 2 main operations: Flash device Consists of eraseblocks Eraseblocks are larger ( typically 128KB) Maintains 3 main operations: read sector read from eraseblock write sector write to eraseblock erase eraseblock Persistent log with UBI ulog
  6. 6. Flash device vs block device 2/2 Block device Bad sectors are re-mapped and hidden by hardware (at least in modern LBA hard drives) Sectors are devoid of the wear-out property Flash device Bad eraseblocks are not hidden and should be dealt with in software Eraseblocks wear-out and become bad and unusable after about 103 (for MLC NAND) - 105 (NOR, SLC NAND) erase cycles Flash device is more difficult to handle (ecc, bad block, eraseblock, ...) Persistent log with UBI ulog
  7. 7. MTD MTD stands for "Memory Technology Devices" Provides an abstraction of flash devices Hides many aspects specific to particular flash technologies Provides uniform API to access various types of flashes E.g., MTD supports NAND, NOR, ECC-ed NOR, DataFlash, OneNAND, etc Provides partitioning capabilities Persistent log with UBI ulog
  8. 8. UBI Stands for "Unsorted Block Images" Provides an abstraction of "UBI volume" Has kernel API and user space API (/dev/ubi0) Provides wear-leveling Hides bad eraseblocks Allows run-time volume creation, deletion, and re-size Is somewhat similar to LVM, but for MTD devices Persistent log with UBI ulog
  9. 9. UBI volume vs MTD device MTD device Consists of physical (PEB), typically 128 KB Has 3 main operations eraseblocks UBI device Consists of logical eraseblocks (LEB), slightly smaller than PEB (e.g 126/124 KB) Has 3 main operations read from PEB read from LEB write to PEB write to LEB erase PEB erase LEB May have bad PEBs PEBs wear out MTD devices are static Persistent log with UBI ulog Does not have bad LEB (handle a certain amount of bad PEB) LEBs do not wear out - UBI spread the I/O load across the whole flash UBI volumes are dynamic
  10. 10. Main idea behind UBI Maps LEBs to PEBs Any LEB may be mapped to any PEB Eraseblock headers store mapping information and erase count Persistent log with UBI ulog
  11. 11. Other Handle bit-flips by moving data to a different PEB Configurable wear-leveling threshold Atomic LEB change Volume update/rename operation Suitable for MLC NAND Performs operations in background Works on NAND, NOR and other flash types Tolerant to power cuts Simple and robust design easy support in bootloader Persistent log with UBI ulog
  12. 12. UBIFS Filesystem on top of UBI (2.6.27 2008-10) Needs a minimal number of LEBs to work : 17 http://www.linuxmtd.infradead.org/faq/ubifs.html#L_few_lebs Complex filesystem : few (rare) corruptions seen on products Persistent log with UBI ulog
  13. 13. log over UBI we have a UBI device (for linux kernel) with free space UBIFS has too much overhead Persistent log with UBI ulog
  14. 14. UBI user API include/mtd/ubi-user.h device attach device detach volume create volume delete volume resize volume rename volume update (static volume) LEB erase LEB atomic change LEB map LEB unmap lseek read write Persistent log with UBI ulog
  15. 15. ulog A log entry is much smaller than a page size (512B-4KB) A cache is needed (in flash) ulog uses a dynamic volume to have per LEB write Persistent log with UBI ulog
  16. 16. ulog Persistent log with UBI ulog
  17. 17. ulog : flush 1 Persistent log with UBI ulog
  18. 18. ulog : flush 2 Persistent log with UBI ulog
  19. 19. ulog : flush 2 Persistent log with UBI ulog
  20. 20. ulog : flush 3 Persistent log with UBI ulog
  21. 21. ulog : flush 4 Persistent log with UBI ulog
  22. 22. Data flush Copy data from level N to level N+1 Merge pages to try to fill pages from level N+1 if level 0 has 64 entries of about 20 B, it can be merged in a page of 2 KB. Can be recursive ! flush of L0, but L1 is full flush of L1 needed ... Persistent log with UBI ulog
  23. 23. ulog : rotate 1 Persistent log with UBI ulog
  24. 24. ulog : rotate 2 Persistent log with UBI ulog
  25. 25. ulog : rotate 3 Persistent log with UBI ulog
  26. 26. ulog : rotate 4 Persistent log with UBI ulog
  27. 27. ulog : rotate 5 Persistent log with UBI ulog
  28. 28. Data rotate Clean next LEB (if not empty) Write data on it Persistent log with UBI ulog
  29. 29. Data format Header (32 bits) size (24 bits) version (3 bits) (relevant for first page) Log data Persistent log with UBI ulog
  30. 30. Scan Parse header for all LEBs LEB version LEB index (which pages have data) For last level (L2), find current LEB All empty use first (L2) LEB Use LEB version Persistent log with UBI ulog
  31. 31. Scan Parse header for all LEBs LEB version LEB index (which pages have data) For last level (L2), find current LEB All empty use first (L2) LEB Use LEB version Persistent log with UBI ulog
  32. 32. ulog API ulog_init ulog_destroy ulog_read ulog_printf ulog_vprintf ulog_flush Persistent log with UBI ulog
  33. 33. ulog demo Persistent log with UBI ulog
  34. 34. TODO any remark? Persistent log with UBI ulog
  35. 35. TODO any remark? Need to correctly handle all power failure cases during a flush Can be complex in case of cascaded flush Some data are currently duplicated Persistent log with UBI ulog
  36. 36. Questions ? Merci pour votre attention ! Thanks for your attention! Questions? Persistent log with UBI ulog
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×