Flash memory — how it works, and how we use it from Linux, SSDs and ‘real’ flash file systems

1. Flash Memory
David Woodhouse
david@woodhou.se
Kernel Recipes
September 2012
Software and Services Group

2. Overview
• What is flash?
• What makes flash “special”?
• How to use flash effectively... and safely
• Flash Translation Layers
• Other approaches
• Linux support for flash
Software and Services Group 2
2

3. The flash cell
Diagram © Wikipedia user 'Cyferz'
CC BY-SA 3.0
Source Bit
Line Line
Word Line
Control Gate
Floating Gate
N P N
Software and Services Group 3
3

4. NOR flash
Diagram © Wikipedia user 'Cyferz'
CC BY-SA 3.0
Bit Line
N N, GND N N, GND N N, GND N
P
Software and Services Group 4
4

5. NAND flash
Diagram © Wikipedia user 'Cyferz'
CC BY-SA 3.0
Bit Line
N N N N N N N N N N N
P
Software and Services Group 5
5

6. NAND flash: one string
I/O
Software and Services Group 6
6

7. NAND flash: one eraseblock
Page
I/O
I/O
16896 bits
per page
I/O
(2048 bytes
+ 64 bytes)
I/O
64 pages per eraseblock
Software and Services Group 7
7

8. Data loss on flash
• Charge loss over time
• Charge builds up in insulator
• Write disturb
• Read disturb
Software and Services Group 8
8

9. Programming NAND flash
Other Programmed Other
Page Page Page
+10V +20V +10V
+10V I/O
0V I/O
+10V I/O
0V I/O
Programmed cells
Software and Services Group 9
9

10. Write disturb
Other Programmed Other
Page Page Page
+10V +20V +10V
+10V I/O
0V I/O
+10V I/O
0V I/O
Disturbed cells
Software and Services Group 10
10

11. Read disturb
Other Read Other
Page Page Page
+5V 0V +5V
0V I/O
0V I/O
0V I/O
0V I/O
Disturbed cells
Software and Services Group 11
11

12. Single Level Cell
Voltage
Distribution of 0-level cells
Vt – threshold voltage
Distribution of 1-level cells
Software and Services Group 12
12

13. Multi Level Cell
Voltage
Distribution of (0,0) cells
Distribution of (1,0) cells
Distribution of (0,1) cells
Distribution of (1,1) cells
Software and Services Group 13
13

14. NAND electrical interface
CE RE WE
Address Latch Enable (ALE)
Command Latch Enable (CLE)
NAND
Ready/Busy (R/B) chip
D0-7
Software and Services Group 14
14

15. Coping with flash
• Error correction (ECC)
• Bad block management
• Programming pages sequentially
• Garbage collection
• Wear levelling
• Paired pages?
Software and Services Group 15
15

16. Flash Translation Layers
1
2
3
4
5
6
7
8
Software and Services Group 16
16

17. Flash Translation Layers
1 1
2 2
3 3
4 4
5 chain 5
6 6
7 7
8 8
Write sectors 1-4
Software and Services Group 17
17

18. Flash Translation Layers
1 1 1
2 2 2
3 3 3
4 4 4
5 chain 5 chain 5
6 6 6
7 7 7
8 8 8
Write sectors 2, 4, 6
Software and Services Group 18
18

19. Flash Translation Layers
1 1 1
2 2 2
3 3 3
4 4 4
5 chain 5 chain 5
6 6 6
7 7 7
8 8 8
“Fold” chain into its final block
Software and Services Group 19
19

20. Flash Translation Layers
1 1
2 2
3 3
4 4
5 chain 5
6 6
7 7
8 8
Erase previous blocks in chain
Software and Services Group 20
20

21. Flash Translation Layers
1
2
3
4
5
6
7
8
Erase previous blocks in chain
Software and Services Group 21
21

22. TRIM
1 1 1
2 2 2
3 3 3
4 4 4
5 chain 5 chain 5
6 6 6
7 7 7
8 8 8
Software and Services Group 22
22

23. TRIM
1 1 1
2 2 2
3 3 3
4 4 4
5 chain 5 chain 5
6 6 6
7 7 7
8 8 8
TRIM sectors 1, 3
Software and Services Group 23
23

24. TRIM
1 1 1
2 2 2
3 3 3
4 4 4
5 chain 5 chain 5
6 6 6
7 7 7
8 8 8
“Fold” chain into its final block
Software and Services Group 24
24

25. TRIM
1 1
2 2
3 3
4 4
5 chain 5
6 6
7 7
8 8
Erase previous blocks in chain
Software and Services Group 25
25

26. TRIM
1
2
3
4
5
6
7
8
Erase previous blocks in chain
Software and Services Group 26
26

27. An alternative Flash Translation Layer
Sect #1
Sect #2
Software and Services Group 27
27

28. An alternative Flash Translation Layer
Sect #1
Sect #2
Sect #1
Sect #2
Sect #3
Sect #4
Software and Services Group 28
28

29. An alternative Flash Translation Layer
Sect #1
Sect #2
Sect #1
Sect #2
Sect #3
Sect #4
Sect #2
Sect #4
Sect #6
Software and Services Group 29
29

30. File system on top of file system
• Efficiency
– Hard to optimise for opaque lower file system
– Missed garbage collection opportunity
– Real file system knows more about data longevity
– Real file system knows about transactions
• Reliability
– Closed source file system implementation
– No access to underlying medium for fsck / recovery
– Often just implemented badly
Software and Services Group 30
30

31. MTD support in Linux
• Synchronous read/write
• Asynchronous erase
• ECC for NAND
• Bad Block management
Software and Services Group 31
31

32. UBI
http://www.linux-mtd.infradead.org/doc/ubidesign/ubidesign.pdf
• Static volumes
– Atomically overwritable
– Wear levelled
– CRC protected
• Dynamic volumes
– Writable flash
– Simple logical <-> physical mapping
– Deferred erase (unmap)
– Scrubbing
Software and Services Group 32
32

33. Ideal hardware wish list
• Fast queued DMA transfers
• Full error correction in hardware
– With reporting of corrected errors
– Scrubbing
• Page copy
Software and Services Group 33
33

34. Questions?
Software and Services Group 34
34