- Hardware such as DRAM and NAND flash are facing scaling challenges as density increases, which could impact performance and cost. New non-volatile memory (NVM) technologies may provide opportunities to address these challenges but require software and system architecture changes to realize their full potential. Key considerations include persistence, performance, and programming models.

1. ( , 67
ts Odʼ’a
L P
( , 67

2. ( , 67
l s x– ~∼y
l s
r
v v
l s v p
l
k v x
y
o v
k C6
o ~∼ fi ~∼ v v v
l
k –BFp v
(

3. ( , 67
-‐‑‒ -‐‑‒ ʼ’mg
i L e
t
)
DRAM(and(Flash(Scaling:((
“The(End(is(Nigh”(
1985( 1990( 1995( 2000( 2005( 2010( 2015( 2020(
Density,
DRAM(
SLC(NAND(
M FdWTa 4FC BF( N

4. ( , 67
Summary(
Register((
Cache(
STT&RAM,,
NV&DIMM,
ns=class(NVM(
RRAM,,PCM,,
Low(us=class(NVM(
NAND,SSD,
High(us=class(NVM(
HDD(
Low(ms=class(Mass(Storage(
Capacity( 10(
”
v
k so ~∼
k s –~∼
M FdWTa 4FC BF( N

5. ( , 67
whg Ld
l v p
– fi t
l ” ~∼“ fi
k p p p r
k
l
l p ~∼
k p

6. ( , 67
ts
l ~∼ p” t
k ]G p v t
k v t
l 77EpA4A7 t
k v t
l v
k z v t ~∼ ~∼
k ” ~∼ x ~∼y
l sA4A7
,

7. ( , 67
v x CH F8y
V X oL
oL
-‐‑‒
2015 2030
vv v
hmd
fl
v
v fi
Lcndl
v
rr
r
rr
r

8. ( , 67
l (,
l )
l JSP v
.
Vaa 0 b Wa OW a U] X OW Z aO
W[ bZ S W[ bZ S Va[Z

9. CPU
NAND
RAM
•
•
•
•
RAM
“ ”

10. ( , 67
: : I
The$Machine$could$be$six$%mes$more$powerful$than$an$
equivalent$conven2onal$design,$while$using$just$1.25$
percent$of$the$energy$and$being$around$1/100$the$size.
h:p://www.hpl.hp.com/research/systems@research/themachine/

11. ( , 67
l “ •~∼
k B S 6][ baS C ]XSQa 9OQSP]] ”
k EOQ FQOZS 6][ baW U aSZ
k 8ea S[SZf FV W W U 6][ baW U 5
k GVS OQVW S C
k 9W S5]e H65
k 6GE 6] ] aWb[ G
l
k v x p p v y
k v
k v
l – p p p ~∼–

12. ( , 67 (
IoTpCPSpHPCp p
v
• v v v
• v v
• SoC (eFlash?)
• v v
• v v v v
• v
• DRAM

13. ( , 67
gin a
k
(
k Z]OR a] S
k v v
)
k v
)
NVM)SSD)Challenges)
• So:ware)overheads)in)
kernel)
7)
0
5
10
15
20
25
BaseLatency(us)
PCM
Ring
DMA
Wait
Interrupt
Issue
Copy
Schedule
OS/User
Software is Critical
• Baseline Latencies:
– Hardware: 8.2 us
– Software: 13.4 us
Hardware costs
11[Caulfield,)SC’10])
M6ObZjSZR F6 N
v

14. ( , 67
-‐‑‒ -‐‑‒
An application using NVM.FILE mode may or may not be using memory-mapped file
behavior.
The NVM.FILE mode describes NVM extensions including:
• Discovery and use of atomic write features
• The discovery of granularities (length or alignment characteristics)
4.3.3 NVM.PM.VOLUME mode overview
NVM.PM.VOLUME mode describes the behavior for operating system components (
file systems) accessing persistent memory. NVM.PM.VOLUME mode provides a soft
abstraction for Persistent Memory hardware and profiles functionality for operating sy
components including:
• the list of physical address ranges associated with each PM volume
• the capability to determine whether PM errors have been reported
Figure 5 NVM.PM.VOLUME and NVM.PM.FILE mode examples
Application
PM device PM device PM device. . .
User space
Kernel space
MMU
MappingsPM-aware file system
NVM PM capable driver
Load/
store
Native file
API
PM-aware kernel module
PM device
NVM.PM.VOLUME mode
NVM.PM.FILE mode
4.3.4 NVM.PM.FILE mode overview
NVM.PM.FILE mode describes the behavior for applications accessing persistent me
The commands implementing NVM.PM.FILE mode are similar to those using NVM.F
l AI 5Z]Q []RS
l AI 9WZS []RS
l CS W aS a S[] f I]Zb[S []RS
l CS W aS a S[] f 9WZS []RS
Note that there are other models for connecting a non-PM file system to PM hardware.
4.3 NVM programming modes
4.3.1 NVM.BLOCK mode overview
NVM.BLOCK and NVM.FILE modes are used when NVM devices provide block storage
behavior to software (in other words, emulation of hard disks). The NVM may be exposed as a
single or as multiple NVM volumes. Each NVM volume supporting these modes provides a
range of logically-contiguous blocks. NVM.BLOCK mode is used by operating system
components (for example, file systems) and by applications that are aware of block storage
characteristics and the block addresses of application data.
This specification does not document existing block storage software behavior; the
NVM.BLOCK mode describes NVM extensions including:
• Discovery and use of atomic write and discard features
• The discovery of granularities (length or alignment characteristics)
• Discovery and use of ability for applications or operating system components to mark
blocks as unreadable
Figure 4 NVM.BLOCK and NVM.FILE mode examples
Application
NVM block capable driver
File system
Application
NVM device NVM device
User space
Kernel space
Native file
API
NVM.BLOCK mode
NVM.FILE mode
4.3.2 NVM.FILE mode overview
NVM.FILE mode is used by applications that are not aware of details of block storage
hardware or addresses. Existing applications written using native file I/O behavior should work
FA 4 C ]U O[[W U ]RSZ I]Z

15. ( , 67
l v CS W aS a S[] f C
t
l – 4C p
C fi p 4C
k [OZZ]Q – fSa O ]aVS [OZZ]Q
l v v t
k
k v

16. ( , 67
Lcndl
,
6CH
AIE4 7E4
6CH
AIE4
7E4
6CH
AIE4
4 ES ZOQS RW 5 FVO SR ORR S OQS 6 8 aW SZf AIE4
5Z]Q 9
S[] f 9
QOQVS QOQVSQOQVS
AIE4 B AIE4

17. ( , 67
c mg M N
l AIE4
k Fa] S • AIE4 ~∼
k r r v
-‐‑‒
• Recovery depends on write ordering
CPU
Write-back
Cache
NVM
V D
VD
STORE data[0] = 0xFOOD
STORE data[1] = 0xBEEF
STORE valid = 1
Crash
D
D
CPU
Persistent Memory (PM) Ordering
M FdWTa 4FC BF( N

18. ( , 67
c mg MRN
l
k ”“ t
k aSZ CS W aS a S[] f v
l 6 9 HF BCG
l 6 J5 C6B G
l ” t
.
dering with Existing Hardware
er writes by flushing cachelines via CLFLUSH
CLFLUSH:
talls the CPU pipeline and serializes execution
STORE data[0] = 0xFOOD
STORE data[1] = 0xBEEF
CLFLUSH data[0]
CLFLUSH data[1]
STORE valid = 1
ata[0] ST CLFLUSH
CLFLUSHOPT
• Provides unordered version of CLFLUSH
• Supports efficient cache flushing
data[1]
valid
ST CLFLUSHOPT
ST
data[0] ST CLFLUSHOPT
time
data[1]
valid
ST CLFLUSH
ST
data[0] ST CLFLUSH
time 20
M FdWTa 4FC BF( N
aOZZ
h(

19. ( , 67
X
/
F]b QS0 GEF ( )
l u x p p p
r y
l – ~∼
k y •

20. ( , 67
X
l AI T WS RZf OQQS OaaS
l p
k J WaS SOR ~∼–
v •
l v
k BF
k v
(
MRAMDRAM
VM
$
VM $
$
ff
v
VMVMVM

21. ( , 67
S T
a
l )
k s
x yo o o o
k p – fi
l • p
l ff ~∼
k Q T Gb P] 5]] a
(
•~∼
p –
v

22. ( , 67
l C OdO S 9WZS Ff aS[
k C 9F M8b ]Ff N F6 9F MF6 N
5C9F MFBFC /N
l C WP O f
k AI SO M4FC BF N 677F MHF8A K N
S[] f S M4FC BF N 4aZO MBBCF 4 N
l 7OaOPO S
k 9B87HF MF : B7 N
l a b[S aOaW]
k AI F64I8A:8E M C7CF (N
((

23. ( , 67
l 4FC BF ( Gba] WOZ
k C ]U O[[W U O R H OUS ]RSZ T] A] I]ZOaWZS
S[] f
k Vaa 0 S SO QV Q dW Q SRb ] O aba] WOZ
l 6EB ( Gba] WOZ
k 7OaOQS aS FW[bZOaW] SaV]R]Z]UWS
k Vaa 0 S] ZS Rb S SRb hPQZ aba] WOZ R [
()

24. ( , 67
C
l W be
k 7E4 –~∼
k
–~∼
k K C SKSQbaW] CZOQS
l 74K 7W SQa 4QQS
k AIE4 v
–~∼
k K C
(
Linux – PDA
Agenda VR3 (2001)