1.
2
Legal Notices and Disclaimers
This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS
OR IMPLIED, IN THIS SUMMARY.
Intel technologies’ features and benefits depend on system configuration and may require
enabled hardware, software or service activation. Performance varies depending on system
configuration. Check with your system manufacturer or retailer or learn more at intel.com.
 This sample source code is released under the Intel Sample Source Code License Agreement.
Intel, the Intel logo, Intel Core, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in
the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2017, Intel Corporation. All rights reserved.
Microsoft, Windows, and the Windows logo are trademarks, or registered trademarks of
Microsoft Corporation in the United States and/or other countries.

2.
3
Topics
• Introduction to Persistent Memory
• Introduction to Persistent Memory Developers Kit (PMDK)
• Working with PMDK
• Building PMDK
• Emulating Persistent Memory with DRAM
• Persistent Memory Programming with PMDK C++ Bindings
• Allocating Simple Data Structure in Persistent Memory
• Transactions
• Thread Synchronization
• Sample ( putting it all together)
• Container Enabling
• Summary
• Q&A

3.
4
What is Persistent
Memory?
• Byte Addressable
• Persistent Like
Storage
• Load /Store Access
• No page caching
• Cache Coherent
• Memory-like
Performance
Why Does it Matter now?
• Adds a new tier between
DRAM and Block Storage
(SSD/HDD)
• Larger Capacity, Higher
Endurance, Consistent low
latency
• Ability to do in-place
persistence
• No Paging, No Context
Switching, No Interrupts,
No Kernel Code Running
• Ability to do DMA & RDMA
Introduction to Persistent Memory
What’s the impact on the
Applications?
• Need Ways to Enable
Access to New High
Performance Tier
• May need to re-
architect to unlock the
new features and
performance

4.
5
New Type of Memory
• Persistent, Large Capacity & Byte Addressable
• 6 TB per two-socket system
• Cheaper than DRAM
• DIMM Capacities
• 128 GB
• Demonstrated at SAP Sapphire and Oracle Open World 2017
• Availability
• 2018
Intel Persistent Memory

5.
6
What does Re-architecting to Byte-Addressable
Persistent Memory Mean
• Applications Need to
• Ensure Data Persistence
• Stores are not guaranteed persistence until
flushed out of CPU caches
• Assure Data Consistency
• Maintain Transactional Consistency of
Persistent Data structures
• Preventing torn updates using Transactions
• Use Persistent Memory Aware Allocator
• Prevent Persistent Memory Leaks
• Detect and Handle Persistent Memory Errors
WPQ
ADR
-or-
WPQ Flush
(kernel only)
Core
L1 L1
L2
L3
W
P
Q
MOV
DIMM
CPUCACHES
CLWB + fence
-or-
CLFLUSHOPT
+ fence
-or-
CLFLUSH
-or-
NT stores +
fence
-or-
WBINVD
(kernel only)
Minimum Required
Power fail protected
domain:
Memory subsystem
Custom
Power fail protected
domain
indicated by ACPI
property:
CPU Cache Hierarchy

6.
NVDIMMs
7
Byte Addressable Storage With Memory Mapped
Files– Before and After
User
Space
Kernel
Space
Application
NVDIMM
NVDIMM
DRAM
… …
load/store
access
mmap()
msync()
User
Space
Kernel
Space
Application
NVDIMM
NVDIMM
DRAM
… …
Page cache
load/store
access
Avoid Overhead of Paging/Context
Switching into Kernel
Page cache
Before After
pmem-
Aware
File System
mmap()

7.
8
Topics
• Introduction to Persistent Memory Programming
• Persistent Memory Developers Kit (PMDK)
• Working with PMDK
• Building PMDK
• Emulating Persistent Memory with DRAM
• Deep Dive with C++ Bindings with Code Samples and Demos
• Summary

8.
9
The SNIA NVM Programming Model – Exposing Persistent
Memory to Applications
NVDIMMs
User
Space
Kernel
Space
Standard
File API
NVDIMM Driver
Application
File System
ApplicationApplication
Standard
Raw Device
Access
Load/Store
Management Library
Management UI
Standard
File API
pmem-Aware
File System
MMU
Mappings

9.
Support for
volatile
memory
usage
PMDK : A Suite of Open Source of Libraries
memkind
Low level
support for
local persistent
memory
libpmem
Low level
support for
remote access to
persistent
memory
librpmem
In Development
Link to Open Source :
http://pmem.io/pmdk/
NVDIMM
User
Space
Kernel
Space
Application
Load/Store
Standard
File API
pmem-Aware
File System
MMU
Mappings
Libraries
Interface to create
arrays of pmem-
resident blocks of
same size for atomic
updates
Interface for persistent
memory allocation,
transactions and
general facilities
Interface to create
a persistent
memory resident
log file
libpmemblklibpmemlog libpmemobj
Link to Intel Developer Zone:
https://software.intel.com/en-
us/persistent-memory
Support
Transactions
C++ C Java* Python*
Low-level support

10.
11
Why PMDK
• Simplifies/Facilitates Adoption of Persistent Memory Programming with Higher Level
Language Support
• Built on top of SNIA Programming Model
• Builds on DAX capabilities on Linux* and Windows*
• Data Access Granularity at Cache Line
• Provides API to
 Ensure Data Persistence
 Data Consistency Assurance
• Transactional Operations
• Power Fail Atomicity
 Allocate/Manage Persistent Memory
• Developer Tools For Debugging and Error Handling

11.
12
libpmemobj: Core Concepts
 Data Consistency Assurance
• Support for Transactional & Atomic Updates
 Implicit Persistence during Transactions
• Knows When and How to Flush the Data to Persistence Domain
 Persistent Memory Allocation & Management API
• Memory Mapped Region Exposed as Object Pools
• Thread-safe
• Allocate/Free Persistent Data Structures
• Position Independent
• Root Object
• Known Location in the Persistent Memory Pool
• Anchor for Objects in the Pool
 Multi-thread Support
Persistent
Memory Pool
Application
Pool Pool Pool
1 2 n
Pool
Root Object
Memory mapped files
Threads
text
data
BSS
heap
memory
mapped
stack
kernel
persistent
memory
Application
libpmem
Load/Store
memory-mapped pmem
libpmemobj
transactionsatomic
operations
allocatorlists
locks
A
P
I
A
P
I
A
P
I

12.
13
PMDK Binary Packages
• Download Pre-built Packages:
• https://github.com/pmem/pmdk/releases
• Works on 64-bit Linux* or Windows* 2016
• Upstreamed into multiple Linux Distros
• For example: Fedora* 24
• dnf install libpmemobj–devel
• Installs libpmem, libpmem-devel, libpmemobj, libpmemobj-devel
• C++ header files are not part of this install

13.
14
• git clone
• https://github.com/pmem/pmdk
• cd pmdk
• make
• make install (as root)
• Installs man pages and libraries
• Install libraries in non-standard
locations
• make install prefix=/xxx/xxx
• Customizing Build Settings
• Compile with persistent memory
aware Valgrind*
• make EXTRA_CFLAGS=-DUSE_VG_PMEMCHECK
Building Tech Preview PMDK Code from GitHub*

14.
15
Get Started with Persistent Memory Programming
without Real Hardware
• Persistent memory aware applications can be written/tested
• Using SSD or Hard Disk
• Without DAX
• Any Linux* or Windows* distro that supports Memory Mapped Files
• With DAX
• EXT4, XFS, NTFS
• Emulate Persistent Memory with DRAM
• https://software.intel.com/en-us/articles/how-to-emulate-persistent-
memory-on-an-intel-architecture-serverStart with Emulation; Validate & Tune on Platforms with Real DIMMs

15.
Emulating Persistent Memory Using DRAM (Linux*)
• Reserve PMEM through Kernel Configuration(kernel >4.2)
• Enable BIOS to treat Memory Marked as e820- type 12 as PMEM
• Enable Direct Access (DAX support)
• Identify Usable regions
• Specify Kernel Parameter (Grub file)
• memmap=<size of pmem>!<offset in DRAM>
• On reboot
• pmem region created
• Kernel offers this space as PMEM to the PMEM driver
• OS sees this DRAM region as PMEM & Creates
• /dev/pmem0, /dev/pmem1 ..
• Make File system
• EXT4 and XFS modified to support PMEM
• sudo mkfs.ext4 /dev/pmem0 or sudo mkfs.xfs /dev/pmem0
• Mount it on /dev/pmemx with DAX option
• Ex: sudo mount –o dax /dev/pmem0 /mnt/pmem/

16.
17
Persistent Memory Programming with C++
Bindings

17.
18
Sample Code
Download the sample code used in this presentation from GitHub* at
https://github.com/pmem/pmdk-examples/tree/master/cpp_queue

18.
19
Main Features of C++ Bindings
• Support for Transactions
• C++ Wrappers for
• Scoped Transactions
• Both Manual and Automatic
• Closure-like Transactions
• Support for Automatic Snapshotting of Basic Data Types in a Transaction
• p<> Property : Wrapper for basic types
• Indicates persistent variable
• Persistent Pointer : C++ Wrapper with pointer semantics
• Designed to handle complex classes
• Support for Allocating/Deallocating functions
• Transactions & Atomic API
• Does not work with polymorphic types

19.
20
Topics
• Introduction to Persistent Memory Programming
• Introduction to PMDK
• Persistent Memory Programming with PMDK C++ Bindings
• Deep Dive on C++ Bindings with Code Samples & Demos
• Transactions
• P<>
• Persistent_ptr<>
• Synchronization
• Summary

20.
21
Video: Code Sample Introduction
This video provides an introduction to the sample queue application that will
be made persistent memory-aware.
Click this link: https://software.intel.com/en-us/videos/introduction-to-the-
pmem-with-c-bindings-hello-world-code-demonstration.

21.
Transactions
22

22.
23
Intro to Transactions
• Undo log based transactions
• In case of interruption it is rolled-back or completed upon next pool open
• ACID like properties
• Can be nested
• Locks are held until the end of a transaction

23.
24
Manual Transaction Example
auto pop = pool<root>::open("/path/to/poolfile", "layout string");
{
transaction::manual(pop, persistent_mtx, persistent_shmtx);
// do some work...
transaction::commit();
}
auto aborted = transaction::get_last_tx_error();

24.
25
Manual Transactions
• Based on the familiar RAII concept
• Fairly easy to use
• Explicit transaction commit because of std::uncaught_exception
• Does not throw en exception on transaction abort
• By default aborts to account for third-party exceptions or amnesia
• Accepts an arbitrary number of (persistent memory resident) locks

25.
26
Automatic Transaction Example
auto pop = pool<root>::open("/path/to/poolfile", "layout string");
try {
transaction::automatic(pop, persistent_mtx, persistent_shmtx);
// do some work...
} catch (...) {
// do something meaningful
}
auto aborted = transaction::get_last_tx_error();

26.
27
Automatic Transactions
• Functionally and semantically almost identical to the manual transaction
• No explicit transaction commit
• Need C++17
• Relies on std::uncaught_exceptions

27.
28
Closure Transaction Example
auto pop = pool<root>::open("/path/to/poolfile", "layout string");
transaction::exec_tx(pop, [] {
// do some work...
}, persistent_mtx, persistent_shmtx);
auto aborted = transaction::get_last_tx_error();

28.
29
Closure Transactions
• Take an std::function object as transaction body
• No explicit transaction commit
• Available with every C++11 compliant compiler
• Throw an exception when the transaction is aborted
• Take an arbitrary number of locks
• Unfortunately at the very end

29.
30
Video: Modify Queue to Use Transactions
This video demonstrates how to use transactions to safely modify our queue in
a persistent memory context.
Click this link: https://software.intel.com/en-us/videos/use-transactions-to-
modify-our-persistent-memory-queue.

30.
Resides in persistent memory

31.
32
The p<> Property
• AKA the workhorse
• Overloads operator= for snapshotting in a transaction
• Overloads a bunch of other operators for seamless integration
• Arithmetic
• Logical
• Should be used for fundamental types
• No convenient way to access members of aggregate types
• No operator. to overload

32.
33
Video: The p<> property
In this video, we put the p<> template to work in our queue example.
Click this link: https://software.intel.com/en-us/videos/use-the-p-template-in-
our-queue-example.

33.
35
Persistent Smart Pointer
• Points to objects within a persistent memory pool
• Manages translating persistent addresses to runtime addresses
• Is a random access iterator
• Has primitives for flushing contents to persistence
• std::allocator friendly

34.
36
Persistent Smart Pointer
• Does not manage object lifetime
• Does not automatically add contents to the transaction
• But it does add itself to the transaction
• Does not point to polymorphic objects
• No good way to rebuild runtime state after pool reboot

35.
37
Transactional Allocation Functions
• Can be used only within transactions
• Use transaction logic to enable allocation/delete rollback of persistent
state
• make_persistent calls appropriate constructor
• Syntax similar to std::make_shared
• delete_persistent calls the destructor
• Not similar to anything found in std

36.
38
Video: Using the Persistent Pointer
In this video, we use the persistent pointer to safely manage queue data.
Click this link: https://software.intel.com/en-us/videos/use-the-persistent-
pointer-in-queue-example.

37.
Persistent memory resident

38.
40
Persistent Memory Synchronization
• Types:
• mutex
• shared_mutex
• timed_mutex
• condition_variable
• All with an interface similar to their std counterparts
• Auto reinitializing
• Can be used with transactions

39.
41
• Webinar Code Sample:
• https://github.com/pmem/pmdk-
examples/tree/master/cpp_queue
• PMDK Resources:
• Intel® Developer Zone:
https://software.intel.com/persistent-
memory
• pmem.io - http://pmem.io/
• Google group:
https://groups.google.com/forum/#!foru
m/pmem
• Source : https://github.com/pmem/pmdk
• Additional Resources
• https://www.usenix.org/system/files/l
ogin/articles/login_summer17_07_ru
doff.pdf
• An overview of persistent memory
programming
• http://www.snia.org/PM
• SNIA Standards Portfolio - NVM
Programming Model (NPM)
• https://www.snia.org/tech_activities/st
andards/curr_standards/npm
Summary