Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

C++ Programming and the Persistent Memory Developers Kit


Published on

Introduction to Persistent Memory
Introduction to Persistent Memory Developers Kit (PMDK)
Working with PMDK
Persistent Memory Programming with PMDK C++ Bindings

Published in: Technology
  • Login to see the comments

  • Be the first to like this

C++ Programming and the Persistent Memory Developers Kit

  1. 1. 2 Legal Notices and Disclaimers This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at  This sample source code is released under the Intel Sample Source Code License Agreement. Intel, the Intel logo, Intel Core, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Copyright © 2017, Intel Corporation. All rights reserved. Microsoft, Windows, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.
  2. 2. 3 Topics • Introduction to Persistent Memory • Introduction to Persistent Memory Developers Kit (PMDK) • Working with PMDK • Building PMDK • Emulating Persistent Memory with DRAM • Persistent Memory Programming with PMDK C++ Bindings • Allocating Simple Data Structure in Persistent Memory • Transactions • Thread Synchronization • Sample ( putting it all together) • Container Enabling • Summary • Q&A
  3. 3. 4 What is Persistent Memory? • Byte Addressable • Persistent Like Storage • Load /Store Access • No page caching • Cache Coherent • Memory-like Performance Why Does it Matter now? • Adds a new tier between DRAM and Block Storage (SSD/HDD) • Larger Capacity, Higher Endurance, Consistent low latency • Ability to do in-place persistence • No Paging, No Context Switching, No Interrupts, No Kernel Code Running • Ability to do DMA & RDMA Introduction to Persistent Memory What’s the impact on the Applications? • Need Ways to Enable Access to New High Performance Tier • May need to re- architect to unlock the new features and performance
  4. 4. 5 New Type of Memory • Persistent, Large Capacity & Byte Addressable • 6 TB per two-socket system • Cheaper than DRAM • DIMM Capacities • 128 GB • Demonstrated at SAP Sapphire and Oracle Open World 2017 • Availability • 2018 Intel Persistent Memory
  5. 5. 6 What does Re-architecting to Byte-Addressable Persistent Memory Mean • Applications Need to • Ensure Data Persistence • Stores are not guaranteed persistence until flushed out of CPU caches • Assure Data Consistency • Maintain Transactional Consistency of Persistent Data structures • Preventing torn updates using Transactions • Use Persistent Memory Aware Allocator • Prevent Persistent Memory Leaks • Detect and Handle Persistent Memory Errors WPQ ADR -or- WPQ Flush (kernel only) Core L1 L1 L2 L3 W P Q MOV DIMM CPUCACHES CLWB + fence -or- CLFLUSHOPT + fence -or- CLFLUSH -or- NT stores + fence -or- WBINVD (kernel only) Minimum Required Power fail protected domain: Memory subsystem Custom Power fail protected domain indicated by ACPI property: CPU Cache Hierarchy
  6. 6. NVDIMMs 7 Byte Addressable Storage With Memory Mapped Files– Before and After User Space Kernel Space Application NVDIMM NVDIMM DRAM … … load/store access mmap() msync() User Space Kernel Space Application NVDIMM NVDIMM DRAM … … Page cache load/store access Avoid Overhead of Paging/Context Switching into Kernel Page cache Before After pmem- Aware File System mmap()
  7. 7. 8 Topics • Introduction to Persistent Memory Programming • Persistent Memory Developers Kit (PMDK) • Working with PMDK • Building PMDK • Emulating Persistent Memory with DRAM • Deep Dive with C++ Bindings with Code Samples and Demos • Summary
  8. 8. 9 The SNIA NVM Programming Model – Exposing Persistent Memory to Applications NVDIMMs User Space Kernel Space Standard File API NVDIMM Driver Application File System ApplicationApplication Standard Raw Device Access Load/Store Management Library Management UI Standard File API pmem-Aware File System MMU Mappings
  9. 9. Support for volatile memory usage PMDK : A Suite of Open Source of Libraries memkind Low level support for local persistent memory libpmem Low level support for remote access to persistent memory librpmem In Development Link to Open Source : NVDIMM User Space Kernel Space Application Load/Store Standard File API pmem-Aware File System MMU Mappings Libraries Interface to create arrays of pmem- resident blocks of same size for atomic updates Interface for persistent memory allocation, transactions and general facilities Interface to create a persistent memory resident log file libpmemblklibpmemlog libpmemobj Link to Intel Developer Zone: us/persistent-memory Support Transactions C++ C Java* Python* Low-level support
  10. 10. 11 Why PMDK • Simplifies/Facilitates Adoption of Persistent Memory Programming with Higher Level Language Support • Built on top of SNIA Programming Model • Builds on DAX capabilities on Linux* and Windows* • Data Access Granularity at Cache Line • Provides API to  Ensure Data Persistence  Data Consistency Assurance • Transactional Operations • Power Fail Atomicity  Allocate/Manage Persistent Memory • Developer Tools For Debugging and Error Handling
  11. 11. 12 libpmemobj: Core Concepts  Data Consistency Assurance • Support for Transactional & Atomic Updates  Implicit Persistence during Transactions • Knows When and How to Flush the Data to Persistence Domain  Persistent Memory Allocation & Management API • Memory Mapped Region Exposed as Object Pools • Thread-safe • Allocate/Free Persistent Data Structures • Position Independent • Root Object • Known Location in the Persistent Memory Pool • Anchor for Objects in the Pool  Multi-thread Support Persistent Memory Pool Application Pool Pool Pool 1 2 n Pool Root Object Memory mapped files Threads text data BSS heap memory mapped stack kernel persistent memory Application libpmem Load/Store memory-mapped pmem libpmemobj transactionsatomic operations allocatorlists locks A P I A P I A P I
  12. 12. 13 PMDK Binary Packages • Download Pre-built Packages: • • Works on 64-bit Linux* or Windows* 2016 • Upstreamed into multiple Linux Distros • For example: Fedora* 24 • dnf install libpmemobj–devel • Installs libpmem, libpmem-devel, libpmemobj, libpmemobj-devel • C++ header files are not part of this install
  13. 13. 14 • git clone • • cd pmdk • make • make install (as root) • Installs man pages and libraries • Install libraries in non-standard locations • make install prefix=/xxx/xxx • Customizing Build Settings • Compile with persistent memory aware Valgrind* • make EXTRA_CFLAGS=-DUSE_VG_PMEMCHECK Building Tech Preview PMDK Code from GitHub*
  14. 14. 15 Get Started with Persistent Memory Programming without Real Hardware • Persistent memory aware applications can be written/tested • Using SSD or Hard Disk • Without DAX • Any Linux* or Windows* distro that supports Memory Mapped Files • With DAX • EXT4, XFS, NTFS • Emulate Persistent Memory with DRAM • memory-on-an-intel-architecture-serverStart with Emulation; Validate & Tune on Platforms with Real DIMMs
  15. 15. Emulating Persistent Memory Using DRAM (Linux*) • Reserve PMEM through Kernel Configuration(kernel >4.2) • Enable BIOS to treat Memory Marked as e820- type 12 as PMEM • Enable Direct Access (DAX support) • Identify Usable regions • Specify Kernel Parameter (Grub file) • memmap=<size of pmem>!<offset in DRAM> • On reboot • pmem region created • Kernel offers this space as PMEM to the PMEM driver • OS sees this DRAM region as PMEM & Creates • /dev/pmem0, /dev/pmem1 .. • Make File system • EXT4 and XFS modified to support PMEM • sudo mkfs.ext4 /dev/pmem0 or sudo mkfs.xfs /dev/pmem0 • Mount it on /dev/pmemx with DAX option • Ex: sudo mount –o dax /dev/pmem0 /mnt/pmem/
  16. 16. 17 Persistent Memory Programming with C++ Bindings
  17. 17. 18 Sample Code Download the sample code used in this presentation from GitHub* at
  18. 18. 19 Main Features of C++ Bindings • Support for Transactions • C++ Wrappers for • Scoped Transactions • Both Manual and Automatic • Closure-like Transactions • Support for Automatic Snapshotting of Basic Data Types in a Transaction • p<> Property : Wrapper for basic types • Indicates persistent variable • Persistent Pointer : C++ Wrapper with pointer semantics • Designed to handle complex classes • Support for Allocating/Deallocating functions • Transactions & Atomic API • Does not work with polymorphic types
  19. 19. 20 Topics • Introduction to Persistent Memory Programming • Introduction to PMDK • Persistent Memory Programming with PMDK C++ Bindings • Deep Dive on C++ Bindings with Code Samples & Demos • Transactions • P<> • Persistent_ptr<> • Synchronization • Summary
  20. 20. 21 Video: Code Sample Introduction This video provides an introduction to the sample queue application that will be made persistent memory-aware. Click this link: pmem-with-c-bindings-hello-world-code-demonstration.
  21. 21. Transactions 22
  22. 22. 23 Intro to Transactions • Undo log based transactions • In case of interruption it is rolled-back or completed upon next pool open • ACID like properties • Can be nested • Locks are held until the end of a transaction
  23. 23. 24 Manual Transaction Example auto pop = pool<root>::open("/path/to/poolfile", "layout string"); { transaction::manual(pop, persistent_mtx, persistent_shmtx); // do some work... transaction::commit(); } auto aborted = transaction::get_last_tx_error();
  24. 24. 25 Manual Transactions • Based on the familiar RAII concept • Fairly easy to use • Explicit transaction commit because of std::uncaught_exception • Does not throw en exception on transaction abort • By default aborts to account for third-party exceptions or amnesia • Accepts an arbitrary number of (persistent memory resident) locks
  25. 25. 26 Automatic Transaction Example auto pop = pool<root>::open("/path/to/poolfile", "layout string"); try { transaction::automatic(pop, persistent_mtx, persistent_shmtx); // do some work... } catch (...) { // do something meaningful } auto aborted = transaction::get_last_tx_error();
  26. 26. 27 Automatic Transactions • Functionally and semantically almost identical to the manual transaction • No explicit transaction commit • Need C++17 • Relies on std::uncaught_exceptions
  27. 27. 28 Closure Transaction Example auto pop = pool<root>::open("/path/to/poolfile", "layout string"); transaction::exec_tx(pop, [] { // do some work... }, persistent_mtx, persistent_shmtx); auto aborted = transaction::get_last_tx_error();
  28. 28. 29 Closure Transactions • Take an std::function object as transaction body • No explicit transaction commit • Available with every C++11 compliant compiler • Throw an exception when the transaction is aborted • Take an arbitrary number of locks • Unfortunately at the very end
  29. 29. 30 Video: Modify Queue to Use Transactions This video demonstrates how to use transactions to safely modify our queue in a persistent memory context. Click this link: modify-our-persistent-memory-queue.
  30. 30. Resides in persistent memory
  31. 31. 32 The p<> Property • AKA the workhorse • Overloads operator= for snapshotting in a transaction • Overloads a bunch of other operators for seamless integration • Arithmetic • Logical • Should be used for fundamental types • No convenient way to access members of aggregate types • No operator. to overload
  32. 32. 33 Video: The p<> property In this video, we put the p<> template to work in our queue example. Click this link: our-queue-example.
  33. 33. 35 Persistent Smart Pointer • Points to objects within a persistent memory pool • Manages translating persistent addresses to runtime addresses • Is a random access iterator • Has primitives for flushing contents to persistence • std::allocator friendly
  34. 34. 36 Persistent Smart Pointer • Does not manage object lifetime • Does not automatically add contents to the transaction • But it does add itself to the transaction • Does not point to polymorphic objects • No good way to rebuild runtime state after pool reboot
  35. 35. 37 Transactional Allocation Functions • Can be used only within transactions • Use transaction logic to enable allocation/delete rollback of persistent state • make_persistent calls appropriate constructor • Syntax similar to std::make_shared • delete_persistent calls the destructor • Not similar to anything found in std
  36. 36. 38 Video: Using the Persistent Pointer In this video, we use the persistent pointer to safely manage queue data. Click this link: pointer-in-queue-example.
  37. 37. Persistent memory resident
  38. 38. 40 Persistent Memory Synchronization • Types: • mutex • shared_mutex • timed_mutex • condition_variable • All with an interface similar to their std counterparts • Auto reinitializing • Can be used with transactions
  39. 39. 41 • Webinar Code Sample: • examples/tree/master/cpp_queue • PMDK Resources: • Intel® Developer Zone: memory • - • Google group:!foru m/pmem • Source : • Additional Resources • ogin/articles/login_summer17_07_ru doff.pdf • An overview of persistent memory programming • • SNIA Standards Portfolio - NVM Programming Model (NPM) • andards/curr_standards/npm Summary