PNUTS is Yahoo!'s scalable, highly available distributed database system for hosting web applications. It provides record-level operations and asynchronous consistency across geographically distributed data centers. The system architecture uses a distributed hash table for data storage and retrieval. Consistency is achieved through a per-record timeline model and a message broker for replication. PNUTS supports flexible schemas, queries, and bulk loading while providing high performance and availability.

1. PNUTS: Yahoo!’s Hosted Data Serving Platform B.F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H. Jacobsen, N. Puz, D. Weaver and R. Yerneni Yahoo! Research Seminar Presentation for CSE 708 by Ruchika Mehresh Department of Computer Science and Engineering 22 nd February, 2011

4. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

6. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

8. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

9. System Architecture Animation

10. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

13. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

18. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

20. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

24. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

27. Experiments Zipfian Distribution

28. Experiments

30. PNUTS Data Storage and Retrieval Features Data and Query Model System Architecture Consistency (Yahoo! Message Broker) Query Processing Experiments Recovery Structure Future Work

48. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Current version Stale version Stale version Read

49. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Read up-to-date Current version Stale version Stale version

50. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Read ≥ v.6 Current version Stale version Stale version Read-critical(required version):

51. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Write Current version Stale version Stale version

52. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Write if = v.7 ERROR Current version Stale version Stale version Test-and-set-write(required version)

53. Consistency model Time v. 1 v. 2 v. 3 v. 4 v. 5 v. 7 Generation 1 v. 6 v. 8 Write if = v.7 ERROR Current version Stale version Stale version Back Mechanism: per record mastership

54. What is PNUTS? Parallel database Geographic replication Structured, flexible schema Hosted, managed infrastructure E 75656 C A 42342 E B 42521 W C 66354 W D 12352 E F 15677 E E 75656 C A 42342 E B 42521 W C 66354 W D 12352 E F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E

55. Storage units Routers Tablet controller REST API Clients Message Broker Detailed architecture Data-path components

56. Storage units Routers Tablet controller REST API Clients Local region Remote regions YMB Detailed architecture

57. Accessing data SU SU SU Get key k 1 2 Get key k 3 Record for key k 4 Record for key k

58. Bulk read SU SU SU Scatter/ gather server 1 {k 1 , k 2 , … k n } 2 Get k 1 Get k 2 Get k 3

59. Range queries Router Apple Avocado Banana Blueberry Canteloupe Grape Kiwi Lemon Lime Mango Orange Strawberry Tomato Watermelon Storage unit 1 Storage unit 2 Storage unit 3 Grapefruit…Pear? Grapefruit…Lime? Lime…Pear? MIN-Canteloupe SU1 Canteloupe-Lime SU3 Lime-Strawberry SU2 Strawberry-MAX SU1 SU1 Strawberry-MAX SU2 Lime-Strawberry SU3 Canteloupe-Lime SU1 MIN-Canteloupe

60. Updates Write key k Sequence # for key k Sequence # for key k Write key k SUCCESS Write key k Routers Message brokers 1 2 Write key k 7 8 SU SU SU 3 4 5 6

61. Asynchronous replication Back