Recommended
More Related Content
Similar to Rnotify
Similar to Rnotify (20)
Recently uploaded
Recently uploaded (20)
Rnotify
- 1. Rnotify A Scalable Distributed filesystems Notifications Solution for Applications Ashwin Raghav www.rnotifications.com github.com/ashwinraghav/rnotify-c/ 1 1Tuesday, April 30, 13
- 2. Agenda • Motivation • Problem Statement / State of the art • General Overview • Hypothesis • Approach • Evaluation • Conclusion 2 2Tuesday, April 30, 13
- 3. Motivation • Applications need File System Notifications • Previously applications polled file systems naively • Now,All Operating Systems provide FS Notifications API 3 3Tuesday, April 30, 13
- 4. Problem VFS is an abstraction to treat all filesystems uniformly All FS read/writes happen viaVFS - ideal place to implement notifications Does not work with Distributed File Systems 4 4Tuesday, April 30, 13
- 5. Problems / State of the art Use ad-hoc (polling) implementations for Distributed FS. Polling creates an unfortunate tension between resource consumption and timeliness Any general solution must be location transparent, scalable, tunable. Use inotify to subscribe to local filesystems 5 5Tuesday, April 30, 13
- 6. Requirements • Compatibility with existing applications that use Inotify • Provide Horizontal Scalability, Decomposition of Functionality, Tunable Performance • Location Transparency • High Throughput notifications per client 6 6Tuesday, April 30, 13
- 7. Assumptions • Relaxing Reliability Guarantees • Modifying Notification Semantics • Congestion Control Semantics • Failure Notification Semantics 7 7Tuesday, April 30, 13
- 8. Related Work • FAM (File Alteration Monitor) - does not scale • Internet scale systems like Thialfi, Zookeeper are built for larger scales of clients. • Bayeux, Scribe, Siena, Hermes, Swag etc assume overlay networks to establish multicast trees for message dissemination • Inotify was introduced in kernel 2.6.13 - for local FS notifications 8 8Tuesday, April 30, 13
- 10. Hypothesis As a result of clearly decomposing functionality into replicable components, Rnotify can be tuned to fit different notification workloads to consistently deliver notifications at low latency. 10 10Tuesday, April 30, 13
- 11. Key Properties • Low Latency Notifications (under 10ms) • Compatible with applications that use Inotify • Tuned to fit workloads • Greedy Applications can use Rnotify by distributing their workloads across nodes. 11 11Tuesday, April 30, 13
- 12. Approach • Registration • Notification • Replica Configuration Management 12 12Tuesday, April 30, 13
- 13. Registration • Inform the Proxy about the newly watched file • Place Registrations on preferred Publishers 13 13Tuesday, April 30, 13
- 14. • Client Driven Registration • Registration is transactional from the application ‘s point of view • Client Driven Migration of subscriptions Client Library & API usage 14 14Tuesday, April 30, 13
- 16. Notification Pipeline • Congestion Control • Opportunistic Batching • Publisher Selection 16 16Tuesday, April 30, 13
- 17. Dispatchers • Serialize notification blocks • Congestion Control • Dispatch to Publisher 17 17Tuesday, April 30, 13
- 18. Congestion Control at Dispatcher Subscription Id Number of notifications in Time window 1 1000 2 3000 Frequency List Frequency List Frequency List NOTIFICATION_BURST is sent to Publisher 18 18Tuesday, April 30, 13
- 19. Avoid atomic broadcasts Frequency List Frequency List Frequency List Frequency List 19 19Tuesday, April 30, 13
- 20. Publishers • Identify the subscribers for a notification • Dispatch to the subscribers 20 20Tuesday, April 30, 13
- 21. Representing State - Publisher Get all Subscribers Get all Notifications File Id IP address of Subscribers 1 192.168.1.2:3000 192.168.3.4:3001 2 192.168.1.2:3000 192.168.3.4:3001 Subscriber Undelivered Notifications 192.168.1.2:3000 N1, N2, N3 192.168.3.4:3001 N4, N5, N6 File Id Notifications 1 N1, N2, N3, 2 N4, N5 Append new Notification 21 21Tuesday, April 30, 13
- 22. Publisher Selection How do the dispatchers and Registrar maintain a shared understanding of ‘preferred’ publishers? 22 22Tuesday, April 30, 13
- 23. Partition and Placement of Publishers pos3 = SHA1(Publisher3_IP_ADDR) pos4 = SHA1(Publisher4_IP_ADDR) pos2 = SHA1(Publisher2_IP_ADDR) pos1 = SHA1(Publisher1_IP_ADDR) 23 23Tuesday, April 30, 13
- 24. Partition and Placement of Subscriptions file3 = SHA1(File_Path3) file4 = SHA1(File_Path4) file2 = SHA1(File_Path2) file1 = SHA1( File_Path1) 24 24Tuesday, April 30, 13
- 25. Arrival of Publisher new_publisher = SHA1(New_Pub_IP_Addr) Reissue_registrations_between(pos1, pos2) Lock free way to make configuration eventually consistent 25 25Tuesday, April 30, 13
- 26. Dispatcher Replication • Dispatcher is provided the registrar location at startup • It acquires the publisher list from the registrar transactionally. • Inform the Proxies independently 26 26Tuesday, April 30, 13
- 27. Evaluation Strategy Mid size GlusterFS deployment on EC2 Postmark Benchmark to represent FS activity Using Chef to startup serviced clients Measure Latency end to end 8xl machines with 32 cores each helped simulate several clients each All machines were acquired within a placement group 27 27Tuesday, April 30, 13
- 28. Evaluation - Scalability Tune Dispatchers based on FS throughput Tune Publishers based on number of clients 28 28Tuesday, April 30, 13
- 29. Scalability - Overactive FileSystems Post Mark threads writing to different directories29 29Tuesday, April 30, 13
- 30. Scalability - Overactive FileSystems PostMark threads writing to same directory30 30Tuesday, April 30, 13
- 31. PostMark threads writing to different files PostMark threads writing to same files Applications like web/mail server HPC applications Scalability - Overactive FileSystems 31 31Tuesday, April 30, 13
- 32. Scalability - Servicing many clients 32 32Tuesday, April 30, 13
- 33. Performance Demonstrate consistency Demonstrate footprint in comparison to naive polling 33 33Tuesday, April 30, 13
- 34. Performance - Consistency 34 34Tuesday, April 30, 13
- 35. Comparison to naive Polling • Developed a poller Node.js REST API • For just 100 clients , 5 files, 50000 stats per second • Has an extremely heavy footprint on the FS performance 35 35Tuesday, April 30, 13
- 36. Greedy Applications • Increasing the number of notifications delivered per client • Linear increase in latency • Messages spend more time in queues 36 36Tuesday, April 30, 13
- 37. Inotify - Inefficient Applications 37 37Tuesday, April 30, 13
- 38. Greedy Applications If you need to consume more notifications, Distribute yourself Inefficient Application 38 38Tuesday, April 30, 13
- 39. Summary - Why is this work different? • FAM does not scale and is obsolete. • All PubSub systems do not cater to many notifications per client • Multicast Trees are established for reliability (Performance suffers) • Pub Sub systems provide a richer set of semantics with lower performance 39 39Tuesday, April 30, 13
- 40. Future Work • Introduce a security model • Introduce message ordering • Provide message delivery reliability 40 40Tuesday, April 30, 13
- 41. Conclusion • Rnotify is a solution to receive notifications from POSIX compliant Distributed File Systems • Tuned to fit different notification workloads • Incrementally Scalable, location transparent and mimics Inotify • We have tested Rnotify to scale to 2.5 million notifications per second • Latency under 10ms for 88% notifications 41 41Tuesday, April 30, 13
- 43. Subscription Proxy • Resides on the File Host & Proxies subscriptions & notifications. • Idempotent API wrappers for subscription 43 43Tuesday, April 30, 13
- 44. Design Alternatives • File System Modification • VFS Modification • Modifying Inotify Implementation 44 44Tuesday, April 30, 13
- 45. Latency Tests - Zero 45 45Tuesday, April 30, 13
- 46. Throughput Tests - Zero 46 46Tuesday, April 30, 13