Stig: Social Graphs & Discovery at Scale


Published on

Stig is a distributed graph database built from scratch at Tagged. It handles our scale of data and user demands, which means 100M+ users and 6B+ page views per month. It is particularly suited to graph-based applications involving large volumes of data, transactional updates, and inference-driven queries.

The goal of the stig project is to increase the productivity of web programmers. To this end, the system hides the details of its distributed architecture and provides the application programmer a single, consistent, and reliable path to data. The query language is highly expressive and composable, but also easy to use and stocked with helpful libraries.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Stig: Social Graphs & Discovery at Scale

  1. 1. Social graphs and discovery. At scale.Jason Lucas, Scalability Architect, Tagged
  2. 2. Stig is...• A very large-scale non-SQL database... • But it speaks and can emulate SQL• A graph-oriented data store... • But can look like a key value store, relational tables, file system• A foundation for building general web applications... • But it particularly excels at social apps• A distributed system with a shared-nothing architecture... • But it gives developers an easy-to-manage path to data• A solution to complex problem of CAP-limited systems...
  3. 3. Part 1: Stig Project GoalsPart 2: Stig ConceptsPart 3: Lunch Workshop
  4. 4. Part 1: Stig Project Goals • Facilitate the developer. • Be like a good waiter • Easy should be easy, complex should be possible • Untangle some existing messes • Scale like crazy. (Without driving ops crazy.) • Go big • Go fast • Go smooth • Exceed expectations. • Enable previously unthinkable features
  5. 5. Facilitate the developer• Decrease the burden. • Provide a single path to data • Create a uniform representation available to multiple application languages • Reduce the need for “defensive programming”• Enforce consistency. • Re-introduce atomic transactions • Control assumptions with assertions• Promote correctness. • Provide a more robust data representation • Support unit testing
  6. 6. Facilitate the developer• Offer power in simplicity. • Offer a robust expression language • Describe effects rather than details of distribution• Above all else: • “I want to feel like Im doing a good job.”
  7. 7. Scale like crazy...• Use a distributed architecture. • Shard data over multiple machines • Use commodity hardware • Scale as linearly as possible • Use replicas to speed average access• Move queries to data. • Decompose queries by separating areas of concern • Farm sub-queries to shards which hold the relevant data • Use comprehensions instead of realizations when possible
  8. 8. Scale like crazy...• Build for the web. • Provide durable sessions • Allow clients to disconnect and reconnect at will • Continue running in the background• Increase concurrency. • Break large objects down into smaller ones • Escrow deltas around fields which are partitioned or contentious • Use assertions instead of locks to permit interleaving of operations
  9. 9. Without driving Ops crazy• Be highly available. • Replicate storage across multiple machines • Shift responsibilities between machines transparently • Bring machines back into service transparently• Tolerate partitioning. • Fall back transparently to lower levels of service • Reconcile database automatically when partitions rejoin
  10. 10. Without driving Ops crazy• Simplify maintenance. • Tolerate unreliable hardware • Make software upgrades easy to manage • Be flexible with regard to physical topology • Make system status, performance, and capacity easy to measure and comprehend • Degrade gracefully under load • To the greatest degree possible, make the system maintain itself
  11. 11. Exceed expectations• Enable previously unthinkable features. • Don’t include histories in your schemas; the database keeps histories • Design apps with real-time, multi-user communications; database sessions are “chatty” • Feel free to compute Erdős Numbers or routes to Kevin Bacon • Test for the existence of interesting data states in constant time, not log time • Execute queries in time proportionate to the size of the answer, not the size of the database
  12. 12. Exceed expectations• Decrease development cycle time. • Build working apps on your desktop; the database can be simulated • Evolve your schema at will; the database doesn’t make a distinction between data and metadata • Use any language you like; the database looks the same from all clients
  13. 13. Part 2: Stig Concepts• Representing Graphs• Deconstructing Commits• Making Time Flow• Finding Meaning• Querying
  14. 14. Representing Graphs... Without Stig• Graphs in Tables. • Walks spread outward in waves • Self-joins proliferate• Graphs Key-Value Stores. • Generally node-centric • Edges are denormalized conjugate sets • Non-transactional multi-set is deadly• Graphs in XML Stores. • Floating chunk syndrome • Worst of both worlds• Graphs in Doc & Graph Stores. • Typeless, interned at nodes
  15. 15. Representing Graphs... With Stig Locations, Nodes & Edges /user/ /user/bob@baz.gak pets owns pets person player player person mafia mafia player player
  16. 16. Deconstructing Commits... Without Stig•Two States. • Uncommitted: only me • Committed: everybody else • One sandbox per connection• Variable Isolation. • High isolation limits concurrency • Low isolation hard to cope with• Two Guarantees. • Written to disk • Ephemeral• Some NoSQL Options. • No transactional integrity • Post-hoc reconciliation
  17. 17. Deconstructing Commits... With Stig• Private. • Only me, but I get as many as I want; maybe ephemeral• Shared. • Restricted scope, rapid communication; maybe ephemeral• Global. • A singleton, same as commit• Guarantees • Self-consistent • Replicated in data center • Written to disks • Replicated to other data centers
  18. 18. Deconstructing Commits...With Stig Points of View in Diplomacy Alice Bob Carol (Private) (Private) (Private) Alice/Bob Alliance (Shared) Diplomacy Game (Shared) (Global)
  19. 19. Making Time Flow... Without Stig• Time Flows Naturally. • System clock is OK• Execution Time ≈ Query Time. • A query made after an update will see the results of the update because time flow is linear • The order of events is definite• Locks Enforce Consistency. • Updates block each other• MVCC in Lieu of Locks. • Reads are writes • Collisions are rollbacks
  20. 20. Making Time Flow... With Stig• Time is Uncertain. • Distributed machines cannot rely on their system clocks• Declared Dependencies. • Each query declares its predecessors, so causality is a graph • The order of events is unknowable, but any topological sort of the graph is OK• Assertions Enforce Consistency. • MVCC with Paxos facilitates time travel • Query: seek a time in the past at which assertions are true • Update: seek a time in the future at which assertions are still true
  21. 21. Making Time Flow... With Stig Checkout Time Enter Credit Card Display Request Confirm Shopping Gift Wrap Order Cart Specify Shipping Update Qty. of Item
  22. 22. Finding Meaning... Without Stig• Tables & Views. • Tables store the base data • Views collect data from tables and other views • Views often present performance bottlenecks• Analysis Belongs to Data Definition. • Adding or changing a view or index is a schema change • Programmers must work with DBAs, limiting individual initiative • Changes have the potential to degrade the data service as a whole
  23. 23. Finding Meaning... With Stig• Asserted & Inferred Edges. • Asserted edges store the base data • Inferred edges collect data from asserted and inferred edges • Inference is distributed, on-going, and subject to time- travel• Analysis Belongs to Program Definition. • Inference rules aren’t “special” • Programmers can invent as they like • Scope of risk is limited
  24. 24. Finding Meaning... With Stig Inferring Friends & Stalkers Alice Alice <a, ‘is friend of’ b>has friendship if <a, ‘has friendship’, x> and <b, ‘has friendship’, x> and a is not b; x is friend of <a, ‘is stalking’ b> if <a, ‘is friend of’, b> and a.age >= 18has friendship and b.age < 18; Bob Bob
  25. 25. Querying... Without Stig• SQL • Easy-to-use, commonly known, and mostly harmless • Suffers from poor composability and is useless as a general-purpose programming language• Map-Reduce, Erlang, etc. • Not so easy-to-use, not so commonly known, and capable of shooting you in the foot • Often requires knowledge of underlying distributed architecture and are still not front-runners as general- purpose programming languages
  26. 26. Querying... With Stig• Robust and General-Purpose Language. • Purely functional, lazily evaluated, and strictly, robustly typed • Pattern-oriented notation for describing walks across graph• Composability Rules. • Comprehensions of sequences form the foundation • Transformations of sequences (map, reduce, filter, zip, etc.) are the building blocks• Distributed Evaluation Rocks. • Queries are broken down and sent to the servers where they need to be • Evaluation occurs in parallel
  27. 27. Querying... With Stig• Compiled & Stored. • Queries compile down to machine code and get stored in the graph itself • Stored programs are subject to on-going analysis • Programs can call each other• Library-Driven. • Language fundamentals support construction of libraries • We can emulate other languages, such as LINQ and Python• Clients. • Currently Java, Perl, PHP, Python, and C/C++ • We can also serve HTTP directly
  28. 28. Querying... With Stig Mutual Friendso /* function definition */ mutual_friends x y = solve f: [ <x, ‘is friend of’, f>; <y, ‘is friend of’, f> ];o /* function application */ mutual_friends person@/users/alice person@/users/bob;o /* results */ [ { f = person@/users/carol }, { f = person@/users/dave } ];
  29. 29. Wrapup
  30. 30. Is your project...?• Graph-shaped? • Representing graphs as graphs (instead of as tables or key pairs) simplifies your life • Stig graphs are fat, meaning theyre really any number of simultaneous, intersecting graphs, so go nuts• Transactional? • Reliably atomic state transitions also simplify your life • Asynchronous transaction management makes it more tolerable• Real-time? • Control the influence of updates with shared points-of- view • Never be blocked waiting for the database to respond
  31. 31. Is your project...?• Really huge? • The store scales very close to linearly, so more data just means more machines • The size of the cluster doesnt generally doesnt affect the performance of individual operations• Deeply analytic? • Use inferences to describe relations and conditions youre interested in • Build up arbitrarily complex libraries of inference to extract meaning from data
  32. 32. Open-sourcing this year!• About our Code. • Written in C++0x and Haskell, with Python for tools • Entirely unit-test driven and designed for easy adoption• Why Open Source? • We want to give back • We benefit first and most • Competitive advantage would be temporary anyway • Knowing it’s open keeps us on our toes • There’s more to do than we can do ourselves • We attract the kind of people we want to work with
  33. 33. Our doors are open• About Tagged • #3 in social networking and growing (100+ Million members) • Located in downtown SF, 10 Ten Places to work by San Francisco Business Journal • Profitable since 2008. We answer only to ourselves and our users
  34. 34. Our doors are open• About the Stig Team. • Five full-time engineers with backgrounds in compilers, databases, distributed systems, and AI • Interns year-round with opportunities to publish • And yes, were hiring!
  35. 35. Got ideas?• Contact us! • Sign up for Stig news at: • Follow the Tagged Dev Blog at: • Jason Lucas Architect of Scalable Infrastructure
  36. 36. Part 3: Lunch Workshop• But wait, there’s more!• Join us as we get our hands messy with food and take a deep dive into the Stig query language and the Stig API!• Lunch N Learn 01:15 PM - 02:15 PM