This document proposes using a key-key-value store model to efficiently process graph data in the cloud for social networks. It describes how social network data exhibits locality that can be leveraged. An on-line graph partitioning algorithm is presented that assigns related user data and profiles to the same machines to reduce the number of connections needed. The key-key-value store model extends traditional key-value stores to also store connections between users. Experimental results show the on-line partitioning algorithm performs comparably to static algorithms while adapting to dynamic changes.

1. Key-Key-Value Stores for Efficiently Processing Graph Data in the CloudAlexander G. ConnorPanos K. ChrysanthisAlexandrosLabrinidisAdvanced Data Management Technologies LaboratoryDepartment of Computer ScienceUniversity of Pittsburgh

2. Data in social networksA social network manages user profiles, updates and connectionsHow to manage this data in a scalable way?Key-value stores offer performance under high loadSome observations about social networksA profile view usually includes data from a user’s friendsSpatial localityA friend’s profile is often visited nextTemporal localityRequests might ask for updates from several usersWeb pages might include pieces of several user profilesA single request requires connecting to many machines

3. Connections in a Social NetworkAlice

4. Leveraging LocalityCan we take advantage of the connections?What if we stored connected user’s profiles and data in the same place?Locality can be leveraged The number of connections is reducedUser data can be pre-fetchedWe can think of this as a graph partitioning problem…Partitions = machinesVertices = user profiles, including updateEdges = connectionsObjective: minimize the number of edges that cross partitions

5. Example – graph partitioningMany edges cross partitions

6. Accessing a vertex’s neighbors requires accessing many partitions

7. In a social network, requesting updates from followed users requires connecting to many machines

8. Far fewer edges cross partitions

9. Accessing a vertex’s neighbors requires accessing few partitions

10. In a social network, fewer connections are made and related user data can be pre-fetchedKey-Key-Value StoresOur proposed approach: extend the key-value modelData can be stored key-valuesUser profilesData can also be stored as key-key-valuesUser connections“Alice follows Bob”Use key-key-values to compute localityOn-line graph partitioning algorithmAssign keys to grid locations based on connectionsEach grid cell represents a data hostKeys that are related are kept together

11. OutlineIntroductionData in Social NetworksLeveraging LocalityKey-Key-Value StoresSystem ModelClient APIAdding a Key-Key-ValueLoad managementOn-line partitioning algorithmSimulation ParametersResultsConclusion

12. Address Table: Mapping Storea transactional, distributed hash table

13. maps keys to virtual machinesPhysical Layer: Physical machinescan be added or removed dynamically as demands changeLogical Layer: Virtual machinesOrganized as a square grid

14. Run the KKV store software

15. Manage replication

16. Can be moved between physical machines as neededApplication Layer: Client APImaintain client sessions

17. cached dataApplication SessionsAddress tableVirtual hostsPhysical hosts

18. Client API and SessionsClients use a simple API that includes the get, put and sync commandsData is pulled from the logical layer in blocksGroups of related keysThe client API keeps data in an in-memory cacheData is pushed out asynchronously to virtual nodes in blocksPush/pull can be done synchronously if requested by the clientOffers stronger consistency at the cost of performance

19. Adding a key-key-valueput(alice, bob, follows)The on-line partitioning algorithm moves Alice’s data to Bob’s node because they are connectedTwo users: Alice and BobWrite the data to that nodeWrite the same data to that nodeUse the Address Table to determine the virtual machine (node) that hosts Alice’s dataUse the address table to determine the node that hosts Bob’s dataAddress tablebob8,88,8alice1,1Virtual hostskv(bob, ...)...kkv(alice, bob, follows)kv(alice, ...)...kkv(alice, bob, follows)1,18,8

20. Once the split is complete, new physical machines can be turned onVirtual nodes can be transferred to these new machinesIf one node becomes overloaded, it can initiate a splitTo maintain the grid structure, nodes in the same row and column must also splitVirtual hostsSplitting a Node

21. OutlineIntroductionData in Social NetworksLeveraging LocalityKey-Key-Value StoresSystem ModelClient APIAdding a Key-Key-ValueLoad managementOn-line Partitioning AlgorithmSimulation ParametersResultsConclusion

22. On-line Partitioning AlgorithmRuns periodically in parallel on each virtual nodeAlso after a split or mergeFor each key stored on a nodeDetermine the number of connections (key-key-values) with keys on other nodesCan also be sum of edge weightsFind the node that has the most connectionsIf that node is different than the current nodeIf the number of connections to that node is greater than the number of connections to the current nodeIf this margin is greater than some thresholdMove the key to the other nodeUpdate the address tableDesigned to work in a distributed, dynamic settingNOT a replacement for off-line algorithms in static settings

23. Partitioning Example2,11,11,2NodeSum(Edges)1,1 02,1 21,2 1

24. Partitioning Example2,11,11,2

25. Experimental Parameters

26. Partitioning Quality Results% Edges in partitionVertices in graphOn-line partitions as well as Kernighan-Lin

27. Partitioning Performance ResultsVertices movedVertices in graphOn-line partitions 2x faster than Kernighan-Lin!

28. ConclusionsContributions:A novel model for scalable graph data stores that extends the key-value modelKey-key-valuestoreA high-level system designA novel on-line partitioning algorithmPreliminary experimental resultsOur proposed algorithm shows promise in the distributed, dynamic setting

29. What’s Ahead?Prototype system implementationJava, PostgreSQLPerformance Analysis against MongoDB, CassandraSensitivity AnalysisCloud Deployment

30. Thank You!AcknowledgmentsDaniel Cole, Nick Farnan, Thao Pham, Sean SnyderADMT Lab, CS Department, Pitt GPSA, Pitt A&S GSO, Pitt A&S PBC