Recommended
More Related Content
What's hot
What's hot (20)
Similar to Md sal clustering internals
Similar to Md sal clustering internals (20)
Recently uploaded
Recently uploaded (20)
Md sal clustering internals
- 2. MD-SAL Clustering Internals Moiz Raja, Technical Lead, Cisco Systems IRC: moizer #ODSummit
- 3. ▪ Abhishek Kumar ▪ Basheeruddin Ahmed ▪ Colin Dixon ▪ Harman Singh ▪ Kamal Rameshan ▪ Robert Varga ▪ Tony Tkacik My Collaborators 3 Tom Pantelis ▪ Luis Gomez ▪ Phillip Shea ▪ Radhika Hirannaiah ▪ and many more…
- 4. ▪ Architecture ▪ Modules ▪ Flows ▪ Diagnostics ▪ Questions Agenda
- 6. Subsystems 6 member-1 member-2 member-3 Distributed Data Store member-1 member-2 Remote RPC Connector
- 7. High Level Architecture Distributed Data Store Remote RPC Connector Persistence Remoting Clustering
- 8. Actor Systems 8 Distributed Data Store Remote RPC Connector Actor Hierarchy Configuration Dispatchers
- 9. Data Synchronization 9 Data store Synchronized Data Tree Raft for Distributed Consensus Remote RPC Synchronized RPC Registry Gossip for data distribution
- 10. #ODSummit Distributed Data Store Architecture
- 11. Accessing Remote Data 11 Client member-1 member-2
- 16. Module Based Shards 16 / /inventory /topology /toaster
- 17. HA 17 member-2 member-3 inventory – follower -1 inventory – follower - 2 Client member-1 DistributedDataStore inventory – leader
- 18. Raft Distributed Consensus 18 discoversnodewithhigherterm Follower Candidate Leader starts up/ recovers times out, starts elections receives votes from majority of nodes times out, restarts elections follower-2follower-1 leader Election Replication/Consensus
- 23. Invoking a Remote RPC 23 Consumer member-1 member-2 Provider
- 26. RPC Registry Replication - Gossip 26 version=1 version=2 modify change version Local bucket updates change version m1,v1 m2,v5 m3,v7 All buckets and their versions known to all members Every 1 second members send all known bucket versions to any one peer m1 m2 m3status m2 m3 m1 local versions higher – send update local versions lower – send status to sender
- 29. ▪ Some common messages ▪ Actor base classes ▪ The Protobuf messages used in Helium ▪ The Protobuf NormalizedNode serialization code ▪ The NormalizedNode streaming code ▪ Other miscellaneous utility classes sal-clustering-commons 29
- 30. ▪ Implementation of the Raft Algorithm on top of akka ▪ Uses akka-persistence for durability ▪ Provides a base class called RaftActor which when can be extended by anyone who wants to replicate state ▪ See sal-akka-raft-example which provides a simple implementation of a replicated HashMap sal-akka-raft 30
- 31. ▪ ConcurrentDOMDataBroker ▪ DistributedDataStore ▪ Implementation of the DOMStore SPI ▪ Shard built on top of RaftActor ▪ Creates Shards based on Sharding strategy ▪ Code for a client to interact with the Shard Leader sal-distributed-datastore 31
- 32. ▪ RemoteRpcProvider ▪ Default RPC Provider. Invoked when an RPC is not found in the local MD-SAL registry. ▪ Code for BucketStore which provides a mechanism to replicate state based on Gossip ▪ Code for RpcBroker which allows invoking a remote rpc sal-remoterpc-connector 32
- 34. Startup 34 DistributedConfigDataStoreProviderModule DistributedDataStore ShardManager Shard1 Shard Shard3 Shard4 createInstance ActorContextwaitTillReadyLatch create & waitTillReady
- 35. Recovery 35 Shard1 Shard Shard3 Shard4 ShardManager read last known state from disk ready waitTillReadyLatch countDown
- 36. ▪ Recovery must be complete ▪ All Shard Leaders must be known ▪ Three messages are monitored by ShardManager ▪Cluster.MemberStatusUp ▪Used to figure out the address of a cluster member ▪LeaderStateChanged ▪Used to figure out if a Follower has a different Leader ▪ShardRoleChanged ▪Use to figured out any changes in a Shard’s Role ▪ Waiting is not infinite, by default it lasts only 90 seconds but is configurable ▪ Will block config sub-system Waiting for Ready 36
- 39. Transactions 39 Client DistributedDataStore inventory – leader Client DistributedDataStore inventory – leader Local Transaction Remote Transaction member-1 member-1member-2
- 40. Local Transaction Optimization 40 LocalTransactionContext Shard - Leader write merge delete ready member-1
- 41. Remote Transaction Optimization 41 RemoteTransactionContext Shard Leader write merge delete ready write mod merge mod delete mod member-1 member-2
- 42. Transaction Rate Limiting 42 rate-limit = 100 Tx/Sec Tx Cohort Shard Leader member-2 20ms Tx Cohort 50ms Tx Cohort 15ms after rate-limit/2 transactions done…. new-rate-limit = 25 Tx/Sec
- 43. Operation Limiting 43 RemoteTransactionContext Shard Leader write merge delete write mod merge mod delete mod member-1 member-2 … … block
- 44. Commit Coordination 44 Shard Leader member-2 Shard CommitCoordinator Tx1 - ready Tx2 - ready Tx3 - ready Tx1 - commit Tx3 - commit Tx3 - abort Tx2 - commit Tx1 Tx2 Tx3
- 45. Managing the in-memory journal Replicated To All 45 Client leader follower-1 follower-2 commit transaction txn txn txn
- 46. Managing the in-memory journal Cluster member unavailable 46 Client leader follower-1 commit transaction txn txn txn txn txn txn txn txn
- 47. Data Change Notifications 47 Client leader follower-1 follower-2 commit transaction txn txn txn notify
- 49. Startup 49 RemoteRpcBrokerModule createInstance RpcManager RemoteRpcProvider RpcBroker RpcRegistry RemoteRpcImpl RpcListener
- 50. Default RPC Delegate 50 RpcManager SchemaContext DOMRpcProviderService read all rpc definitions registerImplementation(remoteRpcImpl)
- 52. Invoking a Remote RPC 52 RemoteRpcImpl invokeRpc RpcRegistry Route found? RpcBroker ExecuteRpc FooService throw Exception
- 53. Invoking a Remote RPC 53 RemoteRpcImpl Consumer Provider member-1 member-2 RpcBroker RpcRegistry invokeRpc invokeRpcfindRoute ExecuteRpc
- 55. Transaction Tracing 55 Created txn member-2-txn-9400 of type READ_WRITE on chain member-2-txn-chain-13 Client Server Tx member-2-txn-9400 read /(urn:opendaylight:inventory?... member-3-shard-inventory-operational: Creating transaction : shard-member-2-txn-9400 Tx member-2-txn-9400 Readying 1 transactions for commit Tx member-2-txn-9400 commit member-3-shard-inventory-operational: Readying transaction member-2-txn-9400 member-3-shard-inventory-operational: Committing transaction member-2-txn-9400 Tx member-2-txn-9400: commit succeeded Cluster Member Initiator Counter Transaction Type Module Data store type
- 56. Replication Tracing 56 Leader Sending AppendEntries to follower member-2-shard-topology-operational: AppendEntries [term=2, leaderId=member-1-shard- topology-operational, prevLogIndex=520, prevLogTerm=2, entries=[Entry{index=521, term=2}], leaderCommit=520, replicatedToAllIndex=-1] Follower handleAppendEntries: AppendEntries [term=2, leaderId=member-2-shard-topology-operational, prevLogIndex=520, prevLogTerm=2, entries=[Entry{index=521, term=2}], leaderCommit=520, replicatedToAllIndex=-1] handleAppendEntries returning : AppendEntriesReply [term=2, success=true, logLastIndex=521, logLastTerm=2, followerId=member-1-shard-topology-operational] handleAppendEntriesReply from member-2-shard-topology-operational: applying to log – commitIndex: 521, lastAppliedIndex: 520 handleAppendEntriesReply - FollowerLogInformation for member-2-shard-topology-operational updated: matchIndex: 521, nextIndex: 522
- 57. Shard MBean 57 org.opendaylight.controller:type=DistributedOperationalDataStore,Category=Shards,name=member-1-shard-inventory-operational Operational Config member-1 member-2 member-3 default inventory topology operational config Attributes AbortTransactionsCount CommitIndex CommittedTransactionsCount CurrentTerm FailedTransactionsCount FollowerInfo FollowerInitialSync Status InMemoryJournalData Size InMemoryJournalLogSize LastApplied LastCommittedTransactionTime LastIndex LastTerm Leader RaftState ReadOnlyTransaction Count ReadWriteTransactionCount WriteOnlyTransaction Count VotedFor and more….
- 59. Data store GeneralRuntimeInfo MBean 59 org.opendaylight.controller:type=DistributedConfigDatastore,name=GeneralRuntimeInfo Operational Config Attributes • TransactionCreationRateLimit
- 60. Transaction Commit Rate MBean 60 org.opendaylight.controller.cluster.datastore:name=distributed-data-store.config.commit.rate Attributes • 50thPercentile • 75thPercentile • 90thPercentile • and so on… operational config • Count • Min • Max • StdDev
- 61. Data store GeneralRuntimeInfo MBean 61 org.opendaylight.controller:type=DistributedConfigDatastore,name=GeneralRuntimeInfo Operational Config Attributes • TransactionCreationRateLimit
- 62. Message Statistics MBean 62 org.opendaylight.controller.actor.metric:name=/user/shardmanager-config.msg-rate.ActorInitialized Attributes • 50thPercentile • 75thPercentile • 90thPercentile • and so on… operational config • Count • Min • Max • StdDev Message Name
- 64. RemoteRpcBroker MBean 64 org.opendaylight.controller:type=RemoteRpcBroker,name=RemoteRpcRegistry Attributes • BucketVersions • GlobalRpc • LocalRegisteredRoutedRpc Operations • findRpcByName • findRpcByRoute
- 65. Message Statistics MBean 65 org.opendaylight.controller.actor.metric:name=/user/rpc/registry.msg-rate.AddOrUpdateRoutes Attributes • 50thPercentile • 75thPercentile • 90thPercentile • and so on… • Count • Min • Max • StdDev Message Name
- 66. 66
- 67. ▪ Deploy a cluster ▪ Run clustering integration tests ▪ Write an application that works in the cluster ▪ Write bugs to report features which you find missing ▪ Try running dsBenchMark on a cluster ▪ Test out replication using the dummy data store ▪ Check out the code ▪ Send email to controller-dev@lists.opendaylight.org with questions Suggested Next Steps… 67
Editor's Notes
- There are 2 subsystems supported by the MD-SAL Clustering implementation. The distributed datastore which ensures that data stored in the datastore is accessible for all members of the cluster and which allows the MD-SAL data tree to be broken up into sub-trees so that it can be distributed around the cluster. The Remote RPC connector which ensures that an RPC implemented by a remote RPC provider is accessible from anywhere in the cluster.
- The MD-SAL clustering implementation is built on top of akka. Akka is a library which allows us to create autonomous components call actors which you can interact with by sending them asynchronous messages. Since Akka can ensure that only one message may be concurrently process by an actor it makes it easy to create thread safe components. Akka also has several modules which make it easy to build a clustered application or infrastructure on top of it. The distributed datastore for example uses akka-persistence. This is a module which helps the datastore persist state to disk and to allow it to recover based on that persisted state. Every modification that is made to the datastore gets persisted to disk and occasionally we create a snapshot of the state (tree) so that when we restart the data store we can recover the state faster. The Distributed Data Store and Remote RPC Connector both use akka-remoting. This module is used for one cluster member to send a message to an actor on a remote cluster member. This is an essential building block of our clustering implementation. Remoting in turn uses netty. Finally both the Distributed Data Store and Remote RPC connector use akka-clustering to discover new members in an akka cluster so that they can then communicate with their peers on other members.
- Distributed Data Store and Remote RPC Connector both use their own akka Actor System. An akka actor system consist of three things, An actor hierarchy. It’s called a hierarchy because actors can have supervisors and children. Failures flow up the supervision chain. An actor can also delegate some work to it’s children so that more work can be done in parallel. Configuration and Dispatchers which schedule messages to be delivered to an actor.
- The Distributed Data Store is a strongly consistent store – this strong consistency is ensured by the use of the Raft Distributed Consensus algorithm which requires that all data modifications are initiated by a Leader which replicates data to it’s followers – thus ensuring that the order in which modifications are done on all cluster members is the same. The Remote RPC connector also replicates data. It replicates all the registered Rpcs on each node to all the other nodes in the cluster.
- RpcBroker executes Rpc requests received from remote notes RpcRegistry manages the replication of the rpc registry – it extends BucketStore RemoteRpcImpl is the default delegate which is invoked when an rpc registration is not found in the md-sal core RpcBrokers’ registry RpcListener receives notifications whenever a rpc is registered by a Provider