![Message
consumer
group
• Similar
to
Message
groups
in
AcMveMQ
–
guaranteed
ordering
of
the
processing
of
related
messages
across
a
single
queue
–
load
balancing
of
the
processing
of
messages
across
mulMple
consumers
–
high
availability
/
auto-‐failover
to
other
consumers
if
a
JVM
goes
down
• Applicable
to
many
messaging
pub/sub
systems
like
ka]a,
rabbitmq
etc
8](https://image.slidesharecdn.com/apachehelixdevopslspeinmeetup06132015-150613131956-lva1-app6891/85/Apache-Helix-DevOps-LSPE-IN-Meetup-8-320.jpg)
Uploaded byShahnawaz Saifi
Apache Helix DevOps & LSPE-IN Meetup
This document provides an overview of Helix, an open source framework for building distributed systems. It describes Helix's architecture, programming model based around state machines, and how systems are defined and configured using Zookeeper. The document also provides examples of how LinkedIn uses Helix to manage various distributed data systems and services.
More Related Content
Similar to Apache Helix DevOps & LSPE-IN Meetup
![DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ](https://cdn.slidesharecdn.com/ss_thumbnails/byomq-160414230807-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache Helix DevOps & LSPE-IN Meetup
- 1. Simplifying Distributed Systems Shahnawaz Saifi, @shahsaifi h:p://www.linkedin.com/in/shahsaifi June’2015
- 2. Outline • Introduc)on • Architecture • How to use Helix • Helix usage • Q & A 2
- 3. Examples of distributed data systems 3
- 4. Lifecycle Single Node MulM node • ParMMoning • Discovery • Co-‐locaMon Fault tolerance • ReplicaMon • Fault detecMon • Recovery Cluster Expansion • Thro:le data movement • Re-‐distribuMon 4
- 5.
- 6.
- 7.
- 8. Message consumer group • Similar to Message groups in AcMveMQ – guaranteed ordering of the processing of related messages across a single queue – load balancing of the processing of messages across mulMple consumers – high availability / auto-‐failover to other consumers if a JVM goes down • Applicable to many messaging pub/sub systems like ka]a, rabbitmq etc 8
- 9.
- 10. We need high level primiMves. ApplicaMon Zookeeper • Filesystem • Lock • Ephemeral Zookeeper provides low level primiMves. ApplicaMon Framework Consensus System • Node • ParMMon • Replica • State • TransiMon 10
- 11.
- 12. Outline • IntroducMon • Architecture • How to use Helix • Helix usage 12
- 13.
- 14. Core concept 14
- 15.
- 16.
- 17.
- 18.
- 19. Helix Based System Roles 19
- 20.
- 21. Outline • IntroducMon • Architecture • How to use Helix • Helix usage 21
- 22. Helix based soluMon 1. Define 2. Configure 3. Run 22
- 23. Step 1, Define: State model definiMon • States – All possible states – Priority • TransiMons – Legal transiMons – Priority • Applicable to each parMMon of a resource • e.g. MasterSlave ConMnued… 23
- 24. Define: State model Builder = new StateModelDefinition.Builder(“MASTERSLAVE”); //Add states and their rank to indicate priority. builder.addState(MASTER, 1); builder.addState(SLAVE, 2); builder.addState(OFFLINE); //Set the initial state when the node starts builder.initialState(OFFLINE); //Add transitions between the states. builder.addTransition(OFFLINE, SLAVE); builder.addTransition(SLAVE, OFFLINE); builder.addTransition(SLAVE, MASTER); builder.addTransition(MASTER, SLAVE); ConMnued… 24
- 25. Define: constraints State Transi)on ParMMon Y Y Resource -‐ Y Node Y Y Cluster -‐ Y State Transi)on ParMMon M=1,S=2 -‐ ConMnued… 25
- 26. Define: constraints //static constraint builder.upperBound(MASTER, 1); // dynamic constraint builder.dynamicUpperBound(SLAVE, “R”); // Unconstrained builder.upperBound(OFFLINE, -1); ConMnued… 26
- 27. Define: parMcipant plug-‐in code 27
- 28.
- 29.
- 30. Step 3: Run 30
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. ExecuMon modes • Who controls what 36
- 37. Auto rebalance v/s Auto 37
- 38.
- 39.
- 40. Custom mode: handling failure • Custom code invoker – Code that lives on all nodes, but acMve in one place – Invoked when node joins/leaves the cluster – Computes new idealstate – Helix controller fires the transiMon without viola)ng constraints
- 41. Helix usage at LinkedIn Primary DB Data Change Events Standardi zaMon Standardi zaMon Standardi zaMon Standardi zaMon Standardi zaMon Search Index Standardi zaMon Standardi zaMon Graph Index Standardi zaMon Standardi zaMon Read Replicas Updates Databus Espresso 41
- 42. Summary • Helix: A Generic framework for building distributed systems • Modifying/enhancing system behavior is easy – AbstracMon and modularity is key • Simple programming model: declaraMve state machine 42
- 43. website h:p://helix.apache.org codeRepo h:p://github.com/apache/helix user user@helix.apache.org dev dev@helix.apache.org twi:er @apachehelix, @shahsaifi