Speaker: Poorna Chandra from Cask Big Data Applications Meetup, 06/23/2015 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to talk: https://www.youtube.com/watch?v=SToVcHRyZiM About the talk: At Salesforce, we are building a new service, code-named Webhooks, that enables our customers' own systems to respond in near real-time to system events and customer behavioral actions from the Salesforce Marketing Cloud. The application should process millions of events per day to address the current needs and scale up to billions of events per day for future needs, so horizontal scalability is a primary concern. In this talk, Poorna Chandra from Cask will discuss how Webhooks is built using HBase for data storage and Cask Data Application Platform (CDAP), an open source framework for building applications on Hadoop.

1. Poorna Chandra
Software Engineer, Cask
1
Near-real time event processing with guaranteed delivery
of HTTP callbacks
Webhooks

2. ● Webhooks use case
● Implementation in CDAP
● Q&A
Overview
2

3. ● Webhooks is a near-real time event delivery platform with guaranteed delivery
○ Subscribers generate events by engaging with messages
○ Deliver events to customers over HTTP within seconds
○ Customers react to events in near real time
What is Webhooks?
3

4. ● A purchase receipt email fails to be delivered
● A mail bounce event is pushed to a service hosted by the retailer
● Retailer’s customer service is immediately aware of the failure
Example use cases
4

5. 1. Process a stream of near real time events based on customer defined actions.
2. Guarantee delivery of processed events emitted to third party systems.
General problem statement
5

6. ● High data integrity
○ Commerce, health, and finance messaging subject to government regulation
● Horizontal scalability
● Short time to market
○ Accessible developer experience
○ Existing Hadoop/YARN/HBase expertise and infrastructure
● Open Source
Primary concerns
6

7. ● Some events need pieces of information from other event streams
○ Example: An email click needs the email send event for contextual information
● Wait until other events arrive to assemble the final event
● Join across streams
● Configurable TTL to wait to join (optional)
Implementation concern - Joins
7

8. ● Configurable per customer endpoint
○ Retry
○ Throttle
○ TTL to deliver (optional)
● Reporting metrics, SLA compliance
Implementation concern - Delivery guarantees
8

9. High level architecture
Ingest
Join
Route
Store
HTTP POST
Kafka
Source
External
System
9

10. public class EventRouter {
private Map<EventType, Route> routesMap;
public void process(Event e) {
Route route = routesMap.get(e.clientId());
if (null != route) {
httpPost(e, route);
}
}
}
Business logic
10

11. public class EventJoiner {
private Map<JoinKey, SendEvent> sends;
public void process(ResponseEvent e) {
SendEvent send = sends.get(e.getKey());
if (null != send) {
Event joined = join(send, e);
routeEvent(joined);
}
}
}
Business logic
11

12. ● Scaling data store is easy - use HBase
● Scaling application involves
○ Transactions
○ Application stack
○ Lifecycle management
○ Data movement
○ Coordination
How to scale?
12

13. 13

14. ● An open source framework to build and deploy data applications on
Apache™ Hadoop®
● Provides abstractions to represent data access and processing
pipelines
● Framework level guarantees for exactly-once semantics
● Transaction support on HBase
● Supports real time and batch processing
● Built on YARN and HBase
Cask Data Application Platform (CDAP)
14

15. Webhooks in CDAP
15

16. Business logic
public class EventJoiner {
private Map<JoinKey, SendEvent> sends;
public void process(ResponseEvent e) {
SendEvent send = sends.get(e.getKey());
if (null != send) {
Event joined = join(send, e);
routeEvent(joined);
}
}
}
16

17. Business logic in CDAP - Flowlet
public class EventJoiner extends AbstractFlowlet {
@UseDataSet(“sends”)
private SendEventDataset sends;
private OutputEmitter<Event> outQueue;
@ProcessInput
public void join(ResponseEvent e) {
SendEvent send = sends.get(e.getKey());
if (send != null) {
Event joined = join(e, send);
outQueue.emit(joined);
}
}
}
17

18. public class EventJoiner extends AbstractFlowlet {
@UseDataSet(“sends”)
private SendEventDataset sends;
private OutputEmitter<Event> outQueue;
@ProcessInput
public void join(ResponseEvent e) {
SendEvent send = sends.get(e.getKey());
if (send != null) {
Event joined = join(e, send);
outQueue.emit(joined);
}
}
}
Access data with Datasets
18

19. Chain Flowlets with Queues
public class EventJoiner extends AbstractFlowlet {
@UseDataSet(“sends”)
private SendEventDataset sends;
private OutputEmitter<Event> outQueue;
@ProcessInput
public void join(ResponseEvent e) {
SendEvent send = sends.get(e.getKey());
if (send != null) {
Event joined = join(e, send);
outQueue.emit(joined);
}
}
}
19

20. Tigon Flow
Event Joiner
Flowlet
HBase Queue HBase Queue
Start Tx End Tx
Start Tx
End Tx
Event Router
Flowlet
● Real time streaming processor
● Composed of Flowlets
● Exactly-once semantics
HBase Queue
20

21. Scaling Flowlets
Event Joiner
Flowlets
Event Router
Flowlets
HBase Queue
YARN
Containers
FIFO
Round Robin
Hash Partitioning
21

22. Summary
● CDAP makes development easier by handling the overhead of
scalability
○ Transactions
○ Application stack
○ Lifecycle management
○ Data movement
○ Coordination
22

23. Datasets and Tephra
23

24. Data abstraction using Dataset
● Store and retrieve data
● Reusable data access patterns
● Abstraction of underlying data storage
○ HBase
○ LevelDB
○ In-memory
● Can be shared between Flows (real-time) and MapReduce (batch)
24

25. ● Transactions make exactly-once semantics possible
● Multi-row and across HBase regions transactions
● Optimistic concurrency control (Omid style)
● Open source (Apache 2.0 License)
● http://tephra.io
Transaction support with Tephra
25

26. ● Used today in enterprise cloud applications
● CDAP is open source (Apache 2.0 License)
Use and contribute
http://cdap.io/
26

27. Q&A
Poorna Chandra
poorna@cask.co
http://cdap.io
27