Stream Processing using Samza SQL

1. Samza SQL
Srinivasulu Punuru

2. Agenda
1 What is Samza SQL?
2 Why SQL on Samza?
3 How does it work?
4 Demo
5 Q&A

4. What is Samza SQL?

5. Samza SQL by Example
Count page views of each member in a five minute window.
Send the result to kafka topic PageViewCount.

6. Samza low level task API
Repartitioner Job
public class PageViewRepartitioner implements StreamTask {
SystemStream outputStream = new SystemStream("kafka", "pvMemberId");
@Override
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {
PageViewEvent pageViewEvent = (PageViewEvent) envelope.getMessage();
String key = pageViewEvent.getMemberId();
OutgoingMessageEnvelope outMessage = new OutgoingMessageEnvelope(outputStream, pageViewEvent, key, pageViewEvent);
collector.send(outMessage);
}
}

7. Samza low level task API (contd.)
Page view counter
job
public class PageViewCounter implements StreamTask {
SystemStream outputStream = new SystemStream("kafka", "pageviewCount");
private Instant lastTriggerTime = Instant.now();
private HashMap<String, Integer> counter = new HashMap<>();
@Override
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) {
PageViewEvent pageViewEvent = (PageViewEvent) envelope.getMessage();
String memberId = pageViewEvent.getMemberId();
counter.put(memberId, counter.getOrDefault(memberId, 0) + 1);
if (Duration.between(lastTriggerTime, Instant.now()).toMinutes() > 5) {
counter.forEach((key, value) -> collector.send(new OutgoingMessageEnvelope(outputStream, key, value)));
counter.clear();
}
}
}

8. Samza high level API
public class PageViewCountApplication implements StreamApplication {
@Override public void init(StreamGraph graph, Config config) {
MessageStream<PageViewEvent> pageViewEvents = graph.getInputStream("pageView" );
MessageStream pageViewCount = graph.getOutputStream("pageViewCount" );
pageView
.partitionBy(m -> m.memberId)
.window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5),
initialValue, (m, c) -> c + 1))
.map(MyStreamOutput::new)
.sendTo(pageViewPerMember);
}
}

9. Samza SQL
INSERT INTO kafka.pageViewCount
SELECT memberId, count(*) FROM kafka.pageViewStream
GROUP BY memberId, TUMBLE(current_timestamp, INTERVAL '5' MINUTES)

10. Samza API stack
User can choose the API to
write a Samza job.

11. Why SQL on Samza
• Expand the target audience of stream processing.
• Obtain quick real time insights.
• Create stream processing applications quickly.

12. How does it work?

13. How do we execute below SQL on Samza?
INSERT INTO kafka.NewEmployees
SELECT firstName, lastName FROM kafka.profileUpdateStream
WHERE profile.newCompany = ‘LinkedIn’

14. High level architecture

15. Samza SQL to Calcite relational algebra
INSERT INTO kafka.NewLinkedInEmployees
SELECT firstName, lastName FROM kafka.profileChange
WHERE profile.newCompany = ‘LinkedIn’
LogicalTableModify
LogicalProject
LogicalFilter
LogicalTableScan

16. Samza operator graph conversion
LogicalTableModify
LogicalProject
LogicalFilter
LogicalTableScan
profileChange
.filter(p -> p.getNewCompany().equals("LinkedIn"))
.map(this::getFirstAndLastName)
.sendTo(newLinkedInEmployees);

17. Samza SQL message flow

18. Samza SQL message flow

19. Samza SQL rel message format
public class SamzaSqlRelMessage {
private final List<Object> relFieldValues = new ArrayList<>();
private final List<String> relFieldNames = new ArrayList<>();
public List<String> getRelFieldNames() {
return relFieldNames;
}
public List<Object> getRelFieldValues() {
return this.relFieldValues;
}
}
• Simple relational format that represents a row in a table
• Ordered list of named values

20. Pluggable input/output resolvers
INSERT INTO kafka.NewEmployees
SELECT firstName, lastName FROM kafka.profileUpdateStream
WHERE profile.newCompany = ‘LinkedIn’

21. Samza SQL architecture

22. Demo

23. Demo setup

24. How do you use it?
• Samza SQL is available in Samza 0.14 release.
• Tutorial – http://bit.ly/samzasql

25. Samza– 0.14
• Samza SQL
• Projection, Filtering, UDFs, Flatten, Union, Avro
• Apache Beam runner for Samza
• Azure EventHub support
• Amazon kinesis support
• Multi stage batch support
• High level API improvements
• Durable state
• Programmable SerDe

26. Samza SQL- Future
• Joins (Stream-Stream & Stream-Table)
• Aggregates & aggregate UDF
• Full Subquery support
• Samza SQL as a service

27. Samza SQL- Future
• Joins (Stream-Stream & Stream-Table)
• Aggregates & aggregate UDF
• Full Subquery support
• Samza SQL as a service

28. Questions?

30. Thank you

31. Samza operator graph conversion
LogicalTableModify
LogicalProject
LogicalFilter
LogicalTableScan

32. Pluggable schema and message converters