TouK Nussknacker is a GUI for Apache Flink that was created to address client needs for a more user-friendly interface. It allows defining data pipelines and business rules through expressions that are accessible for semi-technical users. The architecture uses a model-driven approach where the data, sources, sinks, processors and rules are defined through JSON configuration. It has been used successfully in production for over a year at one of the largest Polish telcos for real-time marketing and fraud detection applications.

1. TouK Nussknacker
GUI for Flink

2. Maciek Próchniak
(that’s me)

3. How it all began?
POC with Apache Flink
Great results :)

4. Configuration
Client not (really) happy

5. Scala expression for filters
Client still not very happy

6. Ok, we’ll build you a GUI
Finally, approving nod...

8. Main assumptions
● Model & integration in “normal” code
● Expressions accessible for semi-technicals - like SQL
● Facilitate testing and experimentation

9. Basics

10. Architecture

11. Model + process
JSON
Config
JAR
JSON JSON

14. Model

15. Model
● Data - POJOs, case classes
● Source/Sink - Flink API + ε
● Processor/Enricher - Services

16. Config creator
trait ProcessConfigCreator extends Serializable {
def services(config: Config) : Map[String, WithCategories[Service]]
def sourceFactories(config: Config): Map[String, WithCategories[SourceFactory[_]]]
def sinkFactories(config: Config): Map[String, WithCategories[SinkFactory]]
...
}

17. Service/Enricher
@MethodToInvoke
def invoke(@ParamName("clientId") clientId: String)
(implicit ec: ExecutionContext): Future[Client] = {
...
}

18. Business rules - expressions
● Filters
● Aggregation definitions
● Actions - mail subject, sms content
How to define them...?

19. Expressions
Spring Expression Language!
● Accessible enough
● More or less fast enough
● Ability to do basic validation/code-completion

22. Limitations of SPEL
● Speed
● Synchronous
● Type safety??

23. Advanced?

24. What about...
● Windows?
● State?
● Joins?
● Other Flink goodies?

25. Knowing where to stop
Completness
Difficulty

26. Knowing where to stop

27. Custom stream transformer - POFO
@MethodToInvoke(returnType = classOf[EventCount])
def execute(@ParamName("key") key: LazyInterpreter[String],
@ParamName("length") length: String) =
FlinkCustomStreamTransformation((start: DataStream[InterpretationResult]) => {
start.keyBy(key.syncInterpretationFunction)
.transform("eventsCounter", new CounterFunction(lengthInMillis))
})

28. Tests/Ops

29. “Convincing” to test
We’ll do it “our” way:
first we run on production,
then deploy to test

30. Tests - running

31. Tests - running

33. Generate test data

34. Taking control
● Do we process data?
● What is latency?
● Where do we filter out events?
● Are there errors?

35. Taking control

36. DEMO

37. Where can I use it?
● Telco
● Banking
● Media
● RTM
● Fraud detection

38. Adhoc jobs vs process
Ad-hoc
Query
SQL
RBEA
CI
CD
Process
Monitoring
SAS
Scala
Testing

39. Does it work in production?
● 1 year in production
● One of largest Polish telco’s
● RTM + Fraud detection

40. Does it work in production?

42. Road ahead?
● Asynchronous services
● Pluggable security
● Define model via GUI/Schema registry

43. Road ahead?
● Integration with CEP and SQL
● DAGs instead of Trees in UI?
● Governance?
● Nussknacker as a Service?

44. Try it out!
https://github.com/touk/nussknacker

45. Thanks
@mpproch
@touk_pl