Flink Forward San Francisco 2018: Seth Wiesman - "Testing Stateful Streaming Applications"

April 10, 2018
Seth Wiesman
Testing Stateful Streaming Applications

©2018 MediaMath Inc. 2
val words: DataStream[String] = ???
words
.keyBy(word ⇒ word)
.map(new CountingMap)
Basic Stateful Word Count
“hello” → (“hello”, 1)
“world” → (“world”, 1)

• Stateless jobs are easy to reason about
• The same input always results in the same output
• Referentially Transparent
• That doesn’t mean no state at all!
• Stateless:: (IN, STATE) → (OUT, STATE)
vs Stateless Jobs

Operator State

import java.util.{List ⇒ JList}
class ResourceToStream(path: String)
extends RichSourceFunction[String] with ListCheckpointed[Integer] {
var numLines: Int = 0
override def cancel(): Unit = ???
override def run(ctx: SourceFunction.SourceContext[String]): Unit = ???
override def restoreState(state: JList[Integer]): Unit = ???
override def snapshotState(checkpointId: Long, timestamp: Long): JList[Integer] = ???
}

import java.util.{List ⇒ JList}
class ResourceToStream(path: String)
extends RichSourceFunction[String] with ListCheckpointed[Integer] {
var numLines: Int = 0
override def cancel(): Unit = ???
override def run(ctx: SourceFunction.SourceContext[String]): Unit = ???
override def restoreState(state: JList[Integer]): Unit = ???
override def snapshotState(checkpointId: Long, timestamp: Long): JList[Integer] = ???
}
Business Logic
State Management

it should "process a resource file" in {
val runner = new ResourceToStream("/testfile.txt")
val ctx = new MockSourceContext
runner.run(ctx)
ctx.output.size should be (4)
ctx.output should be (ListBuffer("hello", "world", "good", "day"))
}
Validate Business Logic

it should "restore from state" in {
val runner = new ResourceToStream("/testfile.txt")
val ctx = new MockSourceContext
runner.restoreState(Arrays.asList(2))
runner.run(ctx)
ctx.output.size should be (2)
ctx.output should be (ListBuffer("good", "day"))
}
Consider State Management

val mainStream: DataStream[String] = ???
val filterStream: DataStream[String] = env
.addSource(new ResourceToStream(“”))
mainStream
.connect(filterStream.broadcast)
.flatMap(new BroadcastFilter)
Broadcast State

class BroadcastFilter extends RichCoFlatMapFunction[String, String, String] with ??? {
val filters = mutable.Set.empty[String]
override def flatMap1(value: String, out: Collector[String]): Unit = {
if (!filters.contains(value)) {
out.collect(value)
}
}
override def flatMap2(value: String, out: Collector[String]): Unit = {
filters += value
}

override def initializeState(context: FunctionInitializationContext): Unit = {
state = context.getOperatorStateStore.getUnionListState(
new ListStateDescriptor[String]("filters", BasicTypeInfo.STRING_TYPE_INFO)
)
???
}
CheckpointedFunction

override def snapshotState(context: FunctionSnapshotContext): Unit = {
state.clear()
if (getRuntimeContext.getIndexOfThisSubtask == 0) {
for (filter ← filters) {
state.add(filter)
}
}
}

• Testing business logic is still straightforward
• Testing checkpoint methods is less clear
• Mocking out operator state is not fun
it should "filter out blacklisted words" in {
val filter = new BroadcastFilter
val out = new MockCollector
filter.flatMap2("hello", out)
out.output.size should be (0)
filter.flatMap1("hello", out)
filter.flatMap1("world", out)
}

• Used as part of Flink’s internal test suite
• ⚠️ not part of the public api ⚠️
• Not recommended as a first option
• Try and get by with integration tests first
• Be careful to test your code, not Flink’s!
• Used sparingly, can be very powerful
OperatorTestHarness

it should "restore from state" in {
val initialFilter = new BroadcastFilter
val initialOperator = new CoStreamFlatMap(initialFilter)
val initialTestHarness = new TwoInputStreamOperatorTestHarness(initialOperator)
initialTestHarness.initializeState(new OperatorStateHandles(0, null, null, null, null))
initialTestHarness.open()
initialTestHarness.processElement2(new StreamRecord[String]("hello"))
val snapshot = initialTestHarness.snapshot(0L, 0L)
initialTestHarness.close()

val restoreFilter = new BroadcastFilter
val restoreOperator = new CoStreamFlatMap(restoreFilter)
val restoreTestHarness = new TwoInputStreamOperatorTestHarness(restoreOperator)
restoreTestHarness.initializeState(snapshot)
restoreTestHarness.setup()
restoreTestHarness.open()
restoreTestHarness.processElement1(new StreamRecord[String]("hello"))
restoreTestHarness.getOutput.size() should be (0)
restoreTestHarness.close()
}

Keyed State

• Many keyed state operations can be handled “statelessly”
Keyed State
words
.keyBy(word ⇒ word)
.countWindow(3)
.reduce((word, _) ⇒ word)
• Not always an option

class EveryNthWord(n: Int) extends RichFlatMapFunction[String, String] {
@transient private lazy val countDesc = new ValueStateDescriptor("count", classOf[Int])
override def flatMap(value: String, out: Collector[String]): Unit = {
val countState = getRuntimeContext.getState(countDesc)
val count = Option(countState.value()).map(_.intValue()).getOrElse(1)
if (count == n) {
countState.clear()
out.collect(value)
} else {
countState.update(count + 1)
}
}
}

object StatelessEveryNthWord {
type StatefulFunction = (String, Option[Int]) ⇒ (TraversableOnce[String], Option[Int])
}
FlatMapWithState

class StatelessEveryNthWord(n: Int) extends StatelessEveryNthWord.StatefulFunction {
override def apply(word: String, state: Option[Int]): (TraversableOnce[String], Option[Int])= {
state match {
case Some(count) if count + 1 == n ⇒ (Some(word), None)
case Some(count) ⇒ (None, Some(count + 1))
case None ⇒ (None, Some(1))
}
}
}
FlatMapWithState

it should "initialize state the first time it sees a word" in {
val function = new StatelessEveryNthWord(3)
val (output, state) = function("hello", None)
output should be (None)
state should be (Some(1))
}
FlatMapWithState Test
it should "modify state in the middle of a run" in {
val function = new StatelessEveryNthWord(3)
val (output, state) = function("hello", Some(1))
output should be (None)
state should be (Some(2))
}

override def flatMap(value: String, out: Collector[String]): Unit = {
val state = Option(getRuntimeContext.getState(countDesc).value())
f(value, state) match {
case (output, None) ⇒
output.foreach(out.collect)
getRuntimeContext.getState(countDesc).clear()
case (output, Some(x)) ⇒
getRuntimeContext.getState(countDesc).update(x)
}
Possible FlatMapWithState Implemention
react to results
initialize
context

• Group all elements of the same key that occur within 30 seconds of each other
• similar to a session window
• Output the third instance of every word immediately
• When more than 30 seconds have passed since the last word:
• clear state
• If more than 3 instances of a word have been observed, output to heavy hitter side output
• If less than 3 instances of a word have been observed, output to infrequent side output
A more complex example

object Algebra {
final case class Context(count: Long, watermark: Long)
final case class Result(word: Option[String], timer: Long)
}
trait Algebra {
def evaluateElem(word: String, ctx: Algebra.Context): Algebra.Result
def evaluateTimer(word: String, count: Long): Option[Either[String, String]]
}
Specialized Context

class StatelessAlgebra extends Algebra {
final val private[this] thirty_seconds: Long = 30 * 1000L
override def evaluateElem(word: String, ctx: Algebra.Context): Algebra.Result = {
val output = if (ctx.count == 3) {
Some(word)
} else {
None
}
val timer = ctx.watermark + thirty_seconds
Algebra.Result(output, timer)
}
Stateless Business Logic

override def evaluateTimer(word: String, count: Long): Option[Either[String, String]] = {
if (count < 3) {
Some(Left(word))
} else if (count > 3) {
Some(Right(word))
} else {
None
}
}
}
Stateless Business Logic

getRuntimeContext.getState(wordDesc).update(value)
getRuntimeContext.getReducingState(countDesc).add(1L)
val count = getRuntimeContext.getReducingState(countDesc).get()
val context = Algebra.Context(count, ctx.timerService().currentWatermark())
val Algebra.Result(output, time) = program.evaluateElem(value, context)
getRuntimeContext.getState(timerDesc).update(time)
ctx.timerService().registerEventTimeTimer(time)
Program Runner
react to results
initialize
context

• Business Logic
• Business logic can be tested in isolation of Flink
• Simple to simulate various scenarios
• Runner
• Does not require knowledge of business domain to validate
• Isolates the more complex parts of writing streaming applications
Testing our Implementation

Integration Testing

• Test the full End-to-End pipeline
• More closely simulate production than unit tests
• StreamingMultipleProgramsTestBase
• Runs a local version of Flink
• Beware only testing the happy path
• Consider running each integration test twice – both with and without failure
• See BucketingSink test suite for an example of failing the stream [1]
Integration Testing
[1] https://github.com/apache/flink/blob/7b07fa5a09279a55c99d80db92ebf98a7dcd9644/flink-connectors/flink-connector-
filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSinkFaultToleranceITCase.java#L200

Full Code Examples
github.com/sjwiesman/ff-sf-2018
Thank You

Flink Forward San Francisco 2018: Seth Wiesman - "Testing Stateful Streaming Applications"

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Flink Forward San Francisco 2018: Seth Wiesman - "Testing Stateful Streaming Applications"

Similar to Flink Forward San Francisco 2018: Seth Wiesman - "Testing Stateful Streaming Applications" (20)

More from Flink Forward

More from Flink Forward (20)

Recently uploaded

Recently uploaded (20)

Flink Forward San Francisco 2018: Seth Wiesman - "Testing Stateful Streaming Applications"