Spark is becoming a data processing giant, but it leaves much as an exercise for the user. Developers need to write specialized logic to move between batch and streaming modes, manually deal with late or out-of-order data, and explicitly wire complex flows together. This talk looks at how we tackled these problems over a multi-petabyte dataset at Cerner. We start with how hand-written solutions to these problems evolved to prescriptive practices, opening up development of such systems to a wider audience. From there we look at how the emergence of Google’s Dataflow on Spark is helping us take the next step: the tradeoffs between correctness, latency, and cost are becoming a simple, easily changeable decision rather than a deep analysis for each new need. Finally, we look at challenges unique to doing processing in large organizations, such as making independent units of processing composable into large pipelines — and making them usable in both batch and stream modes.

1. Untangling Healthcare
with Spark and Dataflow
Ryan Brush
@ryanbrush

3. Actual depiction of healthcare data

4. One out of six dollars

6. Three Acts
(Mostly)

7. Act I
Making sense of the pieces

8. answer = askQuestion (allHealthData)

9. 8000 CPT Codes
72,000 ICD-10 Codes
63,000 SNOMED disease codes
Incomplete, conflicting data sets No common person identifier
Standard data models and codes interpreted inconsistently
Different meanings in different contexts
How do we make sense of this?
55 Million Patients 3 petabytes of data

10. Claims
Medical
Records
Pharma
Operational
Link Records
Semantic
Integration
User-entered
annotations
Condition
Registries
Quality
Measures
Analytics
Rules

11. Link Records
Claims
Medical
Records
Pharma
Operational
Semantic
Integration
User-entered
annotations
Condition
Registries
Quality
Measures
Analytics
Rules

12. Medical
Records
Document
Sections
Notes
Addenda
Order
. . .
Normalize
Structure
Clean Data

13. Link Records
Claims
Medical
Records
Pharma
Operational
Semantic
Integration
User-entered
annotations
Condition
Registries
Quality
Measures
Analytics
Rules

14. answer = askQuestion (allHealthData)

15. linkedData = link(clean (pharma),
clean (claims),
clean (records))
normalized = normalize(linkedData)
answer = askQuestion (normalized)

16. Rein in variance
http://fortune.com/2014/07/24/can-big-data-cure-cancer/

17. Rein in variance
oral vs. axillary temperature

18. Join all the things!

19.
JavaRDD<ExternalRecord> externalRecords = ...

20.
JavaRDD<ExternalRecord> externalRecords = ...
JavaPairRDD<ExternalRecord,
ExternalRecord> cartesian =
externalRecords.cartesian(externalRecords);

21.
JavaRDD<Similarity> matches = cartesian.map(t -> {
})
return Similarity.newBuilder()
.setLeftRecordId(left.getExternalId())
.setRightRecordId(right.getExternalId())
.setScore(score)
.build();
ExternalRecord left = t._1();
ExternalRecord right = t._2();
double score = recordSimilarity(left, right);
.filter(s -> s.getScore() > THRESHOLD);

22. Person 1 Person 2 Person 3
Person 1 1 0.98 0.12
Person 2 1 0.55
Person 3 1

23. Reassembly Humpty Dumpty in Code

24.
JavaPairRDD<String,String> idToLink = . . .
JavaPairRDD<String,ExternalRecord> idToRecord = . . .

25.
JavaPairRDD<String,String> idToLink = . . .
JavaPairRDD<String,ExternalRecord> idToRecord = . . .
JavaRDD<Person> people = idToRecord.join(idToLink)
.mapToPair(
// Tuple of universal ID and external record.
item -> new Tuple2<>(item._2._2, item._2._1))
.groupByKey()
.map(SparkExampleTest::mergeExternalRecords);

26. SNOMED:388431003
HCPCS:J1815
ICD10:E13.9
CPT:3046F
SNOMED:43396009, value: 9.4

27. SNOMED:388431003
HCPCS:J1815
InsulinMed InsulinMed
ICD10:E13.9
DiabetesCondition
Diabetic
CPT:3046F SNOMED:43396009, value: 9.4
Retaking Rules for Developers, Strange Loop 2014

29. 29
select * from outcomes where…

30. 30

31. Start with the questions you want
to ask and transform the data to fit.

32. But what questions are
we asking?

34. “The problem is we don’t
understand the problem.”
-Paul MacReady

35. cleanData = clean (allHealthData)
projected = projectForPurpose (cleanData)
answer = askQuestion (projected)

36. Sepsis

38. Early Lessons

39. Make no assumptions
about your data

40. Your errors are a signal

41. Data sources have a signature

42. But the latency!
And the complexity!

43. Act II
Putting health care together…fast

44.
JavaPairRDD<String,String> idToLink = . . .
JavaPairRDD<String,ExternalRecord> idToRecord = . . .
JavaRDD<Person> people = idToRecord.join(idToLink)
.mapToPair(
// Tuple of UUID and external record.
item -> new Tuple2<>(item._2._2, item._2._1))
.groupByKey()
.map(SparkExampleTest::mergeExternalRecords);

45.
JavaPairDStream<String,String> idToLink = . . .
JavaPairDStream<String,ExternalRecord> idToRecord
= . . .
idToLink.join(idToRecord);

46.
JavaPairDStream<String,String> idToLink = . . .
JavaPairDStream<String,ExternalRecord> idToRecord
= . . .
StateSpec personAndLinkStateSpec =
StateSpec.function(new BuildPersonState());
JavaDStream<Tuple2<List<ExternalRecord>,List<String>>>
recordsWithLinks =
idToRecord.cogroup(idToLink)
.mapWithState(personAndLinkStateSpec);
// And a lot more...

47. Link Updates
Record Updates
Grouped Record and Link Updates
Previous
State
Person
Records

48. Link Updates
Record Updates
Grouped Record and Link Updates
Previous
State
Person
Records
What about deletes?

49. Stream processing is not
“fast” batch processing.
Little reuse beyond core functions
Different pipeline semantics
Must implement compensation logic

50. Batch
Processing
Stream
Processing
Reusable Code

51. If you're willing to restrict the flexibility
of your approach, you can almost
always do something better.
-John Carmack

52. public void rollingMap(EntityKey key,
Long version,
T value,
Emitter emitter);
public void rollingReduce(EntityKey key,
S state,
Emitter emitter);

53. emitter.emit(key,value);
emitter.tombstone(outdatedKey);

54. Domain-Specific API
Batch Host Streaming Host
Reusable Code

55. So are we done?
Limited expressiveness
Not composable
Learning curve
Artificial complexity

56. Act III
Reframing the problem

57. Batch
Processing
Stream
Processing
Reusable Code

58. Batch
Processing
Stream
Processing
Kappa
Architecture

59. It’s time for a POSIX of data processing

60. Make everything a stream!
(If the technology can scale to your volume
and historical data.)
(If your problem can be expressed in monoids.)

61. Apache Beam
(was: Google Cloud Dataflow)

62. Potentially a POSIX of data processing
Composable Units (PTransforms)
Unification of Batch and Stream
Spark, Flink, Google Cloud Dataflow runners

63. Bounded: a fixed dataset
Unbounded: continuously updating dataset
Window: time range of your data to process
Trigger: when to process a time range

64. 10:00 11:00 12:008:00 9:00
Windows and Triggers

65. 10:00 11:00 12:008:00 9:00
Windows and Triggers

66. 10:00 11:00 12:008:00 9:00
Windows and Triggers

67. 10:00 11:00 12:008:00 9:00
Windows and Triggers

68. 8:03 8:04 8:058:01 8:02
Windows and Triggers

69. public class LinkRecordsTransform extends
PTransform<PCollectionTuple,PCollection<Person>> {
}
public static final TupleTag<RecordLink> LINKS =
new TupleTag<>();
public static final TupleTag<ExternalRecord> RECORDS =
new TupleTag<>();
@Override
public PCollection<Person> apply(PCollectionTuple input) {
. . .
}

70. PCollection<KV<String,CoGbkResult>> cogrouped = KeyedPCollectionTuple
.of(LINKS, idToLinks)
.and(RECORDS, idToRecords)
// Combines by key AND window
return uuidToRecs.apply(
Combine.<String,ExternalRecord,Person>perKey(
new PersonCombineFn()))
.setCoder(KEY_PERSON_CODER)
.apply(Values.<Person>create());
PCollection<KV<String,ExternalRecord>> uuidToRecs = cogrouped.apply(
ParDo.of(new LinkExternalRecords()))
.setCoder(KEY_REC_CODER);
apply implementation:
.apply(CoGroupByKey.create());

71. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = . . .
PCollection<ExternalRecord> windowedRecs = . . .

72. PCollection<RecordLink> windowedLinks = . . .
PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<ExternalRecord> windowsRecs = . . .

73. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = links.apply(
Window.<RecordLink>into(
FixedWindows.of(Duration.standardMinutes(60)));
PCollection<ExternalRecord> windowsRecs = . . .

74. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = links.apply(
Window.<RecordLink>into(
FixedWindows.of(Duration.standardMinutes(60)))
.withAllowedLateness(Duration.standardMinutes(15))
.accumulatingFiredPanes());
PCollection<ExternalRecord> windowsRecs = . . .

75. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = links.apply(
Window.<RecordLink>into(
FixedWindows.of(Duration.standardMinutes(60)))
.withAllowedLateness(Duration.standardMinutes(15))
.accumulatingFiredPanes());
.triggering(Repeatedly.forever(
AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(Duration.standardMinutes(10)))));
PCollection<ExternalRecord> windowsRecs = . . .

76. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = links.apply(
Window.<RecordLink>into(
SlidingWindows.of(Duration.standardMinutes(120)))
.withAllowedLateness(Duration.standardMinutes(15))
.accumulatingFiredPanes());
.triggering(Repeatedly.forever(
AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(Duration.standardMinutes(10)))));
PCollection<ExternalRecord> windowsRecs = . . .

77. PCollection<Person> people = PCollectionTuple
.of(LinkRecordsTransform.LINKS, windowedLinks)
.and(LinkRecordsTransform.RECORDS, windowedRecs)
.apply(new LinkRecordsTransform());
PCollection<RecordLink> windowedLinks = links.apply(
PCollection<ExternalRecord> windowsRecs = . . .
Window.<RecordLink>into(new GlobalWindows())
.triggering(Repeatedly.forever(
AfterProcessingTime.pastFirstElementInPane().
plusDelayOf(Duration.standardMinutes(5))))
.accumulatingFiredPanes());

78. when was that data created?
what data have I received?
when should I process that data?
how should I group data to process?
what to do with late-arriving data?
should I emit preliminary results?
how should I amend those results?
Untangling Concerns

79. Simple Made Easy
Rich Hickey, Strange Loop 2011
Modular
Composable
Easier to reason about

80. But some caveats:
Runners at varying level of maturity
Retraction not yet implemented (see BEAM-91)
APIs may change

81. MLLib
REPLDataframes
Spark offers a rich ecosystem
Spark SQL
Genome Analysis Toolkit

82. Large, complex,
processing pipelines
Exploration and
transformation of data
Two classes of problems:

83. Actual depiction of healthcare data

84. Time
Understanding
Orientation
Pattern
Discovery
Prescriptive
Frameworks
Scalable
Processing
Web Development

85. Time
Understanding
Orientation
Pattern
Discovery
Prescriptive
Frameworks
Scalable
Processing Web Development

86. Focus on the essence
rather than the accidents.

87. Questions?
@ryanbrush