I had the opportunity in 2020 to power Covid-19 contact tracing. Tools built in Elixir provided a data backbone that worked faster, was changed easier, and was more resilient than the systems we replaced. In this story we’ll meet a cast of characters including Phoenix, Broadway, Oban, as well as Lambda, NodeJS, Ruby on Rails. Some worked well. Some were confusing. Some were both. Some I would use again. https://codesync.global/conferences/code-beam-v-america-2021/

1. Using Elixir to fight Covid-19
Eric Saxby @sax
@ecdysone

2. Introduction

3. Jacob Peter Gowy's The Flight of Icarus (1635–1637) {{PD-1996}}

4. Gustav Moreau's Phaethon (1878) {{PD-1996}}

5. Gustav Doré's Paradise Lost (1866) {{PD-1996}}

6. Timeline of this talk
March
Let’s help with
Covid!
(I wish this was Elixir)
April/May
Let’s help
New York
contact tracing
(could we do this
in Elixir?)
June/July
Let’s rewrite this
with Broadway

7. Using Elixir to fight Covid-19
real talk

8. Luc Viatour's Total Solar Eclipse (1999) {{CC BY-SA 3.0}}

9. March 2020

10. {{PD-1996}}
Gustav Moreau's Diomede devoured by his horses (1865)
Personal
Protective
Equipment

11. {{PD-1996}}
Gustav Moreau's Diomede devoured by his horses (1865)
- Track hospital usage & needs
- Allow supply chain to route to
tomorrow’s needs
- Ruby on Rails
- Heroku
PPE resource status

12. @jasondeblooisphotography
- March 2020 : CHAOS
- Large companies also solving this
problem
- Project purposefully shut down
Retrospective

13. April 2020

14. Caravaggio's Cardsharps (1594) {{PD-1996}}
- Generic platform
- Busy with user workflows
- Unable to add integrations
Contact Tracing

15. Norman Rockwell's Radio Phone (1922) {{PD-US}}
Telephony
- Phoenix / LiveView
- AWS ECS

16. Paolo Veronese's Arachne (Dialects) {{PD-US}}
Data pipeline
- NodeJS
- Lambda

17. Caravaggio's Cardsharps (1594) {{PD-1996}}
Shared Values
- Protect PHI
- As simple as possible

18. Elixir vs NodeJS
John Steeple Davis's Scene of the Battle of Marathon (1900) {{PD-1996}}

19. Giovanni Domenico Tiepolo's The Procession of the Trojan Horse in Troy (1773) {{PD-1996}}
Phoenix vs Rails

20. @jasondeblooisphotography
- Elixir v Node v Ruby
- all allow rapid development
- Elixir turned out simpler
resilient solutions
Retrospective

21. Deeper look @ data pipeline
- State database - HL7 aggregator
- Each lab (possibly each location) creates its
own integration
- Required fields, with minimal/no validations
- No built in Master Patient Index

22. Master Patient Index
- Unique ID for each person in the public
health system
- Deduplication
- Cross-reference
- Each state** runs their own MPI
** or they might not

23. State
database
Somebody
gets the
file
They give
it to this
person
Who
asks someone
to “enrich”
the data
So they can
get an MPI
(hours later)
Then they
give it to
this person
Who undoes some
of the previous steps??
Then gives it to us?
So we can put
the data in
the Contact Tracing
system
every
30m
literally forever
5s
??
N hours
??

24. With “post-processing” that
undoes any deduplication
from the MPI
Only 7% successful
MPI might not
be unique
Changes delimiter,
without escaping
data

25. Consequences
Jacques-Louis David's Sorrow (1773) {{PD-1996}}
- Massive delays, for little benefit
- Multiple calls to the same people
- Overloaded workers
- Low trust

26. June 2020

27. Let’s look at some data

28. 555 555 1234
000 000 0000
1
{wait, is that a hospital?}
{yeah, that’s definitely a hospital}
555 1234
Phone numbers

29. 123 Main St
123 Mani St
{wait, is this a hospital?}
{yeah, that’s definitely a hospital}
Addresses

30. Names
Leonardo / da / Vinci
Leonardo da / / Vinci
Leonardo / / Vinci
Leo / / Vinci
Loenardo / / Vinci

31. Leonardo / da / Vinci / M
Leonardo da / / Vinci / M
Leonardo / / Vinci / M
Leo / / Vinci / M
Leonardo / da / Vinci / F
Gender

32. …with no unique constraints…
…because they come from lab systems…
…that sometimes recycle ids every month…
…with no guarantee that different labs
won’t use the same id…
“Unique” identifiers

33. Can we be an MPI?
Vasily Polenov's Left hand with the index finger (c1885) {{PD-1996}}

34. - Addresses - nope
- Name and birth date?
- Gender? - nope
- Phone numbers? - nope
- Accession #s - nope
Vasily Polenov's Left hand with the index finger (c1885) {{PD-1996}}

35. Eugène Delacroix's Hamlet and Horatio in the Cemetary (1839) {{PD-1996}}
Lets talk to people!
- Data is unreliable
- Duplication is terrible
- Full time positions finding
duplicates

36. Eugène Delacroix's Hamlet and Horatio in the Cemetary (1839) {{PD-1996}}
Insights
- Manual correction is fine*
- * if quantity is low enough
- Problem is duplicate phone calls

37. Can we be an MPI?
Thomas Cole's Prometheus Bound (1846-1847) {{PD-1996}}

38. Maybe??
Why not??
Thomas Cole's Prometheus Bound (1846-1847) {{PD-1996}}

39. Odilon Redon's Icarus (c1890) {{PD-1996}}
Insights
- There was no MPI
- “Good enough” would be much
better than what’s there

40. Odilon Redon's Icarus (c1890) {{PD-1996}}
Giorgio / G / Vasari / M / 1511 / ABC123
Giorgio G / / Vasari / M / 1511 / ABC123
Giorgio / G / Vasari / F / 1511 / ABC123
Gorgio / / Vasari / M / 1511 / ABC124
Giorgio / / Vasari / M / 1151 / ABC125
Giorgio / Geo / Vasari / F / 1511 / ABC126
Giorgio / G / Vasari / M / 1511 / CDE123

41. Odilon Redon's Icarus (c1890) {{PD-1996}}
Giorgio / G / Vasari / M / 1511 / ABC123
Giorgio G / / Vasari / M / 1511 / ABC123
Giorgio / G / Vasari / F / 1511 / ABC123
Gorgio / / Vasari / M / 1511 / ABC124
Giorgio / / Vasari / M / 1151 / ABC125
Giorgio / Geo / Vasari / F / 1511 / ABC126
Giorgio / G / Vasari / M / 1511 / CDE123

42. Can we beat 6 hours?
Ilya Repin's Leisure (1882) {{PD-1996}}

43. A brief history of streams
- Enum - Jan 6, 2012 - “Start working on Enum protocol”
- Stream - June 30, 2013 - “Add streams”
- GenStage - Aug 26, 2015 - initial commit
- Flow - Jan 17, 2017 - initial commit
- Broadway - Feb 25, 2019 - Announcing Broadway

44. Broadway

47. Melchior d'Hondecoeter's The Menagerie (1690) {{PD-1996}}
File
Row
Processor
Row
Batcher
New
Person
Processor
Lab Result
Processor
Lab Result
Batcher
ContactTracing
Adapter
Processor
Batcher

48. File
Row
Processor
Batcher
New
Person
Processor
Lab Result
Processor
Lab Result
Batcher
ContactTracing
Adapter
Processor
Batcher

49. - Producer - handle_demand/2
- Transformer
- Processor - handle_message/3
- Batcher - handle_batch/4
- Acknowledger

50. File Reader

51. Generate me
a checksum
:new
:update

52. Naive approach

55. Back pressure

59. File
Row
Processor
Batcher
New
TestResult
Records

60. DONE

61. Duplicate rows in a file
- Mutexes
- should have just used partition_by

62. does it blend?

63. - Old pipeline - now 3-8hrs
- Broadway - 15m
- +SP 200:200 - 10m
- ETS cache - 25s
500k+ database queries

64. does it blend?

65. Next pipeline!

66. Unprocessed
TestResults
Person/Case
Processor
New/Updated
Case
Records

67. Home-grown
Event Sourcing
Yaaaaay

68. Halting problem
Alexandre Antigna's The forced stop (1855) {{PD-1996}}

70. Next: Extract data!

71. - API calls per county
- minimum page size 100
- N counties

72. handle_demand/2
handle_demand — county 1, page 1
- extra records set into a buffer on state
handle_demand — read from buffer
handle_demand — next page, extras on buffer
- if no next page, increment county
handle_demand — county 2, page 1
…etc

73. Halting problem
Alexandre Antigna's The forced stop (1855) {{PD-1996}}

74. Buffering demand

77. Dispatcher
Dispatcher
Dispatcher
Dispatcher Dispatcher

78. Dispatcher
Dispatcher
Dispatcher
Dispatcher Dispatcher
• Consumers demand work from their
dispatcher
• Dispatchers demand work from
producers
• If the producer does not fill demand,
the dispatcher may wait for more
events to arrive
When there is more demand than data,
producers must track unmet demand

79. handle_demand/2
- handle_demand — county 1, page 1
- extra records set into a buffer on state
- handle_demand — read from buffer
- handle_demand — next page, extras on buffer
- if no next page, increment county
- county 2, page 1
- …etc
Last page
may not fulfill
demand

80. Next: Send data to
contact tracing

81. - HTTP(S) is unreliable
- Don’t trust system uptime
- Broadway does not come with retries
- though it may be implemented in
a producer

82. Hans Gude's Oban Bay (1889) {{PD-1996}}
Oban

83. - Background job runner
- Built-in retry
- Oban Web - paid license
- Quick to set up
- Uses PostgreSQL for queues
Hans Gude's Oban Bay (1889) {{PD-1996}}

84. Melchior d'Hondecoeter's The Menagerie (1690) {{PD-1996}}
File Producer Row
Processor
Row
Batcher
Unprocessed
TestResults Producer
People/Case
Processor
Cases in
CommCare
Push to Oban
Processor
New/Updated
Cases Producer
Test Results
(unique rows in files)
People + Cases
Oban
Cases Updated
in CT Producer
Case
Processor
People + Cases
Cases
updated in
CommCare

85. Broadway vs Oban

86. Testing
Willy Schlobach's Test drive over the Alps (1924) {{PD-1996}}

87. Testing
Willy Schlobach's Test drive over the Alps (1924) {{PD-1996}}

88. Switchover

89. Switchover

90. Timelines

91. Deployment
Giovanni Domenico Tiepolo's Building of the Trojan Horse (1760) {{PD-1996}}

92. Retrospective

93. - Broadway is awesome
- Its limits are fine
- In the future, maybe just GenStage
- Event sourcing
- Our domain schemas were fine
- Maybe eventstore/Kafka next time
- OpenTelemetry
- Endemic problems in health care
- Need more individuals able to predict
consequences

94. Thank you