This is the story of the ten-year journey of a high-load data ingestion pipeline that handles 500 million posts daily, from choreography-based design to fully centralized orchestration, and the valuable lessons learned along the way.

1. LetyShops

2. from orchestration to
choreography and back
10-year journey
@yevhen

3. OS
S
@yevhen

4. YouScan.io
• founded in Kyiv in 2009
• 500K+ online sources
• 500M posts daily

5. added semantics
ML
NL
P
CV
ML
Rules
OC
R

6. pipeline

7. agenda
• functionality
• structure
• patterns
• tools
• problems

8. high-performance pipeline
sync T = (S1 + S2 + … + Sn) * K
async T = (S1 + S2 + … + Sn) + (Slow * K)

9. distributed pipeline
choreography

10. features
high throughput
low latency
backpressure
control parallelism
batching
routing

12. Apache NiFi

13. ingestion pipeline (2013)
o validate
o normalize
o detect sentiment
o detect language
…

14. TPL DataFlow
var fetch = new TransformBlock(async url =>
{
using (var web = new WebClient())
return (url, await
web.DownloadAsync(url));
});
var save = new ActionBlock(async data => {
(string url, byte[] image) = data;
await File.WriteAllBytesAsync(filePath,
image);
});
fetch.LinkTo(save); // links the output from the TransformBlock to
the ActionBlock

15. TPL DataFlow
• backpressure control
• low-latency, high throughput
• limit parallelism
• batching (DIY)
• composition & routing (basic)

16. evolution conundrum
extend
o validate
o normalize
o detect sentiment
o detect language
o match
o delay
o send to client
Microservices, anyone?
vs split
o validate
o normalize
o detect sentiment
o detect language
o match
o delay
o send to client

17. disintegration reasons
+ + +
+
service
functionality
code
volatility
scalability &
throughput
fault
tolerance
data
security
Architecture, The Hard Parts by Neal Ford
(https:www.youtube.com/@thoughtworks)

18. messaging patterns
Pipes and
Filters
Content
Enricher
Point-to-Point
Channel
Document
Message

19. [isFresh]
o match
o delay
o send to client
evolution

20. evolution
o extract
o highlight
o send to client

21. evolution
o detect objects
o detect OCR
o send to client
[hasImage]

22. o match keywords
o send to client
[noOCR]
evolution

23. micromess ©
[isFresh]
[lowPriority]
[noOCR]
[hasImage]
[hasImage]
o match
o delay
o send to client
o validate
o normalize
o detect sentiment
o detect language …
o extract
o highlight
o send to client
o match keywords
o send to client
o detect objects
o detect OCR
o send to client

24. diagram maintenance
problems
tracing is a pain
dependencies and
code duplication
full redeployment on
document schema changes

25. NEWCOMER
S

26. diagram maintenance
problems
tracing is a pain
dependencies and
code duplication
full redeployment on
document schema changes

27. Vanya
Ask Vanya!
WTH!

28. diagram maintenance
problems
tracing is a pain
dependencies and
code duplication
full redeployment on
document schema changes

29. diagram maintenance
problems
tracing is a pain
dependencies and
code duplication
full redeployment on
document schema changes

30. serialization rollercoaster
-> {json} ->
-> {json} ->
-> {json} ->
-> {json} ->
-> {json} ->
-> {json} ->

31. How dare you,
waste it all
this energy!

32. delusive autonomy
Clients do not tolerate partial data!

33. last nail

34. contextmess ©

35. -Illities

36. reintegration reasons
Architecture, The Hard Parts by Neal Ford
(https:www.youtube.com/@thoughtworks)
database
transactions
structural (data)
dependencies
workflow and
choreography

37. orchestration

38. distributed dataflow
[image == null]
indexer percolator image recognition

39. separation of concerns
flow
function
[image == null]
indexer percolator image recognition

40. orchestration patterns
Message
Router
Content-Based
Router

41. simple made easy
Orchestrator Service
Function Services

42. competing consumers

43. scalability
10 VMs

44. new problems

45. complexity distribution
o validate
o normalize
o detect sentiment
o detect language
…
o match
o delay
o send to client
o detect objects
o detect OCR
o send to client o match keywords
o send to client
o extract
o highlight
o send to client

46. big ball of …
o validate
o normalize
o detect sentiment
o detect language
…
o match
o delay
o send to client
o detect objects
o detect OCR
o send to client
o match keywords
o send to client
o extract
o highlight
o send to client
[isFresh
]
[hasImage
]
[noOCR
]
[highPriority]
[isFromChanne
l]
[hasText]

47. flow graph

48. ascii routes

49. straightforward pipeline?

50. Selective Consumer

51. Selective Consumer
[image == null]
[nsfw!]
[!nsfw]
[image]
[!filter -> next]

52. linear pipeline
[image != null] [nsfw!]

53. code
pipeline = new Pipeline<IngestionPipelineContext>(HandleError)
{
validate.LinkTo(CompleteWhen(x => x.Status ==
IndexingStatus.Invalid)),
enrich,
geo,
irCache.Filter(x => x.HasImage),
match,
delay,
nsfw.Filter(x => x.HasImage),
ocr,
… // 20+ stage declarations
complete
};

54. results
easy to:
• understand
• extend
• maintain
• operate
infrastructure
cost reduction
no documentation
overhead
-2000 LOC

55. conclusions
orchestration over
choreography within
single BC
maintainability
matters
you don’t need fancy
pipeline framework

56. When forming analogies, this bias
may lead individuals to selectively
focus on similarities between
two things while ignoring
differences, even if the analogy
is not valid.
Confirmation Bias
the end

57. we’re
hiring!