Download as PDF, PPTX
![About Me
3
Work [ed | s] @
Committer &
PPMC on
Father of 2
Co-Chair for
Apache Airflow](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-3-320.jpg)
![4
[ | ] @
& PPMC Apache Airflow](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-4-320.jpg)
![136
Avro Schema Example
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-136-320.jpg)
![137
Avro
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-137-320.jpg)
![138
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Avro Schema Example](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-138-320.jpg)
![139
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Avro](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-139-320.jpg)
![140
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
Avro Schema Example](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-140-320.jpg)
![141
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
Avro](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-141-320.jpg)
![142
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
3 fields in the record: 1 required, 2
optional
Avro Schema Example](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-142-320.jpg)
![143
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
3
1 2
Avro](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-143-320.jpg)
![144
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Data
x 1,000,000,000
Avro Schema Data File Example
Schema
Data
0.0001 %
99.999 %
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-144-320.jpg)
![145
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Data
x 1,000,000,000
Avro
0.0001 %
99.999 %
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-145-320.jpg)
![146
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-146-320.jpg)
![147
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro
99 %
1 %
Data](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-147-320.jpg)
![148
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data
OVERHEAD!!](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-148-320.jpg)
![149
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro
99 %
1 %
Data
!!](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-149-320.jpg)
![150
Schema
Registry
(Lambda)
Innovation 1 : Avro Schema Registry
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
register_schema
Message
Producer (P)](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-150-320.jpg)
![151
(Lambda)
1 : Avro
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
register_schema
(P)](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-151-320.jpg)
![158
Schema
Registry
(Lambda)
Innovation 1 : Avro Schema Registry
Message
Producer (P)
Data
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-158-320.jpg)
![159
(Lambda)
1 : Avro
(P) (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-159-320.jpg)
![160
Schema
Registry
(Lambda)
Innovation 1 : Avro Schema Registry
Message
Producer (P)
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Message Consumers
• download & cache the schema
• then decode the data](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-160-320.jpg)
![161
(Lambda)
1 : Avro
(P) (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
• &
•](https://image.slidesharecdn.com/cndpv6ja-161104072537/85/Cloud-Native-Data-Pipelines-in-Eng-Japanese-QCon-Tokyo-161-320.jpg)
Uploaded bySid Anand
Cloud Native Data Pipelines (in Eng & Japanese) - QCon Tokyo
Sid Anand discusses cloud-native data pipelines, highlighting the need for big data companies to build large-scale pipelines that can run in the public cloud. The presentation covers key architectural components, including batch and near-real-time processing, and emphasizes the importance of operability, correctness, and cost efficiency in designing data pipelines. Additionally, he explores how tools like Apache Airflow facilitate workflow management in analytics use cases, particularly for message scoring in organizations.
More Related Content
Similar to Cloud Native Data Pipelines (in Eng & Japanese) - QCon Tokyo
Cloud Native Data Pipelines (in Eng & Japanese) - QCon Tokyo
- 1. Cloud Native Data Pipelines SidAnand QCon Shanghai & Tokyo 2016 1
- 2. Sid Anand QCon Shanghai& Tokyo 2016 2 Japanese Translation: Kiro Harada (@haradakiro)
- 3. About Me 3 Work [ed| s] @ Committer & PPMC on Father of 2 Co-Chair for Apache Airflow
- 4. 4 [ | ]@ & PPMC Apache Airflow
- 5.
- 6.
- 7.
- 8.
- 9. Agari : WhatWe Do 9
- 10.
- 11. 11 Agari : WhatWe Do
- 12.
- 13. 13 Agari : WhatWe Do
- 14.
- 15. 15 Agari : WhatWe Do
- 16.
- 17. 17 Agari : WhatWe Do
- 18.
- 19. 19 Enterprise Customers email metadata apply trust models email md + trust score Agari’sPrevious EP Version Agari : What We Do Batch
- 20.
- 21. 21 email metadata apply trust models email md + trustscore Agari’s Current EP VersionEnterprise Customers Agari : What We Do Near-real time Quarantine
- 22.
- 23. Data Pipelines BI vsPredictive 23
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31. BI Predictive Common Focusof this talk Data Pipelines 31 Web Servers OLTP DB Data Warehouse Repor6ng Tools Query Browsers ETL (batch) MySQL, Oracle, Cassandra Terradata, RedShi; BigQuery OLTP DB or cache ETL (batch or streaming) MySQL, Oracle, Cassandra, Redis Spark, Flink, Beam, Storm Web Servers Ranking (Search, News Feed), Recommender Products, Fraud DetecGon / PrevenGon Data Source
- 32.
- 33. Motivation Cloud Native DataPipelines 33
- 34. Cloud Native DataPipelines 34 Big Data Companies like LinkedIn, Facebook, Twitter, & Google build custom, large scale data pipelines that run in their own Data Centers
- 35.
- 36. Cloud Native DataPipelines 36 Big Data Companies like LinkedIn, Facebook, Twitter, & Google build custom, large scale data pipelines that run in their own Data Centers Most start-ups run in the public cloud. Can they leverage aspects of the public cloud to build comparable pipelines?
- 37.
- 38. Cloud Native DataPipelines 38 Cloud Native Techniques Open Source Technogies Custom Data Pipeline Stacks seen in Big Data companies ~
- 39.
- 40. Design Goals Desirable Qualitiesof a Resilient Data Pipeline 40
- 41.
- 42. 42 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 43.
- 44. 44 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability • Pay-as-you-go
- 45. 45 • • ( …) • •SLA • SLA • •
- 46. Quickly Recoverable 46 • Bugshappen! • Bugs in Predictive Data Pipelines have a large blast radius • Optimize for MTTR
- 47.
- 48. Predictive Analytics @Agari Use Cases 48
- 49. Predictive Analytics @Agari 49
- 50. Use Cases 50 Apply trustmodels (message scoring) batch + near real time Build trust models batch (Enterprise Protect)
- 51.
- 52. Use-Case : Message Scoring(batch) Batch Pipeline Architecture 52
- 53.
- 54. Use-Case : MessageScoring 54 enterprise A enterprise B enterprise C S3 S3 uploads an Avro file every 15 minutes
- 55. Use-Case : 55 enterprise A enterpriseB enterprise C S3 Avro 15 S3
- 56. Use-Case : MessageScoring 56 enterprise A enterprise B enterprise C S3 Airflow kicks of a Spark message scoring job every hour (EMR)
- 57. Use-Case : 57 enterprise A enterpriseB enterprise C S3 Airflow Spark (EMR)
- 58. Use-Case : MessageScoring 58 enterprise A enterprise B enterprise C S3 Spark job writes scored messages and stats to another S3 bucket S3
- 59. Use-Case : 59 enterprise A enterpriseB enterprise C S3 Spark S3 S3
- 60. Use-Case : MessageScoring 60 enterprise A enterprise B enterprise C S3 This triggers SNS/SQS messages events S3 SNS SQS
- 61. Use-Case : 61 enterprise A enterpriseB enterprise C S3 SNS/SQS S3 SNS SQS
- 62. Use-Case : MessageScoring 62 enterprise A enterprise B enterprise C S3 An Autoscale Group (ASG) of Importers spins up when it detects SQS messages S3 SNS SQS Importers ASG
- 63. Use-Case : 63 enterprise A enterpriseB enterprise C S3 SQS (ASG) S3 SNS SQS Importers ASG
- 64. 64 enterprise A enterprise B enterpriseC S3 The importers rapidly ingest scored messages and aggregate statistics into the DB S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 65. 65 enterprise A enterprise B enterpriseC S3 S3 SNS SQS Importers ASG DB Use-Case :
- 66. 66 enterprise A enterprise B enterpriseC S3 Users receive alerts of untrusted emails & can review them in the web app S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 67. 67 enterprise A enterprise B enterpriseC S3 WebApp S3 SNS SQS Importers ASG DB Use-Case :
- 68. 68 enterprise A enterprise B enterpriseC S3 S3 SNS SQS Importers ASG DB Airflow manages the entire process Use-Case : Message Scoring
- 69. 69 enterprise A enterprise B enterpriseC S3 S3 SNS SQS Importers ASG DB Airflow Use-Case :
- 70. Tackling Cost &Timeliness Leveraging the AWS Cloud 70
- 71.
- 72. Tackling Cost 72 Between DailyRuns During Daily Runs When running daily, for 23 hours of a day, we didn’t pay for instances in the ASG or EMR
- 73.
- 74. Tackling Cost 74 Between HourlyRuns During Hourly Runs When running daily, for 23 hours of a day, we didn’t pay for instances in the ASG or EMR This does not help when runs are hourly since AWS charges at an hourly rate for EC2 instances!
- 75. Tackling Cost 75 23 ASGEMR AWS
- 76. Tackling Timeliness Auto ScalingGroup (ASG) 76
- 77.
- 78. ASG - Overview 78 Whatis it? A means to automatically scale out/in clusters to handle variable load/traffic A means to keep a cluster/service of a fixed size always up
- 79.
- 80. ASG - DataPipeline 80 importer importer importer importer Importer ASG scaleout/in SQS DB
- 81.
- 82. 82 Sent CPU ACKd/Recvd CPU-based auto-scaling is goodat scaling in/out to keep the average CPU constant ASG : CPU-based
- 83.
- 84. ASG : CPU-based 84 Sent CPU Recv Premature Scale-in PrematureScale-in: • The CPU drops to noise-levels before all messages are consumed • This causes scale in to occur while the last few messages are still being committed
- 85.
- 86. 86 Scale-out: When VisibleMessages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-flight message is ACK’d) This causes the ASG to grow This causes the ASG to shrink ASG : Queue-based
- 87. 87 Scale-out: When VisibleMessages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-flight message is ACK’d) This causes the ASG to grow This causes the ASG to shrink ASG : Queue-
- 88. 88 ASG : Queue-based ShoyuKoto Da!!!!
- 89. 89 ASG : Queue- ShoyuKoto Da!!!!
- 90. 90 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • ASG • EMR Spark Daily • ASG • EMR Spark Hourly ASG • No Cost Savings
- 91. 91 • ASG • EMRSpark • ASG • EMR Spark ASG •
- 92.
- 93.
- 94. 94 A simple wayto author and manage workflows Provides visual insight into the state & performance of workflow runs Integrates with our alerting and monitoring tools Tackling Operability : Requirements
- 95.
- 96. Apache Airflow Workflow Automation& Scheduling 96
- 97.
- 98. 98 Airflow: Author DAGsin Python! No need to bundle many config files! Apache Airflow - Authoring DAGs
- 99. 99 Airflow: DAG Python! Apache Airflow - DAG
- 100. 100 Airflow: Visualizing aDAG Apache Airflow - Authoring DAGs
- 101.
- 102. 102 Airflow: It’s easyto manage multiple DAGs Apache Airflow - Managing DAGs
- 103.
- 104. Apache Airflow -Perf. Insights 104 Airflow: Gantt chart view reveals the slowest tasks for a run!
- 105.
- 106. 106 Apache Airflow -Perf. Insights Airflow: Task Duration chart view show task completion time trends!
- 107.
- 108. 108 Airflow: …And easyto integrate with Ops tools! Apache Airflow - Alerting
- 109. 109 Airflow: …And easyto integrate with Ops tools! Apache Airflow -
- 110. 110 Apache Airflow -Correctness
- 111.
- 112. 112 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 113.
- 114. Use-Case : Message Scoring(near-real time) NRT Pipeline Architecture 114
- 115.
- 116. Use-Case : MessageScoring 116 enterprise A enterprise B enterprise C Kinesis batch put every second K
- 117. : 117 enterprise A enterprise B enterpriseC Kinesis K
- 118. Use-Case : MessageScoring 118 enterprise A enterprise B enterprise C K As ASG of scorers is scaled up to one process per core per kinesis shard Scorers ASG
- 119. : 119 enterprise A enterprise B enterpriseC K ASG Kinesis CPU 1 Scorers ASG
- 120. Use-Case : MessageScoring 120 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Scorers apply the trust model and send scored messages downstream
- 121. : 121 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis
- 122. Use-Case : MessageScoring 122 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG As ASG of importers is scaled up to rapidly import messages DB
- 123. : 123 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG ASG DB
- 124. Use-Case : MessageScoring 124 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG
- 125. : 125 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG DB K Alerters ASG
- 126. Use-Case : MessageScoring 126 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG Quarantine Email
- 127. : 127 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG DB K Alerters ASG Email
- 128.
- 129.
- 130. Apache Avro What isAvro? 130
- 131.
- 132. 132 What is Avro? Avrois a self-describing serialization format that supports primitive data types : int, long, boolean, float, string, bytes, etc… complex data types : records, arrays, unions, maps, enums, etc… many language bindings : Java, Scala, Python, Ruby, etc…
- 133. 133 What is Avro? Avro :int, long, boolean, float, string, bytes, etc… : records, arrays, unions, maps, enums, etc… : Java, Scala, Python, Ruby, etc…
- 134. 134 What is Avro? Avrois a self-describing serialization format that supports primitive data types : int, long, boolean, float, string, bytes, etc… complex data types : records, arrays, unions, maps, enums, etc… many language bindings : Java, Scala, Python, Ruby, etc… The most common format for storing structured Big Data at rest in HDFS, S3, Google Cloud Storage, etc… Supports Schema Evolution!
- 135. 135 What is Avro? Avro :int, long, boolean, float, string, bytes, etc… : records, arrays, unions, maps, enums, etc… : Java, Scala, Python, Ruby, etc… HDFS, S3, Google Cloud Storage !
- 136. 136 Avro Schema Example {"namespace":"agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
- 137. 137 Avro {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
- 138. 138 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Avro Schema Example
- 139. 139 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Avro
- 140. 140 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User Avro Schema Example
- 141. 141 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User Avro
- 142. 142 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User 3 fields in the record: 1 required, 2 optional Avro Schema Example
- 143. 143 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User 3 1 2 Avro
- 144. 144 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Data x 1,000,000,000 Avro Schema Data File Example Schema Data 0.0001 % 99.999 % Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data
- 145. 145 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Data x 1,000,000,000 Avro 0.0001 % 99.999 % Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data
- 146. 146 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro Schema Streaming Example Schema Data 99 % 1 % Data
- 147. 147 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro 99 % 1 % Data
- 148. 148 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro Schema Streaming Example Schema Data 99 % 1 % Data OVERHEAD!!
- 149. 149 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro 99 % 1 % Data !!
- 150. 150 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } register_schema Message Producer (P)
- 151. 151 (Lambda) 1 : Avro {"namespace":"agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } register_schema (P)
- 152. 152 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry register_schema returns a UUID Message Producer (P)
- 153.
- 154. 154 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry Message Producer sends UUID + Message Producer (P) Data Message Consumer (C)
- 155. 155 (Lambda) 1 : Avro UUID+ (P) Data (C)
- 156. 156 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry Message Producer (P) Data Message Consumer (C) getSchemaById (UUID)
- 157. 157 (Lambda) 1 : Avro (P)(C) getSchemaById (UUID)
- 158. 158 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry Message Producer (P) Data Message Consumer (C) getSchemaById (UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
- 159. 159 (Lambda) 1 : Avro (P)(C) getSchemaById (UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
- 160. 160 Schema Registry (Lambda) Innovation 1 :Avro Schema Registry Message Producer (P) Message Consumer (C) getSchemaById (UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Message Consumers • download & cache the schema • then decode the data
- 161. 161 (Lambda) 1 : Avro (P)(C) getSchemaById (UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } • & •
- 162. 162 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG SR SR SR Innovation 1 : Avro Schema Registry
- 163. 163 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG alterer DB K Alerters ASG SR SR SR 1 : Avro
- 164. 164 The Architecture iscomposed of repeated patterns of : ASG-based compute consumer Kinesis transport streams (i.e. AWS’ managed “Kafka”) A Lambda-based Avro Schema Registry Innovation 2 : Repeatable Units Compute i Kinesis i ASG i SR
- 165. 165 You can chainthese repeatable units together to make arbitrary DAGs (Directed Acyclic Graphs) User Hashicorp’s Terraform to compose your DAG through automation The example above is a simple Linear DAG with 3 units Innovation 2 : Repeatable Units Compute i Kinesis i ASG i SR Compute i Kinesis i ASG i SR Compute i Kinesis i ASG i SR
- 166. 166 DAG( ) Hashicorp’s TerraformDAG DAG 2 : Compute i Kinesis i ASG i SR Compute i Kinesis i ASG i SR Compute i Kinesis i ASG i SR
- 167. Airflow Job ReactivelyScales Innovation 3 : Reactive-Scaling (WIP) 167 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG DB K Alerters ASG SR SR SR
- 168. Airflow 3 : 168 enterprise A enterpriseB enterprise C K Scorers ASG Kinesis Importers ASG DB K Alerters ASG SR SR SR
- 169. 169 If the ADRis triggered and a model build or code push was recently done to Compute 1, ADR will revert the last code or model push to ASG Compute 1 Innovation 4 : Anomaly-based Rollback (WIP) ASG Compute 1 Compute 2 Kinesis ASG SR Anomaly- detector & Reverter
- 170. 170 ADR Compute 1 ADR Compute1 4: (WIP) ASG Compute 1 Compute 2 Kinesis ASG SR Anomaly- detector & Reverter
- 171. Open Source Plans 171 Followus to be notified when the following is open- sourced • Avro Schema Registry • Agari (Kinesis+ASG) scaling tool (Airflow Job) • Anomaly-detector & Reverter To be notified, follow @AgariEng & @r39132
- 172. 172 Twitter • Avro SchemaRegistry • Agari (Kinesis+ASG) scaling tool (Airflow Job) • Anomaly-detector & Reverter @AgariEng & @r39132
- 173. Acknowledgments 173 • Vidur Apparao •Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Mike Jones • Scot Kennedy • Thede Loder • Paul Lorence • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang • Julian Mehnle None of this work would be possible without the contributions of the strong team below
- 174. 174 • Vidur Apparao •Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Mike Jones • Scot Kennedy • Thede Loder • Paul Lorence • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang • Julian Mehnle
- 175.
- 176.