Download as PDF, PPTX
![56
Scio
Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]
Verb: I can, know, understand, have knowledge.
Core API similar to spark-core, some ideas from scalding
github.com/spotify/scio](https://image.slidesharecdn.com/fromstreamtorecommendationusingapachebeamwithcloudpubsubandclouddataflow-160406203151/85/From-stream-to-recommendation-using-apache-beam-with-cloud-pubsub-and-cloud-dataflow-56-320.jpg)
![57
WordCount
Almost identical to Spark version
val sc = ScioContext()
sc.textFile("shakespeare.txt")
.flatMap(_.split("[^a-zA-Z']+").filter(_.nonEmpty))
.countByValue()
.saveAsTextFile("wordcount.txt")](https://image.slidesharecdn.com/fromstreamtorecommendationusingapachebeamwithcloudpubsubandclouddataflow-160406203151/85/From-stream-to-recommendation-using-apache-beam-with-cloud-pubsub-and-cloud-dataflow-57-320.jpg)
![58
PageRank in 13 lines
def pageRank(in: SCollection[(String, String)]) = {
val links = in.groupByKey()
var ranks = links.mapValues(_ => 1.0)
for (i <- 1 to 10) {
val contribs = links.join(ranks).values
.flatMap { case (urls, rank) =>
val size = urls.size
urls.map((_, rank / size))
}
ranks = contribs.sumByKey.mapValues((1 - 0.85) + 0.85 * _)
}
ranks
}](https://image.slidesharecdn.com/fromstreamtorecommendationusingapachebeamwithcloudpubsubandclouddataflow-160406203151/85/From-stream-to-recommendation-using-apache-beam-with-cloud-pubsub-and-cloud-dataflow-58-320.jpg)
![61
JSON vs Type Safe BigQuery
JSON approach, a.k.a. everything is Object
sc.bigQuerySelect("...").map { r =>
(r.get("track").asInstanceOf[TableRow]
.get("name").asInstanceOf[String],
r.get("audio").asInstanceOf[TableRow]
.get("tempo").toString.toInt
)
}
Compile
Run job
Wait
NullPointerException or ClassCastException
Repeat
Type safe approach
@BigQueryType.fromQuery("...")
class TrackTempo
sc.typedBigQuery[TrackTempo]().map { t =>
(t.track.name, t.audio.tempo.getOrElse(-1))
}
Compile
Run
Profit](https://image.slidesharecdn.com/fromstreamtorecommendationusingapachebeamwithcloudpubsubandclouddataflow-160406203151/85/From-stream-to-recommendation-using-apache-beam-with-cloud-pubsub-and-cloud-dataflow-61-320.jpg)
Uploaded byNeville Li
From stream to recommendation using apache beam with cloud pubsub and cloud dataflow
The document discusses Spotify's event delivery system utilizing Cloud Pub/Sub and Cloud Dataflow, emphasizing their architecture for managing real-time data and analytics. It outlines the advantages of using Kafka for reliable event delivery and the transition to Cloud Pub/Sub for simplified operations and scalability. Additionally, it touches on the use of Dataflow for streaming ETL jobs, the challenges of interactive processing, and the integration with BigQuery for data analysis.
More Related Content
Similar to From stream to recommendation using apache beam with cloud pubsub and cloud dataflow
![Lect 1 Number systems and base conversions. [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/lect1numbersystemsandbaseconversions-260111134109-67c2d865-thumbnail.jpg?width=640&height=640&fit=bounds)
From stream to recommendation using apache beam with cloud pubsub and cloud dataflow
- 1. Speakers: Igor Maravić& Neville Li, Spotify From stream to recommendation with Cloud Pub/Sub and Cloud Dataflow DATA & ANALYTICS
- 2.
- 3. 3 Client Client Client Client Current event deliverysystem Gateway Syslog Syslog Producer Any Data Centre Groupers Realtime Brokers ETL job Checkpoint Monitor Hadoop Hadoop Data Center Service Discovery ACK Brokers Syslog Consumer Liveness Monitor Brokers
- 4. 4 Client Client Client Client Complex Gateway Syslog Syslog Producer Any Data Centre GroupersRealtime Brokers ETL job Checkpoint Monitor Hadoop Hadoop Data Center Service Discovery ACK Brokers Syslog Consumer Liveness Monitor Brokers
- 5. 5 Client Client Client Client Stateless Gateway Syslog Syslog Producer Any Data Centre GroupersRealtime Brokers ETL job Checkpoint Monitor Hadoop Hadoop Data Center Service Discovery ACK Brokers Syslog Consumer Liveness Monitor Brokers
- 6. 6 Delivered data growth 20072008 2009 2010 2011 2012 2013 2014 2015
- 7.
- 8. 8 Redesigning event delivery Gateway Syslog FileTailer Any data centre Client Hadoop Client Client Client Event Delivery Service Reliable Persistent Queue ETL
- 9. 9 Same API Gateway Syslog File Tailer Anydata centre Hadoop Event Delivery Service Reliable Persistent Queue ETL Client Client Client Client
- 10. 10 Persistence Gateway Syslog File Tailer Any datacentre Hadoop Event Delivery Service Reliable Persistent Queue ETL Client Client Client Client
- 11. 11 Keep it simple Gateway Syslog FileTailer Any data centre Hadoop Event Delivery Service Reliable Persistent Queue ETL Client Client Client Client
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. 18 Event delivery withKafka 0.8 Gateway Syslog File Tailer Any data centre Client Hadoop Client Client Client Event Delivery Service Hadoop data centre Camus (ETL) Brokers Mirror Makers Brokers
- 19. 19 Gateway Syslog File Tailer Any datacentre Client Hadoop Client Client Client Event Delivery Service Hadoop data centre Camus (ETL) Brokers Mirror Makers Brokers Event delivery with Kafka 0.8
- 20.
- 21.
- 22. 22 At least oncedelivery
- 23.
- 24.
- 25.
- 26. 2626 SHUT UP AND TAKE MYMONEY!
- 27.
- 28. Building up trustin Cloud Pub/Sub 28
- 29. 29 Delivered data growth 20072008 2009 2010 2011 2012 2013 2014 2015
- 30.
- 31.
- 32.
- 33. 33 Event delivery withCloud Pub/Sub Gateway Any data centre Client Hadoop Client Client Client Cloud Pub/Sub Event Delivery Service File Tailer Syslog Cloud Storage Dataflow ETL using Cloud Dataflow
- 34. 3434 Streaming ETL jobwith Cloud Dataflow
- 35. 35 Dataflow SDK isa framework
- 36. 36 Cloud Dataflow isa managed service
- 37.
- 38. 38 Single Cloud Pub/Subsubscription
- 39. 39 GCS and HDFSin parallel.
- 40. 40 2016-03-22 03H 2016-03-22 04H Event time basedhourly buckets 2016-03-21 23H 2016-03-22 00H 2016-03-22 01H 2016-03-22 02H
- 41.
- 42.
- 43.
- 44. 44 Event time basedhourly buckets Incremental bucket fill Bucket completeness Late data handling
- 45.
- 46. 46 Windowing @Override public PCollection<KV<String, Iterable<EventMessage>>>apply( final PCollection<KV<String, EventMessage>> shardedEvents) { return shardedEvents .apply("Assign Hourly Windows", Window.<~>into( FixedWindows.of(ONE_HOUR)) .withAllowedLateness(ONE_DAY) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings(AfterPane.elementCountAtLeast(maxEventsInFile)) .withLateFirings(AfterFirst.of( AfterPane.elementCountAtLeast(maxEventsInFile), AfterProcessingTime.pastFirstElementInPane() .plusDelayOf(TEN_SECONDS)))) .discardingFiredPanes()) .apply("Aggregate Events", GroupByKey.create()); }
- 47.
- 48. Where are weright now?
- 49.
- 50. 5050 Scio Scala API forGoogle Cloud Dataflow
- 51. 51 Origin story Scalding andSpark popular for ML, recommendations, analytics @ Spotify 50+ users, 400+ unique jobs Early 2015 - Dataflow Scala hack project
- 52. 52 Why not Scaldingon GCE Pros ● Big community - Twitter, eBay, Etsy, Stripe, LinkedIn, SoundCloud ● Stable and proven Cons ● Hadoop cluster operations ● Multi-tenancy, resource contention and utilization ● No streaming mode
- 53. 53 Why not Sparkon GCE Pros ● Batch, streaming, interactive and SQL ● MLlib, GraphX ● Scala, Python, and R support Cons ● Hard to tune and scale ● Cluster lifecycle management
- 54. 54 Why Dataflow withScala Dataflow ● Hosted solution, no operations ● Ecosystem: GCS, Bigquery, Pubsub, Datastore, Bigtable ● Simple unified model for batch and streaming Scala ● High level DSL, easy transition for developers ● Reusable and composable code via functional programming ● Numerical libraries: Breeze, Algebird
- 55. 55 Cloud Storage Pub/Sub Datastore BigtableBigQuery BatchStreaming Interactive REPL Scio Scala API Dataflow Java SDK Scala Libraries Extra features
- 56. 56 Scio Ecclesiastical Latin IPA:/ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge. Core API similar to spark-core, some ideas from scalding github.com/spotify/scio
- 57. 57 WordCount Almost identical toSpark version val sc = ScioContext() sc.textFile("shakespeare.txt") .flatMap(_.split("[^a-zA-Z']+").filter(_.nonEmpty)) .countByValue() .saveAsTextFile("wordcount.txt")
- 58. 58 PageRank in 13lines def pageRank(in: SCollection[(String, String)]) = { val links = in.groupByKey() var ranks = links.mapValues(_ => 1.0) for (i <- 1 to 10) { val contribs = links.join(ranks).values .flatMap { case (urls, rank) => val size = urls.size urls.map((_, rank / size)) } ranks = contribs.sumByKey.mapValues((1 - 0.85) + 0.85 * _) } ranks }
- 59. 59 SQL and BigData Pipelines SQL is easier to write than data pipelines, but Hive with TSV or Avro ● Row based storage, inefficient full scan ● No integration with other frameworks Parquet ● Inspired by Google Dremel which powers BigQuery ● Immature Hive integration, hard to scale with Spark SQL ● Poor impedance matching with Scalding, Avro, etc.
- 60. 60 BigQuery and Scio BigQuery ●Slicing and dicing, aggregation, etc. ● Scaling independently ● Web UI, Tableau, QlikView etc. Scio ● Custom logic hard to express in SQL ● Seamless integration with BigQuery IO ● Scala macros for type safety
- 61. 61 JSON vs TypeSafe BigQuery JSON approach, a.k.a. everything is Object sc.bigQuerySelect("...").map { r => (r.get("track").asInstanceOf[TableRow] .get("name").asInstanceOf[String], r.get("audio").asInstanceOf[TableRow] .get("tempo").toString.toInt ) } Compile Run job Wait NullPointerException or ClassCastException Repeat Type safe approach @BigQueryType.fromQuery("...") class TrackTempo sc.typedBigQuery[TrackTempo]().map { t => (t.track.name, t.audio.tempo.getOrElse(-1)) } Compile Run Profit
- 62. 62 Spotify Running 60 milliontracks 30 million users * 10 tempo buckets * 25 personalized tracks Audio: tempo, energy, time signature ... Metadata: genres, categories Latent vectors from collaborative filtering
- 63. 63 Rapid prototyping withBigquery
- 64. 64 Spotify Running SELECT user_id,vector FROM UserEntity WHERE ... SELECT track_id, audio.tempo ... FROM TrackEntity WHERE ... most popular per recording top N tracks per artist bucket by tempo vector LSH per bucket GBK GBK GBK RBK top tracks per user + bucket side input Cloud Datastore
- 65.
- 66.
- 67. 67 What’s the catch? Earlystage, some rough edges No interactive mode → Scio REPL (WIP), BigQuery + Datalab No machine learning → TensorFlow Licensed under Apache 2, contribution welcome!
- 68.
- 69. 69 Blog posts @labs.spotify.com Spotify’s Event Delivery - The Road To The Cloud Part I, Part II, Part III
- 70. 7070 Thank You Igor Maravić<igor@spotify.com> Neville Li <neville@spotify.com>