Downloaded 61 times
Uploaded byconfluent
Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3/Hadoop Continuously
Pinterest moved 100TB of data from MySQL databases to S3 and Hadoop continuously using a new data pipeline. The pipeline uses Kafka to stream database change events in real-time. It incorporates periodic compaction to merge snapshots and deltas into a compact format with 15 minute latency. The new system provides reliability, scalability, and enables features like real-time search and recommendations.
More Related Content
![[DSC Europe 22] Overview of the Databricks Platform - Petar Zecevic](https://cdn.slidesharecdn.com/ss_thumbnails/petarzecevic-databricksoverview-221130080703-c60d93de-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...](https://cdn.slidesharecdn.com/ss_thumbnails/draganberic-lakehousearchitecturewithdeltalakeanddatabricks-221130080712-6e817e95-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3/Hadoop Continuously
Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3/Hadoop Continuously
- 2.
- 3.
- 4.
- 5. Data-driven products • Personalized recommendation • SpamControl •SearchQuality • A/BExperiments • RelatedPins • …
- 6. DataPipeline stats • >1PBdata/day • >10Mmessages/second •>800Bmessages/day • >2,000kafkabrokers • >50,000clienthosts
- 7.
- 8.
- 9.
- 10.
- 11. Dataingestion@Pinterest 2016 Pinterest Services Singer Kafka events Real-time consumers Merced Tracker
- 12. Dataingestion@Pinterest 2016 Pinterest Services Singer Kafka events Real-time consumers Databases Merced Tracker
- 13. Dataingestion@Pinterest 2016 Pinterest Services Singer Kafka events Real-time consumers Databases Merced Tracker
- 14. Dataingestion@Pinterest 2016 Pinterest Services Singer Kafka events Real-time consumers Databases Logical backup Merced Tracker
- 15. DBingestion@Pinterest Version1 DatabasesShard1 Slave Shard1 DrSlave Shard1Master Mysqldump Hadoop Streaming Mapper1 Shard2 Slave Shard2 DrSlave Shard2 Master Mysqldump Hadoop Streaming Mapper2
- 16. DBingestion@Pinterest Version2 Databases logical csv backup Tracker Version1 Shard1 Slave Shard1 DrSlave Shard1 Master Mysqldump Hadoop Streaming Mapper1 Shard2 Slave Shard2 DrSlave Shard2 Master Mysqldump Hadoop Streaming Mapper2
- 17. Painpoints Constraints • Reliabilitycausedbymysqlhostshiccup • Pullingover100TBdatadailybutonlyafewTB changedeveryday •Longlatency>24hour Future:DBChangeStreams • Trulycapturesdbtransactions • Across-regioncacheinvalidation • Realtimesearchindexbuilding • RealtimeRecommendationEngine
- 18.
- 19.
- 20.
- 21.
- 22.
- 23. Dataingestion@Pinterest now Pinterest Services Singer Kafka events Real-time consumers Databases DB/Kafka Bridge Merced Watermill
- 24.
- 25. DB/KafkaBridge Replica-SetNode Maxwell_position Maxwell_schema MySQL Processes andSchemas Maxwell Tables Binlog File Shard1 Shard2 Shard3 User Tables
- 26. DB/KafkaBridge Replica-SetNode Maxwell_position Maxwell_schema MySQL Processes andSchemas Maxwell Tables MySQL Processes (Co-located with MySQL Process) Binlog File Shard1 Shard2 Shard3 User Tables Kafka User Topic Kafka Pin Topic BinLog Tailer Thread InMemory Queue Async Kafka Producer Thread • BasedonMaxwell/Binlog-Connector • AddGTIDsupport • Addhandlingforretry/out-of-ordermessages • Co-locatewithmysql • Listensonmaster/slave
- 27.
- 28. Compaction ForOneShard • HashJoinbetweensnapshotanddelta • Deltaloadedinmemoryfirstassidelookup •Basesnapshotwaspipedthroughthemappernodeand compareagainstlookuptable - Lookupfail,snapshotrecordemittooutput - Lookupsucceed,butsnapshotrecordold,skipthe snapshot - Lookupsucceed,butsnapshotrecordnewer,remove lookuprecord • Attheend,appendtheremaininglookuprecordstooutput Delta Shard 1 Old Snapshot Shard 1 Compactor New Snapshot Shard 1
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46. Consistency • MySQLMaster/SlaveFailover,ShardMigration • MySQLTransactions: •Splitbetweentables,splitbetweenKafkamessages • Ordering • BetweenINSERTandUPDATE • BetweenUPDATEandDELETE • SoftDELETEvs.HardDELETE • Consistencybetweenmultiplebootstrapand incrementalstreams • DuplicateRecords
- 47. Scalability • Partitioning • ShardedMySQL -Shardbaseddbsnapshotanddeltafiles - Twolevelsharinginthecasethatoriginalshardsarenot balanced • UnShardeddataset - Usehash+modtopartitionthedataonbothsnapshot anddeltafile • Filefilteringusingpredicatepushdown: • Onshard/partitionlevel • OnS3directory,fileandrecordlevel 10X
- 48. KafkaNuances • MessageOrdering: • Asyncproducerbutstillneedtomaintainmessageorder •MaintainorderbetweenS3fileandwithinS3file • At-least-oncedelivery • Duplicatemessages • MySQLGTIDnotalwaysincreasing • DealwithKafkaclusterhiccup: • produceracks=2 • cleanleaderelection
- 49. S3Nuances ● Eventual Consistency ●Read-after-write is OK, but not PUT followed by LIST ● Directory listing is slow ● Shorter SLA —> More smaller files ● In early iterations, directly listing >> file content reading ● Rate Limit: ● Launching thousands of mappers would quickly hit S3 rate limit
- 50.
- 51. PIIProcessing • username,emailaddressetcneedstobe filteredout • ipaddressneedstobefilteredout john.doe@abc.com JustinBieber 192.168.0.1 ColumnarLayout andIncremental Processing • Useparquetformattosupportfastquerieson subsetofcolumns • ingest_timeasnewcolumntogetthe incrementalresultsincethelastprocessing;
- 52.
- 53.
- 54. Bootstrap,synchronize&rewind(cont) • Wehavetheabilitytosynchronizeandrewind • Incaseofsoftwarebugsornetworkglitches •Snapshot(s)ontoBootstraptosynchronize • AbilitytorewindviatheSnapshots/Bootstrapmechanism MySQL Tracker Batch Backup Backup Snapshot Maxwell Merced Periodic Compaction Snapshot1 Delta Snapshot2 Bootstrapper Kafka
- 55. Schema Management andSchema Change • SchemaisUsedfor • Identifytheprimarykeyoftherow •Drivetheparquetfilegeneration • DealingWithSchemachange • Willissueanewbootstraponofflinetableschema • Compactionwillstillusethesnapshotschema (whichmightbeold) ID C1 C2 123 …. … 124 … …. 125 …. … 126 … …. dbname.table_name new_column …. … …. …
- 56. Validation • Validation • CreatingcompactionbasedonfromandtoGTIDrange •Compactionoutputvsbatchbackupoutput • Monitoring • Error,failure,stall • Latencyoncompaction Backup Snapshot Periodic Compaction Snapshot1 Snapshot2 Differ Bootstrapper
- 57.
- 58.
- 59. Comparison toother technologies • UberHudi(Hoodie) • NotsupportingS3,OnlysupportJava8+,Avro •KafkaConnect • Onlyingestion,nocompacting,synchronizebetween bootstrap/incremental
- 60. Comparison toother technologies • UberHudi(Hoodie) • NotsupportingS3,OnlysupportJava8+,Avro •KafkaConnect/Debezium • Onlyingestion,nocompacting,synchronizebetween bootstrap/incremental • ApacheSqoop • BasedonBatchMode
- 61. Takeaway • Scalability • support100TBofdatabasedata •E2Elatencyof15minutes • Reliability • Strongdatabaseconsistencyonglobaltransactions, messageordering,duplicatemessagehandling • ValidationandMonitoring • Operability • Bootstrap,re-synchronize • Schemamanagement
- 62. Futurework • AdoptingKafkaExact-OnceProcessing Model • Kafkaasthedatabasechangestream •Cacheinvalidationacrossdatacenters • BuildingMaterializedViewsforMySQL • GeneratingIncrementalRecommendationSignals • OpenSource
- 63.
- 65.
- 66. © Copyright, AllRights Reserved, Pinterest Inc. 2018