More Related Content
Similar to Crossing the Production Barrier: Development at Scale
Python + NoSQL in AnimationsShuen-Huei Guan
Progressive Enhancement 2.0 (Conference Agnostic)Nicholas Zakas
What if everything is awesome? Codemotion Madrid 2014Christian Heilmann
From IoT to Software MiniaturisationPtidej Team
Open and Accessible UIMark Meeker
Introducing Intelligence Into Your Malware AnalysisBrian Baskin
Recently uploaded
EuroBSDCon 2023 - (auto)Installing BSD Systems - Cases using pfSense, TrueNAS...Vinícius Zavam
Nymity Framework: Privacy & Data Protection Update in 7 StatesTrustArc
Framing Few Shot Knowledge Graph Completion with Large Language ModelsMODUL Technology GmbH
Mitigating Third-Party Risks: Best Practices for CISOs in Ensuring Robust Sec...TrustArc
Google Cloud Study Jams Info SessionGDSCPCCE
Need for Speed: Removing speed bumps in API ProjectsŁukasz Chruściel
Crossing the Production Barrier: Development at Scale
- 2. The world’s handmade marketplace platform for people to sell homemade, crafts, and vintage goods
- 8. big cluster, 20 shards and adding 5 more
- 9. over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 10. 4TBInnoDBbufferpool over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 11. 4TBInnoDBbufferpool 20TB+datastored over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 12. 60K+queries/secavg 4TBInnoDBbufferpool 20TB+datastored over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 13. 60K+queries/secavg 4TBInnoDBbufferpool 20TB+datastored ~1.2Gbpsoutbound(plaintext) over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 14. 60K+queries/secavg 4TBInnoDBbufferpool 20TB+datastored 99.99%queriesunder1ms ~1.2Gbpsoutbound(plaintext) over 40% increase from last year in QPS (25K last year) additional 30K moving over from postgres 1/3 RAM not dedicated to the pool (OS, disk, network buffers, etc)
- 16. TheProblem been around since ’05, hit this a few years ago, every big company probably has this issue
- 17. DATA sync prod to dev, until prod data gets too big http://www.flickr.com/photos/uwwresnet/6280880034/sizes/l/in/ photostream/
- 18. SomeApproaches subsets have to end somewhere (a shop has favorites that are connected to people, connected to shops, etc) generated data can be time consuming to fake
- 19. SomeApproaches subsetsofdata subsets have to end somewhere (a shop has favorites that are connected to people, connected to shops, etc) generated data can be time consuming to fake
- 20. SomeApproaches subsetsofdata generateddata subsets have to end somewhere (a shop has favorites that are connected to people, connected to shops, etc) generated data can be time consuming to fake
- 21. But... but there is a problem with both of those approaches
- 22. EdgeCases what about testing edge cases, difficult to diagnose bugs? hard to model the same data set that produced a user facing bug http://www.flickr.com/photos/sovietuk/141381675/sizes/l/in/ photostream/
- 23. Perspective another issue is testing problems at scale, complex and large gobs of data real social network ecosystem can be difficult to generate (favorites, follows) (activity feed, “similar items” search gives better results) http://www.flickr.com/photos/donsolo/2136923757/sizes/l/in/ photostream/
- 24. Prod Dev? what most people do before data gets too big, almost 2 days to sync 20Tb over 1Gbps link, 5 hrs over 10Gbps bringing prod dataset to dev was expensive hardware/maint, keeping parity with prod, and applying schema changes would take at least as long
- 25. UseProduction so we did what we saw as the last resort - used production not for greenfield development, more for mature features and diagnosing bugs we still have a dev database but the data is sparse and unreliable
- 26. UseProduction (sometimes) so we did what we saw as the last resort - used production not for greenfield development, more for mature features and diagnosing bugs we still have a dev database but the data is sparse and unreliable
- 27. goes without saying this can be dangerous also difficult if done right, we’ve been working on this for a year http://www.flickr.com/photos/stuckincustoms/432361985/sizes/l/in/ photostream/
- 28. Approach two big things: cultural and technical
- 29. SolveCultureIssuesFirst part of figuring this out was exhausting all other options getting buy-in from major stakeholders
- 32. step1: makeitsafe how to have test data in production, prevent stupid mistakes
- 33. phasedrollout
- 37. How? how did we do it?
- 38. QuickOverview high level view http://www.flickr.com/photos/h-k-d/7852444560/sizes/o/in/ photostream/
- 39. tickets index shard1 shard2 shardN
- 40. tickets index shard1 shard2 shardN UniqueIDs
- 41. tickets index shard1 shard2 shardN ShardLookup
- 42. tickets index shard1 shard2 shardN Store/RetrieveData
- 43. devshard introducing.... dev shard, shard used for initial writes of data created when coming from dev env
- 44. tickets index shard1 shard2 shardN
- 45. tickets index shard1 shard2 shardN DEVshard
- 46. shard1 shard2 shardN DEVshard www.etsy.com www.goulah.vm InitialWrites
- 47. shard1 shard2 shardN DEVshard www.etsy.com www.goulah.vm InitialWrites
- 48. shard1 shard2 shardN DEVshard www.etsy.com www.goulah.vm InitialWrites
- 49. mysqlproxy
- 50. proxy hits all of the shards/index/tickets http://www.oreillynet.com/pub/a/databases/2007/07/12/getting-started-with-mysql-proxy.html
- 51. dangerous/unnecessaryqueries -- filter dangerous queries - (queries without a WHERE) -- remove unnecessary queries - (instead of DELETE, have a flag, ALTER statements don’t run from dev)
- 52. dangerous/unnecessaryqueries (DEV) etsy_rw@jgoulah [test]> select * from fred_test; -- filter dangerous queries - (queries without a WHERE) -- remove unnecessary queries - (instead of DELETE, have a flag, ALTER statements don’t run from dev)
- 53. dangerous/unnecessaryqueries (DEV) etsy_rw@jgoulah [test]> select * from fred_test; ERROR 9001 (E9001): Selects from tables must have where clauses -- filter dangerous queries - (queries without a WHERE) -- remove unnecessary queries - (instead of DELETE, have a flag, ALTER statements don’t run from dev)
- 54. knownin/egressfunnel we know where all of the queries from dev originate from http://www.flickr.com/photos/medevac71/4875526920/sizes/l/in/ photostream/
- 55. explicitlyenabled % dev_proxy on Dev-Proxy config is now ON. Use 'dev_proxy off' to turn it off. Not on all the time
- 57. notify engineers they are using the proxy, this is read-only mode
- 58. read/writemode
- 59. read-write mode, needed for login and other things that write data
- 60. stealthdata hiding data from users (favorites go on dev and prod shard, making sure test user/shops don’t show up) http://www.flickr.com/photos/davidyuweb/8063097077/sizes/h/in/ photostream/
- 62. PCI token exchange only, locked down for most people
- 63. PCI off-limits token exchange only, locked down for most people
- 64. anomalydetection another part of our security setup is detection
- 65. logging basics of anomaly detection is log collection
- 66. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table;
- 67. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date
- 68. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid
- 69. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid sourceip
- 70. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid sourceip uniqueidgeneratedbyproxy
- 71. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid sourceip uniqueidgeneratedbyproxy apprequestid
- 72. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid sourceip uniqueidgeneratedbyproxy apprequestid dest.shard
- 73. 2013-04-22 18:05:43 485370821 devproxy -- /* DEVPROXY source=10.101.194.19:40198 uuid=c309e8db-ca32-4171-9c4a-6c37d9dd3361 [htSp8458VmHlC] [etsy_index_B] [browse.php] */ SELECT id FROM table; date threadid sourceip uniqueidgeneratedbyproxy apprequestid dest.shard script
- 75. login-as (read only, logged w/ reason for access)
- 79. Recovery
- 84. DelayedSlaves pt-slave-delay watches a slave and starts and stops its replication SQL thread as necessary to hold it http://www.flickr.com/photos/xploded/141295823/sizes/o/in/ photostream/
- 85. DelayedSlaves role of the delayed slave also source of BCP (business continuity planning - prevention and recovery of threats)
- 86. 4hourdelaybehindmaster DelayedSlaves role of the delayed slave also source of BCP (business continuity planning - prevention and recovery of threats)
- 87. 4hourdelaybehindmaster producerowbasedbinarylogs DelayedSlaves role of the delayed slave also source of BCP (business continuity planning - prevention and recovery of threats)
- 88. 4hourdelaybehindmaster producerowbasedbinarylogs DelayedSlaves allowforquickrecovery role of the delayed slave also source of BCP (business continuity planning - prevention and recovery of threats)
- 89. pt-slave-delay--daemonize --pid/var/run/pt-slave-delay.pid--log/var/log/pt-slave-delay.log --delay4h--interval1m--nocontinue last 3 options most important, 4h delay, interval is how frequently it should check whether slave should be started or stopped nocontinue - don’t continue replication normally on exitx user/pass eliminated for brevity
- 93. R/W R/W Slave ShardPair HDFS Vertica Parse/ Transform in addition can use slaves to send data to other stores for offline queries 1)parse each binlog file to generate sequence file of row changes 2)apply the row changes to a previous set for the latest version
- 94. somethingbadhappens... bad query is run (bad update, etc) http://www.flickr.com/photos/focalintent/1332072795/sizes/o/in/ photostream/
- 95. A B Slave BeforeRestoration.... master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 96. A B Slave BeforeRestoration.... 1)stopdelayed slavereplication master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 97. B Slave BeforeRestoration.... 1)stopdelayed slavereplication 2)pull sideA A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 98. B Slave BeforeRestoration.... 3)stopmaster-masterreplication 1)stopdelayed slavereplication 2)pull sideA A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 99. > SHOW SLAVE STATUS Relay_Log_File: dbslave-relay.007178 Relay_Log_Pos: 8666654 ondelayedslave get the relay position
- 100. mysql> show relaylog events in "dbslave-relay.007178" from 8666654 limit 1G *************************** 1. row ******************* Log_name: dbslave-relay.007178 Pos: 8666654 Event_type: Query Server_id: 1016572 End_log_pos: 8666565 Info: use `etsy_shard`; /* [CVmkWxhD7gsatX8hLbkDoHk29iKo] [etsy_shard_001_B] [/ your/activity/index.php] */ UPDATE `news_feed_stats` SET `time_last_viewed` = 1366406780, `update_time` = 1366406780 WHERE `owner_id` = 30793071 AND `owner_type_id` = 2 AND `feed_type` = 'owner' 2 rows in set (0.00 sec) ondelayedslave show relaylog events will show statements from relay log pass relay log and position to start
- 101. filterbadqueries cycle through all the logs, analyze Query events rotate events - next log file last relay log points to binlog master (server_id is masters, binlog coord matches master_log_file/pos) http://www.flickr.com/photos/chriswaits/6607823843/sizes/l/in/ photostream/
- 102. B Slave AfterDelayedSlaveDataIsRestored.... A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 103. B Slave AfterDelayedSlaveDataIsRestored.... 1)stop mysqlonA andslave A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 104. B Slave AfterDelayedSlaveDataIsRestored.... 1)stop mysqlonA andslave 2)copy datafiles toA A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 105. B Slave AfterDelayedSlaveDataIsRestored.... 1)stop mysqlonA andslave 2)copy datafiles toA 3)restartBtoAreplication, letAcatchuptoB A master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 106. Slave AfterDelayedSlaveDataIsRestored.... 1)stop mysqlonA andslave 2)copy datafiles toA 3)restartBtoAreplication, letAcatchuptoB A 4)restartAtoBreplication, putAbackin,thenpullB A B master.info should be pointing to the right place step 2 could be flipping physical box (for faster recovery such as index servers)
- 107. OtherFormsofRecovery MigrateSingleObject(user/shop/etc) HadoopDeltas Backup+Binlogs migrate object from delayed slave (similar to shard migration) can generate deltas from hadoop if delayed slave has “played” the bad data, go from last nights backup (slower)
- 108. UseCases what are some use cases? http://www.flickr.com/photos/seatbelt67/502255276/sizes/o/ in/photostream/
- 109. userreportsabug... a user files a bug, i can trace the code for the exact page they're on right from my dev machine
- 110. testing“dry”writes testing how application runs a “dry” write -- r/o mode, exception is thrown with the exact query it would have attempted to run, the values it tried to use, etc.
- 111. searchadscampaign consistency starting campaigns and maintaining consistency for entire ad system is nearly impossible in dev Search ads data is stored in more than a dozen DB tables and state changes are driven by a combination of browsers triggering ads, sellers managing their campaigns, and a slew of crons running anywhere from once per 5 minutes to once a month eg) to test pausing campaigns that run out of money mid-day, can pull large numbers of campaigns from prod and operate on those to verify that the data will still be consistent
- 112. googleproductlistingads GPLA is where we syndicate our listings to google to be used in google product search ads we can test edge cases in GPLA syndication where it would be difficult to recreate the state in dev
- 113. testingprototypes features like similar items search gives better results in production because of the amount of data, allowed us to test the quality of listings a prototype was displaying
- 114. performancetesting need a real data set to test pages like treasury search with lots of threads/avatars/etc the dev data is too sparse, xhprof traces don’t mean anything, missing avatars change perf characteristics
- 115. hadoopgenerated datasets dataset produced from hadoop (recommendations for users, or statistics about usage) but since hadoop is prod data its for prod users/listings/shops, so have to check against prod --- sync to dev would fill dev dbs and data wouldn’t line up (b/c prod data)
- 116. browseslices browse slices have complex population so its easier to test experiment against prod data
- 117. not enough listings to populate the narrower subcategories, and it just takes too long