Download as PDF, PPTX
![Search 101: Inverted Index
/tree/A: “the quick brown fox jumps over …”
/tree/B: “the fox is red”
“the”
[A, B]
“quick”
[A]
“brown”
[A]
“fox”
[A, B]
“red”
[B]](https://image.slidesharecdn.com/stripectf3-140212191208-phpapp02/85/Stripe-CTF3-wrap-up-47-320.jpg)
![Search 102: Arbitrary Substring
● “trigram index”
● Store an inverted index of trigrams
● “the quick brown fox …” →
“the”, “he_”, “e_q”, “_qu”, “qui”, “uic”, …
● To query, look up all trigrams in the search
term, and intersect
● Search(“rown”) →
index[“row”] ∩ index[“own”]
○ Check each result to verify the match](https://image.slidesharecdn.com/stripectf3-140212191208-phpapp02/85/Stripe-CTF3-wrap-up-48-320.jpg)
![Handling substrings
● Option A
○ substrings : word → [all words containing that word]
○ index : word → [list of lines containing that full word]
○
○
for word in substrings[query]:
results += index[word]
○ Can compute substrings table by brute search
○ When indexing lines, just split into words
● Option B
○ index : word → [all lines containing that word,
including as a substring]
○ Need to do the substring search as you index each
line](https://image.slidesharecdn.com/stripectf3-140212191208-phpapp02/85/Stripe-CTF3-wrap-up-51-320.jpg)
Uploaded byStripe
Stripe CTF3 wrap-up
The document outlines a capture-the-flag (CTF) competition involving over 7,500 participants and discusses various technical challenges and solutions related to scaling, DDoS defense, and distributed systems. It also covers topics such as hashing techniques, SQL clusters, and consensus algorithms, mentioning specific strategies and implementations used by contestants. The content reflects a mix of theoretical concepts and practical examples in the context of system architecture and performance optimization.
Stripe CTF3 wrap-up
- 1. Greg Brockman Andy Brody ChristianAnderson Philipp Antoni Carl Jackson Jonas Schneider Siddarth Chandrasekaran Ludwig Pettersson Nelson Elhage Steve Woodrow Jorge Ortiz
- 2. Participation ● ~7.5k participantsfrom 97 countries, 6 continents ● ~9.5k unique IP logins ● 216 capturers
- 3.
- 4. CAPTURE THE FLAG ARCHITECTURE Or,an illustrated guide to flowcharts
- 5. What’s the holygrail of scaling? Ø
- 6. Challenges Scaling up tounknown capacity - 53 instances, 800 ECU at peak User isolation Reliability and Availability (*-abilities)
- 7. git push: whathappens?
- 8. git push →lvl0-asdf@stripe-ctf.com:level0 stripe-ctf.com. IN A ctfweb ctfweb ctfweb (sinatra) gate gate https://stripe-ctf.com/ (nginx, haproxy) submitter submitter submitter (poseidon) colossus (LDAP, scoring) (poseidon) (poseidon) test case test case test case generator test case generator test case generator build generator generator (sinatra) (sinatra) queue queue ctfdb ctfdb ctfdb (mongo) (mongo) (mongo) (RabbitMQ) . (RabbitMQ) (docker) test case test case test case generator test case generator test case generator worker generator generator (docker) gitcoin test case test case test case generator test case generator test case test case generator generator. generator gen
- 9. What Went Wrong –containerization – garbage collection - containers, filesystems, disk space – system stability – bugs, misconfiguration
- 10. What Went Right –containerization – service architecture - queueing, separation of roles – load balancing – horizontal scaling
- 11. Level 0 The MysteriousProgram
- 12.
- 13. Level 0: mmap ●mmap,munmap - map or unmap files or devices into memory ● mmap the dictionary into memory ● You can actually mmap stdin as well! ● Binary search!
- 14. Level 0: Bloomfilters ● ● ● ● ● Hash function: f(str) => int Look at the result of N hash functions. Probabilistic. False positives, but no false negatives. If you run into false positives, just push again!
- 15. Level 0: Minimalperfect hashing Given dictionary D = {w₁, w₂, … wn} use MATH to generate a hash function f f : D → {0..n-1} is one-to-one aka every word hashes to a different small integer ● So you can build a no-collisions hash table ● CMPH - C Minimal Perfect Hashing Library ● Build this ahead of time, link it to the binary ● ● ● ●
- 16.
- 17. Gitcoin commit 000000216ba61aecaafb11135ee43b1674855d6ff7 Author: AlyssaP Hacker <alyssa@example.com> Date: Wed Jan 22 14:10:15 2014 -0800 Give myself a Gitcoin! nonce: tahf8buC diff --git a/LEDGER.txt b/LEDGER.txt index 3890681..41980b2 100644 --- a/LEDGER.txt +++ b/LEDGER.txt @@ -7,3 +7,4 @@ carl: 30 gdb: 12 nelhage: 45 jorge: 30 +user-aph123: 1
- 18. Why crypto currencies? –distributed currency system – git security model – massively parallel problems – using the right tools for the job
- 19.
- 20. Git object model $git cat-file -p 000000effe7d920b391a24633e7298469dcf51b5 tree 7da86a5b10ff6db916598b653ce63e1dc0cb73c8 parent 0000000df4815161b72f4c5ed23e9fbf5deed922 author Alyssa P Hacker <alyssa@example.com> 1391129313 +0000 committer Alyssa P Hacker <alyssa@example.com> 1391129313 +0000 Mined a Gitcoin! nonce 0302d1e2 $ git show 000000 error: short SHA1 000000 is ambiguous. error: short SHA1 000000 is ambiguous.
- 21. Remember wafflecopter? Want todo as few rounds of SHA1 as possible Compute prefix, update only for nonce
- 22. SHA1 is EmbarrassinglyParallel – Each miner can be totally independent – Each miner requires little memory – Each miner requires little code
- 23. Tools: a spoon(bash)
- 24. Tools: a shovel(go)
- 25. Tools: an armyof backhoes (GPU)
- 26. Tools: big machines(ASIC)
- 27. Bonus Round Hash Rates: bash:400 H/s our go miners: 1.9 MH/s 100 cores EC2: 130 MH/s GPU: 1-2 GH/s Network: ~10 GH/s
- 28. WE DID IT! Dogecoinis only at 80 GH/s
- 29.
- 30. Elephants and mice:a DDOS model
- 31. Elephants and mice:a DDOS model
- 32. The stub ofa Node.js proxy
- 33. Balance across thebackends
- 34. Balance across thebackends 1. Round robin
- 35. Balance across thebackends 1. Round robin 2. Choose backend with min. load
- 36. Balance across thebackends 1. Round robin 2. Choose backend with min. load 3. Randomize
- 37. Let the micethrough
- 38. Let the micethrough 1. Reduce the overall load
- 39. Let the micethrough 1. Reduce the overall load 2. Use an off-the-shelf solution
- 40. Let the micethrough 1. Reduce the overall load 2. Use an off-the-shelf solution 3. Learn to recognize a mouse
- 41. If you hada global view
- 42. Recognize a mouse 1.Threshold rate or number 2. Learn by hand or with automation
- 43. The top solutions ➔Balance load in a simple way ➔ Learn to recognize a mouse ➔ Keep an eye on the backends
- 44. Level 3 Instant CodeSearch
- 45. We’re sorry aboutthe Scala
- 46. The problem ● Textsearch over ~100M of text ● Arbitrary substring search (not just whole words) ● There is a “free” indexing stage ● Distribute across up to 4 nodes ● Each node is limited to 500M of RAM
- 47. Search 101: InvertedIndex /tree/A: “the quick brown fox jumps over …” /tree/B: “the fox is red” “the” [A, B] “quick” [A] “brown” [A] “fox” [A, B] “red” [B]
- 48. Search 102: ArbitrarySubstring ● “trigram index” ● Store an inverted index of trigrams ● “the quick brown fox …” → “the”, “he_”, “e_q”, “_qu”, “qui”, “uic”, … ● To query, look up all trigrams in the search term, and intersect ● Search(“rown”) → index[“row”] ∩ index[“own”] ○ Check each result to verify the match
- 49. Sharding ● We giveyou four nodes ● …but they all run on the same physical node during grading ● And we didn’t resource-limit grading containers (other than memory) ● So you don’t actually get more CPU, disk I/O, or memory bandwidth ● Sharding ended up not really mattering
- 50. Winning the contest ●The spec is for arbitrary substring search ● But we only generate/query words from a dictionary ● Some words are substrings of other words… ● … but not too many ● Use an inverted index over dictionary words
- 51. Handling substrings ● OptionA ○ substrings : word → [all words containing that word] ○ index : word → [list of lines containing that full word] ○ ○ for word in substrings[query]: results += index[word] ○ Can compute substrings table by brute search ○ When indexing lines, just split into words ● Option B ○ index : word → [all lines containing that word, including as a substring] ○ Need to do the substring search as you index each line
- 52. Other ways tobeat the level ● Slurp the entire tree into RAM and use a decent string search ○ (not java.lang.String.indexOf() -- that’s slow!) ● Shell out to grep ○ GNU grep is fast
- 53.
- 54. SQLCluster ● ● ● ● 5 SQLite nodes Queriessubmitted to all nodes Random network and node failures Must maintain full linearizability of queries
- 55.
- 56. Octopus ● (Grumpy) networksimulator ● Submits queries and checks for correctness ● Several “monkeys” manipulate the network: ○ ○ ○ ○ Netsplit monkey Lagsplit monkey SPOF monkey etc.
- 57. Consensus Algorithms ● Raft(“In Search of an Understandable Consensus Algorithm”, Diego Ongaro and John Ousterhout, 2013) ● Zab (“Zab: High-performance broadcast for primary-backup systems”, Flavio P. Junqueira, Benjamin C. Reed, and Marco Serafini, 2011) ● Paxos (“The Part-Time Parliament”, Leslie Lamport, 1998, originally submitted 1990) ● Viewstamped Replication (“Viewstamped Replication: A New Primary Copy Method to Support Highly Available Distributed Systems”, Brian M. Oki and Barbara H. Liskov, 1988) etc. etc.
- 58. Consensus Algorithms Almost everyonechose Raft https://github.com/goraft/raft
- 59. Gotchas: Idempotency ● Octopussends node1 a commit ● Node1 forwards it to the leader, node0 ● Node0 processes it and sends it back ● Octopus kills the node0 ⇔ node1 link vs. ● Octopus sends node1 a commit ● Node1 forwards it to the leader, node0 ● Octopus kills the node0 ⇔ node1 link
- 60. Gotchas: Idempotency ● Howdo you tell between these two cases? ● Naive: resubmit the query to find out! ● If the query was processed, return the old result ● Common trick: Idempotency tokens UPDATE ctf3 SET friendCount=friendCount+10, requestCount=requestCount+1, favoriteWord=" jjfqcjamhpghnqq" WHERE name="carl"; SELECT * FROM ctf3
- 61. Making it Fast ●Every top solution replaced sql.go ● Two main strategies: ○ Write your own sqlite (or enough of it to pass) ○ Use sqlite bindings, :memory: database ● These perform roughly equally well ● Raft has a few timers you can tune ● Golf network traffic
- 62.
- 63. First Solve: SingleMaster 0 4 1 3 2
- 64.
- 65. Single Point ofFailure Detection 0 4 1 3 2
- 66.
- 67.
- 68.
- 69. Leaderboard Top 10 ● ● ● ● Everyonechanged SQLite “bindings” Seven solutions used go-raft Two solutions used redirect-to-master One solution implemented Raft in C++ ● If at first you don’t succeed… ○ Mean submissions: 1444 (stddev 1031) ○ Max: 3946 ○ Min: 58 (the C++ solution)
- 70. Speculative: Time-based consensus ● ● ● ● ● Allnodes were run on the same host Local clock: total ordering on events Leaderless state machine ??? Profit? ● Similar ideas to Spanner (Google)
- 71. Greg Brockman Andy Brody ChristianAnderson Philipp Antoni Carl Jackson Jonas Schneider Siddarth Chandrasekaran Ludwig Pettersson Nelson Elhage Steve Woodrow Jorge Ortiz