Testing This                           TESTING     TESTING                           TESTING     TESTINGThursday, April 21...
ZOMG!Thursday, April 21, 2011
GOOD MORNING!Thursday, April 21, 2011
Aaron PattersonThursday, April 21, 2011
@tenderloveThursday, April 21, 2011
WWFMD?Thursday, April 21, 2011
ITWFMWD?Thursday, April 21, 2011
Thursday, April 21, 2011
AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.T...
Miami Vice Principal                    Señor Facebook                       Integration                  Engineering Mana...
ElectThursday, April 21, 2011
Our Time Together                                               12% 2%                                                    ...
Thursday, April 21, 2011
KEYNOTEThursday, April 21, 2011
Our Time Together                                                 10%                                                     ...
Thursday, April 21, 2011
20% More                           TransitionsThursday, April 21, 2011
I enjoy dealing         with behinds         the scenes         information                           Living Behind       ...
DATABASE                           APPLICATION                             CLIENTThursday, April 21, 2011
DATABASE                           APPLICATION                             CLIENTThursday, April 21, 2011
DATABASE                           APPLICATION                             CLIENTThursday, April 21, 2011
DATABASE                           APPLICATION                             CLIENTThursday, April 21, 2011
Code as DataThursday, April 21, 2011
PoliticsThursday, April 21, 2011
We dont vote on                                        commits. It                                        would be too    ...
OSS is an OligarchyThursday, April 21, 2011
Benevolent DictatorThursday, April 21, 2011
Small projects       Large projects have                           have few people,     too many people so                ...
Thursday, April 21, 2011
Care and Feeding of                      Your TyrantThursday, April 21, 2011
CheetosThursday, April 21, 2011
Good                            reproduction                            steps                           Feed them Quality ...
Feed them Patches                               with TestsThursday, April 21, 2011
Gemspec,                                         Jewler                                         Hoe,                      ...
leave test                           frameworksThursday, April 21, 2011
Use +1s CarefullyThursday, April 21, 2011
Thursday, April 21, 2011
Thursday, April 21, 2011
320 commentsThursday, April 21, 2011
210 plus oneThursday, April 21, 2011
Thursday, April 21, 2011
FFFFFFFUUUUU                 UUUUUUUThursday, April 21, 2011
Thursday, April 21, 2011
SuperficialThursday, April 21, 2011
MeritThursday, April 21, 2011
QualityThursday, April 21, 2011
Well discuss                                        this in closing.                           How to fix?Thursday, April 2...
Encodings                            the silent killerThursday, April 21, 2011
Data goes in, Data goes out                  Never a miscommunicationThursday, April 21, 2011
Until there isThursday, April 21, 2011
`encode: "xE9" from ASCII-8BIT to UTF-8                (Encoding::UndefinedConversionError)Thursday, April 21, 2011
Encoding Error?                           You cant explain thatThursday, April 21, 2011
String TaggingThursday, April 21, 2011
Strings with                                               unknown                                               encoding ...
socket.read(4).encoding                   # => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
To understand                                    these bugs, we                                    must find              ...
App CodeThursday, April 21, 2011
Sadness                           Web Server                                          Belt                  NoSQL    App C...
Databases         Post contains                           know the          encoding /                           stored en...
# encoding: utf-8                name =                             user = User.create!(:name => name)                user...
# encoding: utf-8                name =                             user = User.create!(:name => name)                user...
>>         name = "Aaron Patterson"                >>         name.encoding                =>         #<Encoding:UTF-8>   ...
>>         name = "Aaron Patterson"                >>         name.encoding                =>         #<Encoding:UTF-8>   ...
Ruby attempts                                       to convert the                                       binary for you   ...
Ruby attempts                                         to convert the                                         binary for yo...
ASCII-8BIT + other                             should raiseThursday, April 21, 2011
Aaro                                          Opin  ns                                               ion                  ...
Resources Know                              EncodingThursday, April 21, 2011
Prepared StatementsThursday, April 21, 2011
SELECT * FROM "users" WHERE id = 1Thursday, April 21, 2011
Normal Queries                           Parse the Query                           Formulate an Execution Plan            ...
Application                            DatabaseThursday, April 21, 2011
Application                           Result Set                            DatabaseThursday, April 21, 2011
Application                            DatabaseThursday, April 21, 2011
Application                           query: 1234                            DatabaseThursday, April 21, 2011
Application                            DatabaseThursday, April 21, 2011
Application                           Result Set                            DatabaseThursday, April 21, 2011
Preparation                           Parse the Query                           Formulate an Execution PlanThursday, April...
Execution                           Return ResultsThursday, April 21, 2011
Data transfer                            decreasesThursday, April 21, 2011
Query Planner                             ImprovesThursday, April 21, 2011
SecurityThursday, April 21, 2011
Talk about                                       problem                                       inserting rows             ...
INSERT INTO "sessions"                ("data", "session_id")                VALUES (?, ?)                [data,           ...
db = SQLite3::Database.new :memory:                db.trace { |sql| puts sql }Thursday, April 21, 2011
INSERT INTO "sessions"                ("data", "session_id")                VALUES (                BAh7BjoIZm9vSSIIYmF6Bj...
INSERT INTO "sessions"                ("data", "session_id")                VALUES (                BAh7BjoIZm9vSSIIYmF6Bj...
Made a sample                                              program to                                              reprodu...
Resulting                                              Query                INSERT INTO "sessions"                (session...
stmt.bind_param 1, fuu.encode(BINARY)                stmt.executeThursday, April 21, 2011
INSERT INTO "sessions" (session_id)                VALUES (x667575)Thursday, April 21, 2011
Why did sqlite3 do                                            this?                                            Talk about ...
Why is it stored as                                Binary?Thursday, April 21, 2011
Why is the session id                     tagged binary?Thursday, April 21, 2011
>>         x = OpenSSL::Random.random_bytes(10)                =>         "Gx93xFCxB2xCExC0xECxBBxA7W"                >>  ...
>>         y = x.unpack(H*)                =>         ["4793fcb2cec0ecbba757"]                >>         y.first.encoding ...
def generate_sid                  ActiveSupport::SecureRandom.hex(16)                endThursday, April 21, 2011
Should hex have tagged                                             it?                                             Should ...
Find the SourceThursday, April 21, 2011
ARel and                           ActiveRecordThursday, April 21, 2011
Fear of SQLThursday, April 21, 2011
Why?Thursday, April 21, 2011
But not                                        specifically SQL                                        Working with       ...
We must learn to                            work with SetsThursday, April 21, 2011
Avoid SQL for:Thursday, April 21, 2011
code reuseThursday, April 21, 2011
securityThursday, April 21, 2011
from databases                           independenceThursday, April 21, 2011
What is ARel?Thursday, April 21, 2011
Represents SQL                       as a Tree                                        AST   Represents the                ...
SQL CompilerThursday, April 21, 2011
Only until we                           invoke the                           compiler, does the                           ...
Relationship to                           ActiveRecordThursday, April 21, 2011
User.where(something)Thursday, April 21, 2011
class User                  scope :heart where(:name => <3)                end                User.heart.select(name)Thurs...
Our Application                                   ActiveRecord                           ARel                     Database...
Our Application                                      .to_a                                   ActiveRecord                 ...
Our Application                                     ActiveRecord                           to_sql                         ...
Our Application                                   ActiveRecord                                               SELECT * ... ...
Our Application                                    ActiveRecord                                  [{: name => <3}]         ...
ActiveRecord::RelationThursday, April 21, 2011
shortens to...Thursday, April 21, 2011
ARelThursday, April 21, 2011
ARel != ARelThursday, April 21, 2011
:(Thursday, April 21, 2011
Future we can                                          have                                          optimizers,          ...
Interpreter?Thursday, April 21, 2011
module Arel       module Visitors         class Mongo < Arel::Visitors::Visitor           attr_reader :db            def i...
module Arel       module Visitors         class Mongo < Arel::Visitors::Visitor           attr_reader :db            def i...
Thursday, April 21, 2011
gist.github.com/845782Thursday, April 21, 2011
Custom DSLThursday, April 21, 2011
Responds to *                 class Select < Struct.new(:columns)                   def self.* other                     o...
Responds to WHERE                 class From < Struct.new(:table, :conditions)                   def WHERE conditions     ...
Responds to to_s                 class Where < Struct.new(:from, :select)                   def to_s                     A...
Bootstrap Methods                 SELECT = Select                 def FROM table                   From.new table         ...
x = SELECT * FROM("users") .WHERE(:id => 10)                puts xThursday, April 21, 2011
Thursday, April 21, 2011
Write SQL Ruby                            to avoid SQLThursday, April 21, 2011
Talked about                                       data from the                                       back end, talk a   ...
Twitter Streaming APIThursday, April 21, 2011
Needed for                                 persistent                                 connections                    Chunk...
Normal Timeline                                        Buffers data,                                        then sends a  ...
Normal Timeline                                        Buffers data,                                        then sends a  ...
Normal Timeline                                              Buffers data,                                              th...
Chunked Timeline                                        Send data as                                        soon as it is ...
Chunked Timeline                                        Send data as                                        soon as it is ...
Chunked Timeline                                             Send data as                                             soon...
ImplementationThursday, April 21, 2011
Rack APIThursday, April 21, 2011
status, headers, body = app.call(env)Thursday, April 21, 2011
status.to_iThursday, April 21, 2011
headers.is_a?(Hash)Thursday, April 21, 2011
body.responds_to?(:each)Thursday, April 21, 2011
Inside Rack                                         Calls each,                                         possibly calls    ...
Sample Application                 class MyApp                   def call(env)                     # some computation     ...
Delay work until each                 class FooBody                   def each                     yield "hello "         ...
Database                                        connections                                        were lost              ...
Database                                        connections are                                        managed in         ...
"There are two hard problems                    in CS: cache invalidation                        and naming things"       ...
class DbCache < Struct.new(:app)                  def call(env)                    # init cache                    status,...
class BodyProxy < Struct.new(:delegate)                  def each                    delegate.each { |x| yield x }        ...
Update Middleware,                       Database Works! ☺Thursday, April 21, 2011
Same problem in                                        query cache as                                        the database ...
Middleware is                           coupled to the                           callstackThursday, April 21, 2011
~ 24 middlewareThursday, April 21, 2011
~ 5 required a proxyThursday, April 21, 2011
Types of Middleware                           Generators           Rack jams these to                                     ...
Solutions!Thursday, April 21, 2011
Embrace The                           Differences!Thursday, April 21, 2011
Lifecycle Listeners                 class Listener                   def created(event)                   end             ...
Generators                   Synchronous or                                                         Asynchronous          ...
Filters                 class Filter                   def filter(request, response, chain)                     chain.filt...
Synchronous                           Asynchronous                           Freedom from the Callstack                   ...
I stole this API.Thursday, April 21, 2011
TEE HEE!!!!!Thursday, April 21, 2011
fixing our tyrantsThursday, April 21, 2011
Facebook + GithubThursday, April 21, 2011
Get InvolvedThursday, April 21, 2011
Only fix the                                        immediate                                        problem              ...
Then PersuadeThursday, April 21, 2011
Thursday, April 21, 2011
Thursday, April 21, 2011
Questions?Thursday, April 21, 2011
Upcoming SlideShare
Loading in …5
×

Behind the Curtain

6,013 views

Published on

My talk at red dirt ruby conf

Published in: Technology
  • Be the first to comment

Behind the Curtain

  1. 1. Testing This TESTING TESTING TESTING TESTINGThursday, April 21, 2011
  2. 2. ZOMG!Thursday, April 21, 2011
  3. 3. GOOD MORNING!Thursday, April 21, 2011
  4. 4. Aaron PattersonThursday, April 21, 2011
  5. 5. @tenderloveThursday, April 21, 2011
  6. 6. WWFMD?Thursday, April 21, 2011
  7. 7. ITWFMWD?Thursday, April 21, 2011
  8. 8. Thursday, April 21, 2011
  9. 9. AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.Thursday, April 21, 2011
  10. 10. Miami Vice Principal Señor Facebook Integration Engineering ManagerThursday, April 21, 2011
  11. 11. ElectThursday, April 21, 2011
  12. 12. Our Time Together 12% 2% 24% 61% Inspirational Entertaining Informative WastedThursday, April 21, 2011
  13. 13. Thursday, April 21, 2011
  14. 14. KEYNOTEThursday, April 21, 2011
  15. 15. Our Time Together 10% 20% 20% 50% Inspirational Entertaining Informative WastedThursday, April 21, 2011
  16. 16. Thursday, April 21, 2011
  17. 17. 20% More TransitionsThursday, April 21, 2011
  18. 18. I enjoy dealing with behinds the scenes information Living Behind the Curtain 6 subjects, less than 10 minutes on each of themThursday, April 21, 2011
  19. 19. DATABASE APPLICATION CLIENTThursday, April 21, 2011
  20. 20. DATABASE APPLICATION CLIENTThursday, April 21, 2011
  21. 21. DATABASE APPLICATION CLIENTThursday, April 21, 2011
  22. 22. DATABASE APPLICATION CLIENTThursday, April 21, 2011
  23. 23. Code as DataThursday, April 21, 2011
  24. 24. PoliticsThursday, April 21, 2011
  25. 25. We dont vote on commits. It would be too inefficient. "Open Source is a Democracy"Thursday, April 21, 2011
  26. 26. OSS is an OligarchyThursday, April 21, 2011
  27. 27. Benevolent DictatorThursday, April 21, 2011
  28. 28. Small projects Large projects have have few people, too many people so so it is efficient it becomes inefficient to individually consider all feedback Tyrannical LeaderThursday, April 21, 2011
  29. 29. Thursday, April 21, 2011
  30. 30. Care and Feeding of Your TyrantThursday, April 21, 2011
  31. 31. CheetosThursday, April 21, 2011
  32. 32. Good reproduction steps Feed them Quality BugsThursday, April 21, 2011
  33. 33. Feed them Patches with TestsThursday, April 21, 2011
  34. 34. Gemspec, Jewler Hoe, etc leave build systemsThursday, April 21, 2011
  35. 35. leave test frameworksThursday, April 21, 2011
  36. 36. Use +1s CarefullyThursday, April 21, 2011
  37. 37. Thursday, April 21, 2011
  38. 38. Thursday, April 21, 2011
  39. 39. 320 commentsThursday, April 21, 2011
  40. 40. 210 plus oneThursday, April 21, 2011
  41. 41. Thursday, April 21, 2011
  42. 42. FFFFFFFUUUUU UUUUUUUThursday, April 21, 2011
  43. 43. Thursday, April 21, 2011
  44. 44. SuperficialThursday, April 21, 2011
  45. 45. MeritThursday, April 21, 2011
  46. 46. QualityThursday, April 21, 2011
  47. 47. Well discuss this in closing. How to fix?Thursday, April 21, 2011
  48. 48. Encodings the silent killerThursday, April 21, 2011
  49. 49. Data goes in, Data goes out Never a miscommunicationThursday, April 21, 2011
  50. 50. Until there isThursday, April 21, 2011
  51. 51. `encode: "xE9" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)Thursday, April 21, 2011
  52. 52. Encoding Error? You cant explain thatThursday, April 21, 2011
  53. 53. String TaggingThursday, April 21, 2011
  54. 54. Strings with unknown encoding are marked as binary. "hello! <3".encoding # => #<Encoding:UTF-8>Thursday, April 21, 2011
  55. 55. socket.read(4).encoding # => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
  56. 56. To understand these bugs, we must find boundaries BoundariesThursday, April 21, 2011
  57. 57. App CodeThursday, April 21, 2011
  58. 58. Sadness Web Server Belt NoSQL App Code File System DatabaseThursday, April 21, 2011
  59. 59. Databases Post contains know the encoding / stored encoding spec declares default #<Encoding:ASCII-8BIT> is usually a bug!Thursday, April 21, 2011
  60. 60. # encoding: utf-8 name = user = User.create!(:name => name) user.reload user.name + nameThursday, April 21, 2011
  61. 61. # encoding: utf-8 name = user = User.create!(:name => name) user.reload user.name + name incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)Thursday, April 21, 2011
  62. 62. >> name = "Aaron Patterson" >> name.encoding => #<Encoding:UTF-8> >> User.create!(:name => name) >> u = User.find :first >> u.name.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
  63. 63. >> name = "Aaron Patterson" >> name.encoding => #<Encoding:UTF-8> >> User.create!(:name => name) >> u = User.find :first >> u.name.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
  64. 64. Ruby attempts to convert the binary for you name = aaron patterson user = User.create!(:name => name) user.reload user.name + nameThursday, April 21, 2011
  65. 65. Ruby attempts to convert the binary for you name = aaron patterson user = User.create!(:name => name) user.reload user.name + name no errorThursday, April 21, 2011
  66. 66. ASCII-8BIT + other should raiseThursday, April 21, 2011
  67. 67. Aaro Opin ns ion Cor ner! ASCII-8BIT + other should raiseThursday, April 21, 2011
  68. 68. Resources Know EncodingThursday, April 21, 2011
  69. 69. Prepared StatementsThursday, April 21, 2011
  70. 70. SELECT * FROM "users" WHERE id = 1Thursday, April 21, 2011
  71. 71. Normal Queries Parse the Query Formulate an Execution Plan Return resultsThursday, April 21, 2011
  72. 72. Application DatabaseThursday, April 21, 2011
  73. 73. Application Result Set DatabaseThursday, April 21, 2011
  74. 74. Application DatabaseThursday, April 21, 2011
  75. 75. Application query: 1234 DatabaseThursday, April 21, 2011
  76. 76. Application DatabaseThursday, April 21, 2011
  77. 77. Application Result Set DatabaseThursday, April 21, 2011
  78. 78. Preparation Parse the Query Formulate an Execution PlanThursday, April 21, 2011
  79. 79. Execution Return ResultsThursday, April 21, 2011
  80. 80. Data transfer decreasesThursday, April 21, 2011
  81. 81. Query Planner ImprovesThursday, April 21, 2011
  82. 82. SecurityThursday, April 21, 2011
  83. 83. Talk about problem inserting rows with session data SQLite3 Issues :( Could insert session data, but could not find itThursday, April 21, 2011
  84. 84. INSERT INTO "sessions" ("data", "session_id") VALUES (?, ?) [data, BAh7BjoIZm9vSSIIYmF6BjoGRUY=] [session_id, dae0fb8a9c34e6980c9d0fae33a8fff6]Thursday, April 21, 2011
  85. 85. db = SQLite3::Database.new :memory: db.trace { |sql| puts sql }Thursday, April 21, 2011
  86. 86. INSERT INTO "sessions" ("data", "session_id") VALUES ( BAh7BjoIZm9vSSIIYmF6BjoGRUY=, x3633303337623066376536613130343132343 765623763626631616439303631)Thursday, April 21, 2011
  87. 87. INSERT INTO "sessions" ("data", "session_id") VALUES ( BAh7BjoIZm9vSSIIYmF6BjoGRUY=, x3633303337623066376536613130343132343 765623763626631616439303631)Thursday, April 21, 2011
  88. 88. Made a sample program to reproduce the db.trace { |sql| puts sql } problem stmt = db.prepare( INSERT INTO "sessions" (session_id) VALUES (?)) stmt.bind_param 1, fuu stmt.executeThursday, April 21, 2011
  89. 89. Resulting Query INSERT INTO "sessions" (session_id) VALUES (fuu)Thursday, April 21, 2011
  90. 90. stmt.bind_param 1, fuu.encode(BINARY) stmt.executeThursday, April 21, 2011
  91. 91. INSERT INTO "sessions" (session_id) VALUES (x667575)Thursday, April 21, 2011
  92. 92. Why did sqlite3 do this? Talk about column affinity INSERT INTO "sessions" (session_id) VALUES (x667575)Thursday, April 21, 2011
  93. 93. Why is it stored as Binary?Thursday, April 21, 2011
  94. 94. Why is the session id tagged binary?Thursday, April 21, 2011
  95. 95. >> x = OpenSSL::Random.random_bytes(10) => "Gx93xFCxB2xCExC0xECxBBxA7W" >> x.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
  96. 96. >> y = x.unpack(H*) => ["4793fcb2cec0ecbba757"] >> y.first.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
  97. 97. def generate_sid ActiveSupport::SecureRandom.hex(16) endThursday, April 21, 2011
  98. 98. Should hex have tagged it? Should have generate_sid tagged it? def generate_sid sid = ActiveSupport::SecureRandom.hex(16) if sid.respond_to?(:encode!) sid.encode!(UTF-8) end sid endThursday, April 21, 2011
  99. 99. Find the SourceThursday, April 21, 2011
  100. 100. ARel and ActiveRecordThursday, April 21, 2011
  101. 101. Fear of SQLThursday, April 21, 2011
  102. 102. Why?Thursday, April 21, 2011
  103. 103. But not specifically SQL Working with sets We must learn SQL Can fetch correct data.Thursday, April 21, 2011
  104. 104. We must learn to work with SetsThursday, April 21, 2011
  105. 105. Avoid SQL for:Thursday, April 21, 2011
  106. 106. code reuseThursday, April 21, 2011
  107. 107. securityThursday, April 21, 2011
  108. 108. from databases independenceThursday, April 21, 2011
  109. 109. What is ARel?Thursday, April 21, 2011
  110. 110. Represents SQL as a Tree AST Represents the IDEA of a SQL statementThursday, April 21, 2011
  111. 111. SQL CompilerThursday, April 21, 2011
  112. 112. Only until we invoke the compiler, does the AST become a query Represents an IDEAThursday, April 21, 2011
  113. 113. Relationship to ActiveRecordThursday, April 21, 2011
  114. 114. User.where(something)Thursday, April 21, 2011
  115. 115. class User scope :heart where(:name => <3) end User.heart.select(name)Thursday, April 21, 2011
  116. 116. Our Application ActiveRecord ARel DatabaseThursday, April 21, 2011
  117. 117. Our Application .to_a ActiveRecord ARel DatabaseThursday, April 21, 2011
  118. 118. Our Application ActiveRecord to_sql ARel DatabaseThursday, April 21, 2011
  119. 119. Our Application ActiveRecord SELECT * ... ARel DatabaseThursday, April 21, 2011
  120. 120. Our Application ActiveRecord [{: name => <3}] ARel DatabaseThursday, April 21, 2011
  121. 121. ActiveRecord::RelationThursday, April 21, 2011
  122. 122. shortens to...Thursday, April 21, 2011
  123. 123. ARelThursday, April 21, 2011
  124. 124. ARel != ARelThursday, April 21, 2011
  125. 125. :(Thursday, April 21, 2011
  126. 126. Future we can have optimizers, compiler cache, etc SQL CompilerThursday, April 21, 2011
  127. 127. Interpreter?Thursday, April 21, 2011
  128. 128. module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions) private def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end def visit_Arel_Nodes_And o o.children.map { |child| visit child } end def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end def visit_Arel_Attributes_Attribute o return if o.name == *; o.name end def visit_Arel_Nodes_JoinSource o visit o.left end def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end end endThursday, April 21, 2011
  129. 129. module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions) private def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end Mongo Interpreter def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end 64 loc def visit_Arel_Nodes_And o o.children.map { |child| visit child } end def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end def visit_Arel_Attributes_Attribute o return if o.name == *; o.name end def visit_Arel_Nodes_JoinSource o visit o.left end def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end end endThursday, April 21, 2011
  130. 130. Thursday, April 21, 2011
  131. 131. gist.github.com/845782Thursday, April 21, 2011
  132. 132. Custom DSLThursday, April 21, 2011
  133. 133. Responds to * class Select < Struct.new(:columns) def self.* other other.select = new(Arel.sql(*)) other end endThursday, April 21, 2011
  134. 134. Responds to WHERE class From < Struct.new(:table, :conditions) def WHERE conditions self.conditions = conditions Where.new(self) end endThursday, April 21, 2011
  135. 135. Responds to to_s class Where < Struct.new(:from, :select) def to_s Arel::Table.engine = Arel::Sql::Engine.new( FakeRecord::Base.new) table = Arel::Table.new from.table table.project(select.columns).where( from.conditions.map { |k,v| table[k].eq v }).to_sql end endThursday, April 21, 2011
  136. 136. Bootstrap Methods SELECT = Select def FROM table From.new table endThursday, April 21, 2011
  137. 137. x = SELECT * FROM("users") .WHERE(:id => 10) puts xThursday, April 21, 2011
  138. 138. Thursday, April 21, 2011
  139. 139. Write SQL Ruby to avoid SQLThursday, April 21, 2011
  140. 140. Talked about data from the back end, talk a bit about data to the front end StreamingThursday, April 21, 2011
  141. 141. Twitter Streaming APIThursday, April 21, 2011
  142. 142. Needed for persistent connections Chunked ResponsesThursday, April 21, 2011
  143. 143. Normal Timeline Buffers data, then sends a responseThursday, April 21, 2011
  144. 144. Normal Timeline Buffers data, then sends a response Processing ERbThursday, April 21, 2011
  145. 145. Normal Timeline Buffers data, then sends a response Processing ERb Download Asset Download AssetThursday, April 21, 2011
  146. 146. Chunked Timeline Send data as soon as it is availableThursday, April 21, 2011
  147. 147. Chunked Timeline Send data as soon as it is available Processing ERbThursday, April 21, 2011
  148. 148. Chunked Timeline Send data as soon as it is available Processing ERb Download Asset Download AssetThursday, April 21, 2011
  149. 149. ImplementationThursday, April 21, 2011
  150. 150. Rack APIThursday, April 21, 2011
  151. 151. status, headers, body = app.call(env)Thursday, April 21, 2011
  152. 152. status.to_iThursday, April 21, 2011
  153. 153. headers.is_a?(Hash)Thursday, April 21, 2011
  154. 154. body.responds_to?(:each)Thursday, April 21, 2011
  155. 155. Inside Rack Calls each, possibly calls close body.each do |chunk| output(chunk) end body.close rescue nilThursday, April 21, 2011
  156. 156. Sample Application class MyApp def call(env) # some computation body = hello # more computation body << world [200, { X-Hello => World }, [body]] end endThursday, April 21, 2011
  157. 157. Delay work until each class FooBody def each yield "hello " sleep(10) # simulate work yield "world!" end endThursday, April 21, 2011
  158. 158. Database connections were lost Problems ☹Thursday, April 21, 2011
  159. 159. Database connections are managed in middleware MiddlewareThursday, April 21, 2011
  160. 160. "There are two hard problems in CS: cache invalidation and naming things" -- Phil KarltonThursday, April 21, 2011
  161. 161. class DbCache < Struct.new(:app) def call(env) # init cache status, headers, body = app.call(env) # clear cache [status, headers, body] end endThursday, April 21, 2011
  162. 162. class BodyProxy < Struct.new(:delegate) def each delegate.each { |x| yield x } end def close delegate.close rescue nil # clear cache end endThursday, April 21, 2011
  163. 163. Update Middleware, Database Works! ☺Thursday, April 21, 2011
  164. 164. Same problem in query cache as the database handle cache. But Query Cache was broken. ☹Thursday, April 21, 2011
  165. 165. Middleware is coupled to the callstackThursday, April 21, 2011
  166. 166. ~ 24 middlewareThursday, April 21, 2011
  167. 167. ~ 5 required a proxyThursday, April 21, 2011
  168. 168. Types of Middleware Generators Rack jams these to one API. Filters Approach seems naive when examining usage Lifecycle Handlers Tied to callstackThursday, April 21, 2011
  169. 169. Solutions!Thursday, April 21, 2011
  170. 170. Embrace The Differences!Thursday, April 21, 2011
  171. 171. Lifecycle Listeners class Listener def created(event) end def destroyed(event) end endThursday, April 21, 2011
  172. 172. Generators Synchronous or Asynchronous class Resource def service(request, response) 10.times { |i| response.body.write "hello #{i}" } response.body.close Return value is end ignored endThursday, April 21, 2011
  173. 173. Filters class Filter def filter(request, response, chain) chain.filter(request, response) end endThursday, April 21, 2011
  174. 174. Synchronous Asynchronous Freedom from the Callstack Thread SafetyThursday, April 21, 2011
  175. 175. I stole this API.Thursday, April 21, 2011
  176. 176. TEE HEE!!!!!Thursday, April 21, 2011
  177. 177. fixing our tyrantsThursday, April 21, 2011
  178. 178. Facebook + GithubThursday, April 21, 2011
  179. 179. Get InvolvedThursday, April 21, 2011
  180. 180. Only fix the immediate problem Gain TrustThursday, April 21, 2011
  181. 181. Then PersuadeThursday, April 21, 2011
  182. 182. Thursday, April 21, 2011
  183. 183. Thursday, April 21, 2011
  184. 184. Questions?Thursday, April 21, 2011

×