Behind the Curtain
Upcoming SlideShare
Loading in...5
×
 

Behind the Curtain

on

  • 4,750 views

My talk at red dirt ruby conf

My talk at red dirt ruby conf

Statistics

Views

Total Views
4,750
Views on SlideShare
4,749
Embed Views
1

Actions

Likes
1
Downloads
14
Comments
0

1 Embed 1

http://localhost 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Behind the Curtain Behind the Curtain Presentation Transcript

    • Testing This TESTING TESTING TESTING TESTINGThursday, April 21, 2011
    • ZOMG!Thursday, April 21, 2011
    • GOOD MORNING!Thursday, April 21, 2011
    • Aaron PattersonThursday, April 21, 2011
    • @tenderloveThursday, April 21, 2011
    • WWFMD?Thursday, April 21, 2011
    • ITWFMWD?Thursday, April 21, 2011
    • Thursday, April 21, 2011
    • AT&T, AT&T logo and all AT&T related marks are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies.Thursday, April 21, 2011
    • Miami Vice Principal Señor Facebook Integration Engineering ManagerThursday, April 21, 2011
    • ElectThursday, April 21, 2011
    • Our Time Together 12% 2% 24% 61% Inspirational Entertaining Informative WastedThursday, April 21, 2011
    • Thursday, April 21, 2011
    • KEYNOTEThursday, April 21, 2011
    • Our Time Together 10% 20% 20% 50% Inspirational Entertaining Informative WastedThursday, April 21, 2011
    • Thursday, April 21, 2011
    • 20% More TransitionsThursday, April 21, 2011
    • I enjoy dealing with behinds the scenes information Living Behind the Curtain 6 subjects, less than 10 minutes on each of themThursday, April 21, 2011
    • DATABASE APPLICATION CLIENTThursday, April 21, 2011
    • DATABASE APPLICATION CLIENTThursday, April 21, 2011
    • DATABASE APPLICATION CLIENTThursday, April 21, 2011
    • DATABASE APPLICATION CLIENTThursday, April 21, 2011
    • Code as DataThursday, April 21, 2011
    • PoliticsThursday, April 21, 2011
    • We dont vote on commits. It would be too inefficient. "Open Source is a Democracy"Thursday, April 21, 2011
    • OSS is an OligarchyThursday, April 21, 2011
    • Benevolent DictatorThursday, April 21, 2011
    • Small projects Large projects have have few people, too many people so so it is efficient it becomes inefficient to individually consider all feedback Tyrannical LeaderThursday, April 21, 2011
    • Thursday, April 21, 2011
    • Care and Feeding of Your TyrantThursday, April 21, 2011
    • CheetosThursday, April 21, 2011
    • Good reproduction steps Feed them Quality BugsThursday, April 21, 2011
    • Feed them Patches with TestsThursday, April 21, 2011
    • Gemspec, Jewler Hoe, etc leave build systemsThursday, April 21, 2011
    • leave test frameworksThursday, April 21, 2011
    • Use +1s CarefullyThursday, April 21, 2011
    • Thursday, April 21, 2011
    • Thursday, April 21, 2011
    • 320 commentsThursday, April 21, 2011
    • 210 plus oneThursday, April 21, 2011
    • Thursday, April 21, 2011
    • FFFFFFFUUUUU UUUUUUUThursday, April 21, 2011
    • Thursday, April 21, 2011
    • SuperficialThursday, April 21, 2011
    • MeritThursday, April 21, 2011
    • QualityThursday, April 21, 2011
    • Well discuss this in closing. How to fix?Thursday, April 21, 2011
    • Encodings the silent killerThursday, April 21, 2011
    • Data goes in, Data goes out Never a miscommunicationThursday, April 21, 2011
    • Until there isThursday, April 21, 2011
    • `encode: "xE9" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)Thursday, April 21, 2011
    • Encoding Error? You cant explain thatThursday, April 21, 2011
    • String TaggingThursday, April 21, 2011
    • Strings with unknown encoding are marked as binary. "hello! <3".encoding # => #<Encoding:UTF-8>Thursday, April 21, 2011
    • socket.read(4).encoding # => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
    • To understand these bugs, we must find boundaries BoundariesThursday, April 21, 2011
    • App CodeThursday, April 21, 2011
    • Sadness Web Server Belt NoSQL App Code File System DatabaseThursday, April 21, 2011
    • Databases Post contains know the encoding / stored encoding spec declares default #<Encoding:ASCII-8BIT> is usually a bug!Thursday, April 21, 2011
    • # encoding: utf-8 name = user = User.create!(:name => name) user.reload user.name + nameThursday, April 21, 2011
    • # encoding: utf-8 name = user = User.create!(:name => name) user.reload user.name + name incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)Thursday, April 21, 2011
    • >> name = "Aaron Patterson" >> name.encoding => #<Encoding:UTF-8> >> User.create!(:name => name) >> u = User.find :first >> u.name.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
    • >> name = "Aaron Patterson" >> name.encoding => #<Encoding:UTF-8> >> User.create!(:name => name) >> u = User.find :first >> u.name.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
    • Ruby attempts to convert the binary for you name = aaron patterson user = User.create!(:name => name) user.reload user.name + nameThursday, April 21, 2011
    • Ruby attempts to convert the binary for you name = aaron patterson user = User.create!(:name => name) user.reload user.name + name no errorThursday, April 21, 2011
    • ASCII-8BIT + other should raiseThursday, April 21, 2011
    • Aaro Opin ns ion Cor ner! ASCII-8BIT + other should raiseThursday, April 21, 2011
    • Resources Know EncodingThursday, April 21, 2011
    • Prepared StatementsThursday, April 21, 2011
    • SELECT * FROM "users" WHERE id = 1Thursday, April 21, 2011
    • Normal Queries Parse the Query Formulate an Execution Plan Return resultsThursday, April 21, 2011
    • Application DatabaseThursday, April 21, 2011
    • Application Result Set DatabaseThursday, April 21, 2011
    • Application DatabaseThursday, April 21, 2011
    • Application query: 1234 DatabaseThursday, April 21, 2011
    • Application DatabaseThursday, April 21, 2011
    • Application Result Set DatabaseThursday, April 21, 2011
    • Preparation Parse the Query Formulate an Execution PlanThursday, April 21, 2011
    • Execution Return ResultsThursday, April 21, 2011
    • Data transfer decreasesThursday, April 21, 2011
    • Query Planner ImprovesThursday, April 21, 2011
    • SecurityThursday, April 21, 2011
    • Talk about problem inserting rows with session data SQLite3 Issues :( Could insert session data, but could not find itThursday, April 21, 2011
    • INSERT INTO "sessions" ("data", "session_id") VALUES (?, ?) [data, BAh7BjoIZm9vSSIIYmF6BjoGRUY=] [session_id, dae0fb8a9c34e6980c9d0fae33a8fff6]Thursday, April 21, 2011
    • db = SQLite3::Database.new :memory: db.trace { |sql| puts sql }Thursday, April 21, 2011
    • INSERT INTO "sessions" ("data", "session_id") VALUES ( BAh7BjoIZm9vSSIIYmF6BjoGRUY=, x3633303337623066376536613130343132343 765623763626631616439303631)Thursday, April 21, 2011
    • INSERT INTO "sessions" ("data", "session_id") VALUES ( BAh7BjoIZm9vSSIIYmF6BjoGRUY=, x3633303337623066376536613130343132343 765623763626631616439303631)Thursday, April 21, 2011
    • Made a sample program to reproduce the db.trace { |sql| puts sql } problem stmt = db.prepare( INSERT INTO "sessions" (session_id) VALUES (?)) stmt.bind_param 1, fuu stmt.executeThursday, April 21, 2011
    • Resulting Query INSERT INTO "sessions" (session_id) VALUES (fuu)Thursday, April 21, 2011
    • stmt.bind_param 1, fuu.encode(BINARY) stmt.executeThursday, April 21, 2011
    • INSERT INTO "sessions" (session_id) VALUES (x667575)Thursday, April 21, 2011
    • Why did sqlite3 do this? Talk about column affinity INSERT INTO "sessions" (session_id) VALUES (x667575)Thursday, April 21, 2011
    • Why is it stored as Binary?Thursday, April 21, 2011
    • Why is the session id tagged binary?Thursday, April 21, 2011
    • >> x = OpenSSL::Random.random_bytes(10) => "Gx93xFCxB2xCExC0xECxBBxA7W" >> x.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
    • >> y = x.unpack(H*) => ["4793fcb2cec0ecbba757"] >> y.first.encoding => #<Encoding:ASCII-8BIT>Thursday, April 21, 2011
    • def generate_sid ActiveSupport::SecureRandom.hex(16) endThursday, April 21, 2011
    • Should hex have tagged it? Should have generate_sid tagged it? def generate_sid sid = ActiveSupport::SecureRandom.hex(16) if sid.respond_to?(:encode!) sid.encode!(UTF-8) end sid endThursday, April 21, 2011
    • Find the SourceThursday, April 21, 2011
    • ARel and ActiveRecordThursday, April 21, 2011
    • Fear of SQLThursday, April 21, 2011
    • Why?Thursday, April 21, 2011
    • But not specifically SQL Working with sets We must learn SQL Can fetch correct data.Thursday, April 21, 2011
    • We must learn to work with SetsThursday, April 21, 2011
    • Avoid SQL for:Thursday, April 21, 2011
    • code reuseThursday, April 21, 2011
    • securityThursday, April 21, 2011
    • from databases independenceThursday, April 21, 2011
    • What is ARel?Thursday, April 21, 2011
    • Represents SQL as a Tree AST Represents the IDEA of a SQL statementThursday, April 21, 2011
    • SQL CompilerThursday, April 21, 2011
    • Only until we invoke the compiler, does the AST become a query Represents an IDEAThursday, April 21, 2011
    • Relationship to ActiveRecordThursday, April 21, 2011
    • User.where(something)Thursday, April 21, 2011
    • class User scope :heart where(:name => <3) end User.heart.select(name)Thursday, April 21, 2011
    • Our Application ActiveRecord ARel DatabaseThursday, April 21, 2011
    • Our Application .to_a ActiveRecord ARel DatabaseThursday, April 21, 2011
    • Our Application ActiveRecord to_sql ARel DatabaseThursday, April 21, 2011
    • Our Application ActiveRecord SELECT * ... ARel DatabaseThursday, April 21, 2011
    • Our Application ActiveRecord [{: name => <3}] ARel DatabaseThursday, April 21, 2011
    • ActiveRecord::RelationThursday, April 21, 2011
    • shortens to...Thursday, April 21, 2011
    • ARelThursday, April 21, 2011
    • ARel != ARelThursday, April 21, 2011
    • :(Thursday, April 21, 2011
    • Future we can have optimizers, compiler cache, etc SQL CompilerThursday, April 21, 2011
    • Interpreter?Thursday, April 21, 2011
    • module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions) private def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end def visit_Arel_Nodes_And o o.children.map { |child| visit child } end def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end def visit_Arel_Attributes_Attribute o return if o.name == *; o.name end def visit_Arel_Nodes_JoinSource o visit o.left end def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end end endThursday, April 21, 2011
    • module Arel module Visitors class Mongo < Arel::Visitors::Visitor attr_reader :db def initialize db; @db = db; end Query = Struct.new(:collection_name, :fields, :conditions) private def visit_Arel_Nodes_SelectStatement o o.cores.map { |c| visit_Arel_Nodes_SelectCore c }.map { |query| collection = db.collection query.collection_name fields = query.fields selector = Hash[query.conditions] opts = {} opts[:fields] = fields unless fields.empty? opts[:limit] = o.limit.expr.to_i if o.limit collection.find(selector, opts).to_a }.flatten end Mongo Interpreter def visit_Arel_Nodes_SelectCore o fields = o.projections.map { |proj| visit(proj) }.compact conditions = o.wheres.map { |condition| visit(condition) }.flatten(1) Query.new(visit(o.source), fields, conditions) end 64 loc def visit_Arel_Nodes_And o o.children.map { |child| visit child } end def visit_Arel_Nodes_Equality o [visit(o.left), visit(o.right)] end def visit_Arel_Attributes_Attribute o return if o.name == *; o.name end def visit_Arel_Nodes_JoinSource o visit o.left end def visit_Arel_Table o o.name end def literal o; o; end alias :visit_String :literal alias :visit_Fixnum :literal end end endThursday, April 21, 2011
    • Thursday, April 21, 2011
    • gist.github.com/845782Thursday, April 21, 2011
    • Custom DSLThursday, April 21, 2011
    • Responds to * class Select < Struct.new(:columns) def self.* other other.select = new(Arel.sql(*)) other end endThursday, April 21, 2011
    • Responds to WHERE class From < Struct.new(:table, :conditions) def WHERE conditions self.conditions = conditions Where.new(self) end endThursday, April 21, 2011
    • Responds to to_s class Where < Struct.new(:from, :select) def to_s Arel::Table.engine = Arel::Sql::Engine.new( FakeRecord::Base.new) table = Arel::Table.new from.table table.project(select.columns).where( from.conditions.map { |k,v| table[k].eq v }).to_sql end endThursday, April 21, 2011
    • Bootstrap Methods SELECT = Select def FROM table From.new table endThursday, April 21, 2011
    • x = SELECT * FROM("users") .WHERE(:id => 10) puts xThursday, April 21, 2011
    • Thursday, April 21, 2011
    • Write SQL Ruby to avoid SQLThursday, April 21, 2011
    • Talked about data from the back end, talk a bit about data to the front end StreamingThursday, April 21, 2011
    • Twitter Streaming APIThursday, April 21, 2011
    • Needed for persistent connections Chunked ResponsesThursday, April 21, 2011
    • Normal Timeline Buffers data, then sends a responseThursday, April 21, 2011
    • Normal Timeline Buffers data, then sends a response Processing ERbThursday, April 21, 2011
    • Normal Timeline Buffers data, then sends a response Processing ERb Download Asset Download AssetThursday, April 21, 2011
    • Chunked Timeline Send data as soon as it is availableThursday, April 21, 2011
    • Chunked Timeline Send data as soon as it is available Processing ERbThursday, April 21, 2011
    • Chunked Timeline Send data as soon as it is available Processing ERb Download Asset Download AssetThursday, April 21, 2011
    • ImplementationThursday, April 21, 2011
    • Rack APIThursday, April 21, 2011
    • status, headers, body = app.call(env)Thursday, April 21, 2011
    • status.to_iThursday, April 21, 2011
    • headers.is_a?(Hash)Thursday, April 21, 2011
    • body.responds_to?(:each)Thursday, April 21, 2011
    • Inside Rack Calls each, possibly calls close body.each do |chunk| output(chunk) end body.close rescue nilThursday, April 21, 2011
    • Sample Application class MyApp def call(env) # some computation body = hello # more computation body << world [200, { X-Hello => World }, [body]] end endThursday, April 21, 2011
    • Delay work until each class FooBody def each yield "hello " sleep(10) # simulate work yield "world!" end endThursday, April 21, 2011
    • Database connections were lost Problems ☹Thursday, April 21, 2011
    • Database connections are managed in middleware MiddlewareThursday, April 21, 2011
    • "There are two hard problems in CS: cache invalidation and naming things" -- Phil KarltonThursday, April 21, 2011
    • class DbCache < Struct.new(:app) def call(env) # init cache status, headers, body = app.call(env) # clear cache [status, headers, body] end endThursday, April 21, 2011
    • class BodyProxy < Struct.new(:delegate) def each delegate.each { |x| yield x } end def close delegate.close rescue nil # clear cache end endThursday, April 21, 2011
    • Update Middleware, Database Works! ☺Thursday, April 21, 2011
    • Same problem in query cache as the database handle cache. But Query Cache was broken. ☹Thursday, April 21, 2011
    • Middleware is coupled to the callstackThursday, April 21, 2011
    • ~ 24 middlewareThursday, April 21, 2011
    • ~ 5 required a proxyThursday, April 21, 2011
    • Types of Middleware Generators Rack jams these to one API. Filters Approach seems naive when examining usage Lifecycle Handlers Tied to callstackThursday, April 21, 2011
    • Solutions!Thursday, April 21, 2011
    • Embrace The Differences!Thursday, April 21, 2011
    • Lifecycle Listeners class Listener def created(event) end def destroyed(event) end endThursday, April 21, 2011
    • Generators Synchronous or Asynchronous class Resource def service(request, response) 10.times { |i| response.body.write "hello #{i}" } response.body.close Return value is end ignored endThursday, April 21, 2011
    • Filters class Filter def filter(request, response, chain) chain.filter(request, response) end endThursday, April 21, 2011
    • Synchronous Asynchronous Freedom from the Callstack Thread SafetyThursday, April 21, 2011
    • I stole this API.Thursday, April 21, 2011
    • TEE HEE!!!!!Thursday, April 21, 2011
    • fixing our tyrantsThursday, April 21, 2011
    • Facebook + GithubThursday, April 21, 2011
    • Get InvolvedThursday, April 21, 2011
    • Only fix the immediate problem Gain TrustThursday, April 21, 2011
    • Then PersuadeThursday, April 21, 2011
    • Thursday, April 21, 2011
    • Thursday, April 21, 2011
    • Questions?Thursday, April 21, 2011