Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DBSlayer




           Introducing DBSlayer
http://code.nytimes.com/projects/dbslayer
               Derek Gottfrid
     ...
Problem:




Provide Web Apps Access to the Database
Alternate Problem:




Avoid evil horrible doctor show.
Solution




JSON + HTTP + MYSQL
Simple Example




 select ci.* from City ci, Country c where
 c.name = 'United States' and c.code =
ci.CountryCode order ...
HTTP Request


http://localhost:9090/db?%7B%22SQL%22:
%22select%20ci.*%20from%20City%20ci,%
20Country%20c%20where%20c.name...
HTTP Response


HTTP/1.0 200 OK
Date: Wed, 27 Jun 2007 20:55:59 GMT
Server: dbslayer/server beta-9
Connection: Close
Conte...
JSON Payload

{quot;RESULTquot; : {quot;TYPESquot; : [quot;MYSQL_TYPE_LONGquot; , quot;MYSQL_TYPE_STRINGquot; ,
quot;MYSQL...
Python Sample Client



import urllib,urllib2,cjson
db = lambda q: cjson.decode( urllib2.urlopen(
“http://localhost:9090/d...
Other Solutions
    MySQLProxy
●



    SQLRelay
●



    mysqli
●
Requirements
    Independent
●



    Scalable
●



    Flexible
●



    Simple
●
Disclaimer




Works for us.
Problems Not Solved
    Back end scripts –
●


    bulk import/export
    Large Data Retrievals
●


    – cursors
    Nati...
DBSlayer is the middle-man
    Connection Pooling
●



    Multi-threaded
●



    Round robin distribution of queries
●

...
Design Considerations
Flexiblity
    Loose coupling of front and middle
●


        easy upgrade
    –

        abstract location of db – makes ...
Scalability
    Round robin across slaves
●



    Automatic fail over for slaves
●



    Easily add more slaves
●



   ...
Simplicity
    HTTP and JSON - everyone knows them
●



    Transparent protocol – capture, interpret
●



    Lots of too...
Browser Testing
Stats
Pretty Pictures
View Source
    Built on top of MySQL C API and Apache
●


    Runtime Libraries (APR)
    Why APR?
●


        Platform a...
3 Parts
    server – handles network, HTTP parsing,
●


    threading, logging
    db – handle mysql access, type mapping,...
Future / Your Chance
    improve HTTP               caching layer
●                          ●


    complaince           ...
Other
(tighten up them ....)
Another amazing talk...



   Rails Under The Knife
        Jacob Harris
    July 26th – 2:35pm
        Portland 255
BoF
    Amazon S3/EC2 computing against large
●


    datasets – 7:30 F150
    Future Newspapers – 8:30 E143-144
●
NYTIMES.COM
    open.nytimes.com – developers blog
●



    code.nytimes.com – code repository
●
Free Stuff




Visit the NYTimes Booth
The End




Questions: derek@nytimes.com
Os Gottfrid
Os Gottfrid
Upcoming SlideShare
Loading in …5
×

Os Gottfrid

547 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Os Gottfrid

  1. 1. DBSlayer Introducing DBSlayer http://code.nytimes.com/projects/dbslayer Derek Gottfrid The New York Times
  2. 2. Problem: Provide Web Apps Access to the Database
  3. 3. Alternate Problem: Avoid evil horrible doctor show.
  4. 4. Solution JSON + HTTP + MYSQL
  5. 5. Simple Example select ci.* from City ci, Country c where c.name = 'United States' and c.code = ci.CountryCode order by Population desc
  6. 6. HTTP Request http://localhost:9090/db?%7B%22SQL%22: %22select%20ci.*%20from%20City%20ci,% 20Country%20c%20where%20c.name%20 =%20'United%20States'%20and%20c.code %20=%20ci.CountryCode%20order%20by% 20Population%20desc%20limit%2010;%22 %7D
  7. 7. HTTP Response HTTP/1.0 200 OK Date: Wed, 27 Jun 2007 20:55:59 GMT Server: dbslayer/server beta-9 Connection: Close Content-type: text/plain; charset=utf-8 Content-Length: 746
  8. 8. JSON Payload {quot;RESULTquot; : {quot;TYPESquot; : [quot;MYSQL_TYPE_LONGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_STRINGquot; , quot;MYSQL_TYPE_LONGquot;] , quot;HEADERquot; : [quot;IDquot; , quot;Namequot; , quot;CountryCodequot; , quot;Districtquot; , quot;Populationquot;] , quot;ROWSquot; : [[3793 , quot;New Yorkquot; , quot;USAquot; , quot;New Yorkquot; , 8008278] , [3794 , quot;Los Angelesquot; , quot;USAquot; , quot;Californiaquot; , 3694820] , [3795 , quot;Chicagoquot; , quot;USAquot; , quot;Illinoisquot; , 2896016] , [3796 , quot;Houstonquot; , quot;USAquot; , quot;Texasquot; , 1953631] , [3797 , quot;Philadelphiaquot; , quot;USAquot; , quot;Pennsylvaniaquot; , 1517550] , [3798 , quot;Phoenixquot; , quot;USAquot; , quot;Arizonaquot; , 1321045] , [3799 , quot;San Diegoquot; , quot;USAquot; , quot;Californiaquot; , 1223400] , [3800 , quot;Dallasquot; , quot;USAquot; , quot;Texasquot; , 1188580] , [3801 , quot;San Antonioquot; , quot;USAquot; , quot;Texasquot; , 1144646] , [3802 , quot;Detroitquot; , quot;USAquot; , quot;Michiganquot; , 951270]]}}
  9. 9. Python Sample Client import urllib,urllib2,cjson db = lambda q: cjson.decode( urllib2.urlopen( “http://localhost:9090/db? ”+urllib.quote('{“SQL ”:”%s”}' % q)).read())
  10. 10. Other Solutions MySQLProxy ● SQLRelay ● mysqli ●
  11. 11. Requirements Independent ● Scalable ● Flexible ● Simple ●
  12. 12. Disclaimer Works for us.
  13. 13. Problems Not Solved Back end scripts – ● bulk import/export Large Data Retrievals ● – cursors Native Binds / ● Prepared Statements Enterprise Buzzword ● Compliance A Sober Lindsay ●
  14. 14. DBSlayer is the middle-man Connection Pooling ● Multi-threaded ● Round robin distribution of queries ● Automatic fail over to alternate database ● Easy configuration ● Statistics / Logging ● Tiny code base ●
  15. 15. Design Considerations
  16. 16. Flexiblity Loose coupling of front and middle ● easy upgrade – abstract location of db – makes re-config a – snap Language independent ● JSON easily shared across languages – Common place to add additional features ●
  17. 17. Scalability Round robin across slaves ● Automatic fail over for slaves ● Easily add more slaves ● HTTP plays nice with load balancers ● JSON is very cachable ● Stateless is required for effective scaling ●
  18. 18. Simplicity HTTP and JSON - everyone knows them ● Transparent protocol – capture, interpret ● Lots of tools to work with HTTP ● Works with your browser ● HTTP server base – easy to deploy new ● endpoints
  19. 19. Browser Testing
  20. 20. Stats
  21. 21. Pretty Pictures
  22. 22. View Source Built on top of MySQL C API and Apache ● Runtime Libraries (APR) Why APR? ● Platform agnostic – Memory Pools – Networking / Threading Abstractions – Proven –
  23. 23. 3 Parts server – handles network, HTTP parsing, ● threading, logging db – handle mysql access, type mapping, ● connection failover util – json de/serializer ●
  24. 24. Future / Your Chance improve HTTP caching layer ● ● complaince partition / shard ● more database support ● backends (APR new language ● dbd) bindings alternate serializer ● more stats / ● XML logging security ● scriptable filters ●
  25. 25. Other (tighten up them ....)
  26. 26. Another amazing talk... Rails Under The Knife Jacob Harris July 26th – 2:35pm Portland 255
  27. 27. BoF Amazon S3/EC2 computing against large ● datasets – 7:30 F150 Future Newspapers – 8:30 E143-144 ●
  28. 28. NYTIMES.COM open.nytimes.com – developers blog ● code.nytimes.com – code repository ●
  29. 29. Free Stuff Visit the NYTimes Booth
  30. 30. The End Questions: derek@nytimes.com

×