Advanced CouchDB phpday.it

2,106
-1

Published on

A talk I did at phpday.it in Verona, Italy. Using some more advanced CouchDB functionality, like filters, changes feed, shows, geolocation (GeoCouch), and other interesting stuff. It also covers some real use cases.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,106
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
42
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Advanced CouchDB phpday.it

  1. 1. CouchDB relax
  2. 2. CouchDB relax Sander van de Graaf @svdgraaf
  3. 3. http://joind.in/talk/view/3009
  4. 4. RELAX
  5. 5. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff (GeoCouch!)
  6. 6. CouchDB relax
  7. 7. NOSQL
  8. 8. IT’S A MOVEMENT
  9. 9. 1998
  10. 10. Carlo Strozzi
  11. 11. NOSQL == Not Only SQL
  12. 12. “[The NoSQL movement] departs from the relationalmodel altogether, it should therefore have been calledmore appropriately ‘NoREL’, or something to thateffect.” - Carlo Strozzi
  13. 13. CouchDB relax
  14. 14. NUTSHELL
  15. 15. SPEED
  16. 16. APPEND ONLY
  17. 17. NO REPAIR NEEDED
  18. 18. COMPACTING
  19. 19. HTTP SERVER
  20. 20. CAP
  21. 21. CouchDB CAP
  22. 22. CouchDBEVENTUALLY CONSISTENT
  23. 23. FULL REST API
  24. 24. HTTP METHODS• GET • SELECT• PUT • UPDATE• POST • INSERT• DELETE • DELETE• COPY • ...
  25. 25. { "total_rows": 2, "offset": 0, "rows": [ { "id": "abc089b91fadfcf67a438e73f426f71d", "key": [ "2011-05-07 13:52:15", "abc089b91fadfcf67a438e73f426f71d" ], "value": { "_id": "abc089b91fadfcf67a438e73f426f71d", "_rev": "2-a38e08996586cd774b0b5c3006659de0", "message": "A2 HUISARTSENPOST BARNEVELD NIJKERKERWEG 119 3771LA 119" } }, { "id": "abc089b91fadfcf67a438e73f4286fd5", "key": [ "2011-05-07 14:00:16", "abc089b91fadfcf67a438e73f4286fd5" ], "value": { "_id": "abc089b91fadfcf67a438e73f4286fd5", "_rev": "2-691929b5f3d6cde58f27d62681c7b599", "message": "A1 LEEUWERIKSWEIDE 183 WAGENINGEN 6708LH 183" } } ]}
  26. 26. REPLICATION
  27. 27. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  28. 28. RELAX
  29. 29. PHP USAGE
  30. 30. PHP LIBRARIES• PHPillow (LGPL)• PHP Object Freezer (BSD)• PHP On Couch (GPL 2 / 3)• PHP CouchDB Extension (PHP license)• SAG for CouchDB (apache)• Doctrine 2 CouchDB ODM
  31. 31. PHP ON COUCH<?PHP// setup connection for couchdb$client = new Couchdb_Client(http://ponies.couchone.com:5984,rainbows);// fetch a document$doc = $client->getDoc(awesome_pony);// updating document$doc->type = "awesome";try { $client->storeDoc($doc);}catch (Exception $e) { echo "Document storage failed : " . $e->getMessage();}
  32. 32. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  33. 33. REPLICATION
  34. 34. DEFINITION“Replication is the process of sharing information soas to ensure consistency between redundantresources, such as software or hardware components,to improve reliability, fault-tolerance, or accessibility.” Source: wikipedia
  35. 35. CouchDB relax
  36. 36. CouchDB relax CouchDB relax
  37. 37. CouchDB relaxCouchDB relax CouchDB relax CouchDB relax
  38. 38. CouchDB relax CouchDB relax
  39. 39. CouchDB relaxCouchDB relax CouchDB relax
  40. 40. US NL CouchDB relaxCouchDB relax BE CouchDB relax
  41. 41. USE CASE IDG Publishers
  42. 42. IDG US IDG AUSTRALIA CouchDB relaxCouchDB relax CouchDB relax CouchDB relax CouchDB relax CouchDB relax CouchDB relax IDG NL IDG DE
  43. 43. IDG DE IDG Italy IDG Poland IDG HungaryIDG NL IDG US IDG Romania IDG SwedenIDG Brazil IDG UK IDG Spania IDG PortugalIDG Ghana IDG Japan IDG Thailand IDG Vietnam
  44. 44. IMPLICATION Every single bit of IDG content publishedworldwide, is available anywhere, anytime in msec.
  45. 45. P2P WEB
  46. 46. “World Domination”
  47. 47. HOW DO I DO THAT?
  48. 48. CLUSTERING “The fun stuff ”
  49. 49. loadbalancer ...nCouchDB relax CouchDB relax
  50. 50. CHALLENGES• Large amounts of data• Large views (with big/long map/reduce queries)• LOTS of traffic• Location based partitions• For fun and profit
  51. 51. MAP/REDUCE
  52. 52. INPUT IP Bytes212.122.174.13 18271212.122.174.13 191726212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371212.122.174.13 43
  53. 53. MAPPER => REDUCER IP Bytes 18271 191726212.122.174.13 198 43 91272 74.119.8.111 8371
  54. 54. AFTER REDUCE IP Bytes212.122.174.13 210238 74.119.8.111 99643
  55. 55. PARTITION INPUTPartition IP Bytes 0 212.122.174.13 18271 0 212.122.174.13 191726 0 212.122.174.13 198 1 74.119.8.111 91272 1 74.119.8.111 8371 0 212.122.174.13 43
  56. 56. MAPPER => REDUCERPartition IP Bytes 18271 191726 0 212.122.174.13 198 43 91272 1 74.119.8.111 8371
  57. 57. AFTER REDUCE IP Bytes212.122.174.13 210238 74.119.8.111 99643
  58. 58. CLUSTERING OPTIONS• CouchDB Lounge• Pillow• BigCouch
  59. 59. LOUNGE• partitioning/clustering• Nginx module• meebo.com• ‘easy’• http://tilgovi.github.com/couchdb-lounge/
  60. 60. LOUNGE• dumb_proxy => proxy for simple PUT/GET’s• smart_proxy => proxy for map/reduce over shards• replicator => updates all copies, redundantly
  61. 61. nginx dumb_proxy ...nCouchDB relax CouchDB relax
  62. 62. nginx smart_proxy ...nCouchDB relax CouchDB relax
  63. 63. Bonus:other nginx modules work too
  64. 64. PILLOW• Erlang based• router/rereducer (map/reduce over multiple systems)• In development (but promising!)• https://github.com/khellan/Pillow
  65. 65. BIGCOUCH• Fork• 100% api compatible• Open Source/Commercial• https://cloudant.com/#!/solutions/bigcouch
  66. 66. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  67. 67. BACKEND USAGE
  68. 68. PROXIED CouchDB relax
  69. 69. DIRECT CouchDB relax
  70. 70. NOSQL && SQL HYBRID• onSave, onCommit hooks available in every major framework• onSave -> make a JSON representation of your object, and PUT it to couchdb (#protip: only ‘public’ data)• sql db is leading, you don’t care about versioning in couchdb• youcan use your data directly from couchdb within your frontend javascript
  71. 71. MODEL<?phpclass Pony extends Application_models{ public function toArray() { $data = $this->_getData(); // we dont want any private data pushed unset($data[created_on]); unset($data[created_by]); unset($data[access_level]); unset($data[private_data]); $data[tags] = $this->getTags(); $data[categories] = $this->getCategories(); $data[rainbows] = double; return $data; }}
  72. 72. AFTER_SAVE<?phpclass article_module extends admin_module{ public function after_save() { parent::after_save(); $data = $this->toJson(); // store the document in couchdb $res = CouchDB::put($data); // set the local id and revision, so we can // use it the next time $this->_id = $res->_id; $this->_rev = $res->_rev; }}
  73. 73. PROXYRewriteEngine OnRewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]
  74. 74. JSONP?callback=foobar foobar(...)
  75. 75. JAVASCRIPT<script type="text/javascript">$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",function(res){ for(i in res.rows) { doc = res.rows[i].doc; // do stuff }});</script>
  76. 76. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  77. 77. COUCHAPP
  78. 78. “Distributed, scalable, web applications you say?omgwtfbbq!?!1!!!one1!1!eleven”
  79. 79. _attachments
  80. 80. CouchDB relaxCouchDB relax CouchDB relax
  81. 81. INSTALLATION Couchapp 0.7.5
  82. 82. $ couchapp init
  83. 83. LAYOUT
  84. 84. $ couchapp push http://ponies.couchone.com:5984/rainbows
  85. 85. get your own copy:$ couchapp clone http://svdgraaf.couchone.com/couchappdemo/_design/↩ phpday phpday
  86. 86. https://github.com/brandon-beacher/couchapp-tmbundle
  87. 87. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  88. 88. OTHER STUFF
  89. 89. REWRITES
  90. 90. _REWRITE
  91. 91. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _view/best-ponies?descending=true&limit=5&key=”foobar”"
  92. 92. STEP 1{ .... "rewrites": [{ "from": "/best-5-ponies", "to": "ponies/_view/best-ponies", "method": "GET", "query": { "descending": true, "limit": 5, "key": "foobar" } }]}
  93. 93. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _view/best-ponies?descending=true&limit=5&key=”foobar”"
  94. 94. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _rewrite/best-5-ponies"
  95. 95. STEP 2[vhosts]awesomeponies.com = /rainbows/_design/ponies/_rewrite
  96. 96. STEP 2$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
  97. 97. STEP 2$ curl "http://awesomeponies.com/best-5-ponies"
  98. 98. _CHANGES
  99. 99. RELAX
  100. 100. $ curl -X GET "http://ponies.couchone.com/rainbows/_changes"
  101. 101. {"results":[], "last_seq":0}
  102. 102. curl -X PUT http://ponies.couchone.come/rainbows/foobar -d {"type":"awesome"}
  103. 103. {"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }], "last_seq":1}
  104. 104. {"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }, { "seq":2, "id":"foobar2", "changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}] }], "last_seq":2}
  105. 105. _CHANGES OPTIONS• ?since• Longpolling• Continuous
  106. 106. SINCE$ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"
  107. 107. LONGPOLLINGcurl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"
  108. 108. CONTINOUScurl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"
  109. 109. FILTERS
  110. 110. FILTERS/IMPORTANT.JS function(doc, req) { if(doc.priority == high) { return true; } return false; }
  111. 111. $ curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important
  112. 112. FILTERS/CITY.JSfunction(doc, req) { if(doc.city.name == req.query.name) { return true; } return false;}
  113. 113. curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=verona
  114. 114. SHOWS
  115. 115. function(doc, req) { return { body: "Hello World" }}
  116. 116. curl -X"http://db.example.com/db/_design/foobar/_show/showfunction/docid"
  117. 117. function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
  118. 118. USE CASE CDN
  119. 119. From public hash to private file /m/m1gz4l4a1clb.jpga2d85de382c8672925174e731a1d2a89
  120. 120. storage (http based) 5 Varnish proprietary middleware1 2 webserver 3 4 mysql
  121. 121. storage (http based) 4 Varnish nginx + mod_cache1 2 3 middleware
  122. 122. function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
  123. 123. { "_id": "m1gz4l4a1clb", "_rev": "2-d7855bba9445f446c0b71f044e796c28", "content-type": "image/jpeg", "target": "http://10.16.10.1/a2d85de382c8672925174e731a1d2a89",}
  124. 124. LUCENE
  125. 125. [external]fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py[httpd_db_handlers]_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
  126. 126. FULLTEXT/BY_MESSAGE/BY_QUERY.JS function(doc) { var ret=new Document(); ret.add(doc.message); ret.add(new Date(doc.datetime)); return ret; }
  127. 127. curl -X GET"http://db.example.com/db/_fti/_design/app/by-query?q=foobar"
  128. 128. GEOCOUCHhttps://github.com/vmx/couchdb
  129. 129. GEOCOUCH• Supports bbox, polygon, lines, etc.• fork (for now)• outputs via lists, georss possible• directly useable by google maps• can read GIS data• combined with _changes makes interesting use cases possible
  130. 130. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  131. 131. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  132. 132. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  133. 133. Worldwide searchhttp://db.example.com/couchappdemo/_design/phpday/_spatial/points?bbox=0,0,180,90 {"update_seq":3,"rows":[ { "id":"Verona", "bbox":[10.991748,45.438367,10.991748,45.438367], "value":[ "Verona", [10.991748,45.438367] ] }] }
  134. 134. bbox=west,south,east,north
  135. 135. COUCHDB FOR IOShttps://github.com/couchbaselabs/iOS-Couchbase
  136. 136. http://www.iriscouch.com/
  137. 137. http://www.couchbase.com/downloads
  138. 138. Q?
  139. 139. http://joind.in/talk/view/3009
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×