Advanced CouchDB phpday.it
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Advanced CouchDB phpday.it

on

  • 2,332 views

A talk I did at phpday.it in Verona, Italy. Using some more advanced CouchDB functionality, like filters, changes feed, shows, geolocation (GeoCouch), and other interesting stuff. It also covers some ...

A talk I did at phpday.it in Verona, Italy. Using some more advanced CouchDB functionality, like filters, changes feed, shows, geolocation (GeoCouch), and other interesting stuff. It also covers some real use cases.

Statistics

Views

Total Views
2,332
Views on SlideShare
2,329
Embed Views
3

Actions

Likes
4
Downloads
40
Comments
0

3 Embeds 3

http://twitter.com 1
http://10.150.200.76 1
https://duckduckgo.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Advanced CouchDB phpday.it Presentation Transcript

  • 1. CouchDB relax
  • 2. CouchDB relax Sander van de Graaf @svdgraaf
  • 3. http://joind.in/talk/view/3009
  • 4. RELAX
  • 5. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff (GeoCouch!)
  • 6. CouchDB relax
  • 7. NOSQL
  • 8. IT’S A MOVEMENT
  • 9. 1998
  • 10. Carlo Strozzi
  • 11. NOSQL == Not Only SQL
  • 12. “[The NoSQL movement] departs from the relationalmodel altogether, it should therefore have been calledmore appropriately ‘NoREL’, or something to thateffect.” - Carlo Strozzi
  • 13. CouchDB relax
  • 14. NUTSHELL
  • 15. SPEED
  • 16. APPEND ONLY
  • 17. NO REPAIR NEEDED
  • 18. COMPACTING
  • 19. HTTP SERVER
  • 20. CAP
  • 21. CouchDB CAP
  • 22. CouchDBEVENTUALLY CONSISTENT
  • 23. FULL REST API
  • 24. HTTP METHODS• GET • SELECT• PUT • UPDATE• POST • INSERT• DELETE • DELETE• COPY • ...
  • 25. { "total_rows": 2, "offset": 0, "rows": [ { "id": "abc089b91fadfcf67a438e73f426f71d", "key": [ "2011-05-07 13:52:15", "abc089b91fadfcf67a438e73f426f71d" ], "value": { "_id": "abc089b91fadfcf67a438e73f426f71d", "_rev": "2-a38e08996586cd774b0b5c3006659de0", "message": "A2 HUISARTSENPOST BARNEVELD NIJKERKERWEG 119 3771LA 119" } }, { "id": "abc089b91fadfcf67a438e73f4286fd5", "key": [ "2011-05-07 14:00:16", "abc089b91fadfcf67a438e73f4286fd5" ], "value": { "_id": "abc089b91fadfcf67a438e73f4286fd5", "_rev": "2-691929b5f3d6cde58f27d62681c7b599", "message": "A1 LEEUWERIKSWEIDE 183 WAGENINGEN 6708LH 183" } } ]}
  • 26. REPLICATION
  • 27. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  • 28. RELAX
  • 29. PHP USAGE
  • 30. PHP LIBRARIES• PHPillow (LGPL)• PHP Object Freezer (BSD)• PHP On Couch (GPL 2 / 3)• PHP CouchDB Extension (PHP license)• SAG for CouchDB (apache)• Doctrine 2 CouchDB ODM
  • 31. PHP ON COUCH<?PHP// setup connection for couchdb$client = new Couchdb_Client(http://ponies.couchone.com:5984,rainbows);// fetch a document$doc = $client->getDoc(awesome_pony);// updating document$doc->type = "awesome";try { $client->storeDoc($doc);}catch (Exception $e) { echo "Document storage failed : " . $e->getMessage();}
  • 32. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  • 33. REPLICATION
  • 34. DEFINITION“Replication is the process of sharing information soas to ensure consistency between redundantresources, such as software or hardware components,to improve reliability, fault-tolerance, or accessibility.” Source: wikipedia
  • 35. CouchDB relax
  • 36. CouchDB relax CouchDB relax
  • 37. CouchDB relaxCouchDB relax CouchDB relax CouchDB relax
  • 38. CouchDB relax CouchDB relax
  • 39. CouchDB relaxCouchDB relax CouchDB relax
  • 40. US NL CouchDB relaxCouchDB relax BE CouchDB relax
  • 41. USE CASE IDG Publishers
  • 42. IDG US IDG AUSTRALIA CouchDB relaxCouchDB relax CouchDB relax CouchDB relax CouchDB relax CouchDB relax CouchDB relax IDG NL IDG DE
  • 43. IDG DE IDG Italy IDG Poland IDG HungaryIDG NL IDG US IDG Romania IDG SwedenIDG Brazil IDG UK IDG Spania IDG PortugalIDG Ghana IDG Japan IDG Thailand IDG Vietnam
  • 44. IMPLICATION Every single bit of IDG content publishedworldwide, is available anywhere, anytime in msec.
  • 45. P2P WEB
  • 46. “World Domination”
  • 47. HOW DO I DO THAT?
  • 48. CLUSTERING “The fun stuff ”
  • 49. loadbalancer ...nCouchDB relax CouchDB relax
  • 50. CHALLENGES• Large amounts of data• Large views (with big/long map/reduce queries)• LOTS of traffic• Location based partitions• For fun and profit
  • 51. MAP/REDUCE
  • 52. INPUT IP Bytes212.122.174.13 18271212.122.174.13 191726212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371212.122.174.13 43
  • 53. MAPPER => REDUCER IP Bytes 18271 191726212.122.174.13 198 43 91272 74.119.8.111 8371
  • 54. AFTER REDUCE IP Bytes212.122.174.13 210238 74.119.8.111 99643
  • 55. PARTITION INPUTPartition IP Bytes 0 212.122.174.13 18271 0 212.122.174.13 191726 0 212.122.174.13 198 1 74.119.8.111 91272 1 74.119.8.111 8371 0 212.122.174.13 43
  • 56. MAPPER => REDUCERPartition IP Bytes 18271 191726 0 212.122.174.13 198 43 91272 1 74.119.8.111 8371
  • 57. AFTER REDUCE IP Bytes212.122.174.13 210238 74.119.8.111 99643
  • 58. CLUSTERING OPTIONS• CouchDB Lounge• Pillow• BigCouch
  • 59. LOUNGE• partitioning/clustering• Nginx module• meebo.com• ‘easy’• http://tilgovi.github.com/couchdb-lounge/
  • 60. LOUNGE• dumb_proxy => proxy for simple PUT/GET’s• smart_proxy => proxy for map/reduce over shards• replicator => updates all copies, redundantly
  • 61. nginx dumb_proxy ...nCouchDB relax CouchDB relax
  • 62. nginx smart_proxy ...nCouchDB relax CouchDB relax
  • 63. Bonus:other nginx modules work too
  • 64. PILLOW• Erlang based• router/rereducer (map/reduce over multiple systems)• In development (but promising!)• https://github.com/khellan/Pillow
  • 65. BIGCOUCH• Fork• 100% api compatible• Open Source/Commercial• https://cloudant.com/#!/solutions/bigcouch
  • 66. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  • 67. BACKEND USAGE
  • 68. PROXIED CouchDB relax
  • 69. DIRECT CouchDB relax
  • 70. NOSQL && SQL HYBRID• onSave, onCommit hooks available in every major framework• onSave -> make a JSON representation of your object, and PUT it to couchdb (#protip: only ‘public’ data)• sql db is leading, you don’t care about versioning in couchdb• youcan use your data directly from couchdb within your frontend javascript
  • 71. MODEL<?phpclass Pony extends Application_models{ public function toArray() { $data = $this->_getData(); // we dont want any private data pushed unset($data[created_on]); unset($data[created_by]); unset($data[access_level]); unset($data[private_data]); $data[tags] = $this->getTags(); $data[categories] = $this->getCategories(); $data[rainbows] = double; return $data; }}
  • 72. AFTER_SAVE<?phpclass article_module extends admin_module{ public function after_save() { parent::after_save(); $data = $this->toJson(); // store the document in couchdb $res = CouchDB::put($data); // set the local id and revision, so we can // use it the next time $this->_id = $res->_id; $this->_rev = $res->_rev; }}
  • 73. PROXYRewriteEngine OnRewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]
  • 74. JSONP?callback=foobar foobar(...)
  • 75. JAVASCRIPT<script type="text/javascript">$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",function(res){ for(i in res.rows) { doc = res.rows[i].doc; // do stuff }});</script>
  • 76. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  • 77. COUCHAPP
  • 78. “Distributed, scalable, web applications you say?omgwtfbbq!?!1!!!one1!1!eleven”
  • 79. _attachments
  • 80. CouchDB relaxCouchDB relax CouchDB relax
  • 81. INSTALLATION Couchapp 0.7.5
  • 82. $ couchapp init
  • 83. LAYOUT
  • 84. $ couchapp push http://ponies.couchone.com:5984/rainbows
  • 85. get your own copy:$ couchapp clone http://svdgraaf.couchone.com/couchappdemo/_design/↩ phpday phpday
  • 86. https://github.com/brandon-beacher/couchapp-tmbundle
  • 87. CONTENTS• Introduction• PHP Usage• Replication/Scalability• Backend usage• Couchapps• Other stuff
  • 88. OTHER STUFF
  • 89. REWRITES
  • 90. _REWRITE
  • 91. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _view/best-ponies?descending=true&limit=5&key=”foobar”"
  • 92. STEP 1{ .... "rewrites": [{ "from": "/best-5-ponies", "to": "ponies/_view/best-ponies", "method": "GET", "query": { "descending": true, "limit": 5, "key": "foobar" } }]}
  • 93. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _view/best-ponies?descending=true&limit=5&key=”foobar”"
  • 94. STEP 1$ curl "http://ponies.couchone.com/rainbows/_design/ponies/ _rewrite/best-5-ponies"
  • 95. STEP 2[vhosts]awesomeponies.com = /rainbows/_design/ponies/_rewrite
  • 96. STEP 2$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
  • 97. STEP 2$ curl "http://awesomeponies.com/best-5-ponies"
  • 98. _CHANGES
  • 99. RELAX
  • 100. $ curl -X GET "http://ponies.couchone.com/rainbows/_changes"
  • 101. {"results":[], "last_seq":0}
  • 102. curl -X PUT http://ponies.couchone.come/rainbows/foobar -d {"type":"awesome"}
  • 103. {"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }], "last_seq":1}
  • 104. {"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }, { "seq":2, "id":"foobar2", "changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}] }], "last_seq":2}
  • 105. _CHANGES OPTIONS• ?since• Longpolling• Continuous
  • 106. SINCE$ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"
  • 107. LONGPOLLINGcurl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"
  • 108. CONTINOUScurl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"
  • 109. FILTERS
  • 110. FILTERS/IMPORTANT.JS function(doc, req) { if(doc.priority == high) { return true; } return false; }
  • 111. $ curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important
  • 112. FILTERS/CITY.JSfunction(doc, req) { if(doc.city.name == req.query.name) { return true; } return false;}
  • 113. curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=verona
  • 114. SHOWS
  • 115. function(doc, req) { return { body: "Hello World" }}
  • 116. curl -X"http://db.example.com/db/_design/foobar/_show/showfunction/docid"
  • 117. function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
  • 118. USE CASE CDN
  • 119. From public hash to private file /m/m1gz4l4a1clb.jpga2d85de382c8672925174e731a1d2a89
  • 120. storage (http based) 5 Varnish proprietary middleware1 2 webserver 3 4 mysql
  • 121. storage (http based) 4 Varnish nginx + mod_cache1 2 3 middleware
  • 122. function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
  • 123. { "_id": "m1gz4l4a1clb", "_rev": "2-d7855bba9445f446c0b71f044e796c28", "content-type": "image/jpeg", "target": "http://10.16.10.1/a2d85de382c8672925174e731a1d2a89",}
  • 124. LUCENE
  • 125. [external]fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py[httpd_db_handlers]_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
  • 126. FULLTEXT/BY_MESSAGE/BY_QUERY.JS function(doc) { var ret=new Document(); ret.add(doc.message); ret.add(new Date(doc.datetime)); return ret; }
  • 127. curl -X GET"http://db.example.com/db/_fti/_design/app/by-query?q=foobar"
  • 128. GEOCOUCHhttps://github.com/vmx/couchdb
  • 129. GEOCOUCH• Supports bbox, polygon, lines, etc.• fork (for now)• outputs via lists, georss possible• directly useable by google maps• can read GIS data• combined with _changes makes interesting use cases possible
  • 130. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  • 131. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  • 132. SPATIAL INDEX in spatial/points.jsfunction(doc) { if (doc.geo && doc.geo.latitude != && doc.geo.longitude != ) { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); }}
  • 133. Worldwide searchhttp://db.example.com/couchappdemo/_design/phpday/_spatial/points?bbox=0,0,180,90 {"update_seq":3,"rows":[ { "id":"Verona", "bbox":[10.991748,45.438367,10.991748,45.438367], "value":[ "Verona", [10.991748,45.438367] ] }] }
  • 134. bbox=west,south,east,north
  • 135. COUCHDB FOR IOShttps://github.com/couchbaselabs/iOS-Couchbase
  • 136. http://www.iriscouch.com/
  • 137. http://www.couchbase.com/downloads
  • 138. Q?
  • 139. http://joind.in/talk/view/3009