You’re Going To Need A
Bigger Toolbox

DIBI 28th April 2010


gareth rushgrove | morethanseven.net   http://www.flickr.com/...
morethanseven.net




Gareth Rushgrove


gareth rushgrove | morethanseven.net
We Used To Just Build Websites


gareth rushgrove | morethanseven.net
Probably Simple Websites


gareth rushgrove | morethanseven.net
Maybe the odd Ecommerce Site


gareth rushgrove | morethanseven.net
Then We Built Web Applications


gareth rushgrove | morethanseven.net
Now We Just Build.


gareth rushgrove | morethanseven.net
Your Toolbox is Growing


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/gordonr/42555739
1
Embrace Heterogeneous Environments


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/mk1971/254849251...
1
No Development Tool Silver Bullet


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/mk1971/2548492513/
We Build Websites With...


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/freefoto/3436970425/
PHP, MySQL, Apache


gareth rushgrove | morethanseven.net
.NET, MSSQL, IIS


gareth rushgrove | morethanseven.net
Java, Oracle, Tomcat


gareth rushgrove | morethanseven.net
Lots of Others


gareth rushgrove | morethanseven.net
Just One Stack?


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/harrygoldenfeld/4263710200/
The Guardian Java, Python, Oracle, App Engine


gareth rushgrove | morethanseven.net
LastFM PHP, C++, Java, Hadoop, Python


gareth rushgrove | morethanseven.net
GitHub Ruby, Erlang, MySQL, Redis, Sinatra


gareth rushgrove | morethanseven.net
Twitter Ruby, Scala, Java, C/C++


gareth rushgrove | morethanseven.net
Facebook PHP, Erlang, C, MySQL, Cassandra


gareth rushgrove | morethanseven.net
2
2. Know When Your Stack is out of its Depth


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/dk_spoo...
What Should You Know About?


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/takomabibelot/220265100/
wiki.nginx.org




Serve Web Pages With Nginx


gareth rushgrove | morethanseven.net
“Apache is like Microsoft Word, it has a
        million options but you only need six. Nginx
        does those six thing...
server {
               listen 80;
               server_name www.example.com;

                    location / {
         ...
http {
               upstream php {
                   server localhost:8002;
               }
               upstream py...
- Thin - http://code.macournoyer.com/thin/
       - Mongrel - http://github.com/fauna/mongrel/
       - Spawning - http://...
memcached.org




Caching with Memcached


gareth rushgrove | morethanseven.net
from django.core.cache import cache

            key = "/about/"
            content = cache.get(key)
            if not c...
- Squid - http://www.squid-cache.org
       - Varnish - http://varnish-cache.org




Also See


gareth rushgrove | moretha...
lucene.apache.org/solr/




Search with Solr


gareth rushgrove | morethanseven.net
http://solr:8983/solr/products/select/
                ?q=colour:red
                &sort=price%20desc
                &r...
- Xapian - http://xapian.org
       - Sphinx - http://sphinxsearch.com
       - Nutch - http://lucene.apache.org/nutch/
  ...
rabbitmq.com




Asynchronous Processing with RabbitMQ


gareth rushgrove | morethanseven.net
require 'carrot'

            q = Carrot.queue('carrot', :durable => true)
            10.times do |num|
              q.p...
puts "count: #{q.message_count}"

            while msg = q.pop(:ack => true)
              puts msg
              q.ack
 ...
- Apache ActiveMQ - http://activemq.apache.org
       - Beanstalk - http://kr.github.com/beanstalkd/
       - Stomp Server...
couchdb.apache.org




Data Storage With CouchDB


gareth rushgrove | morethanseven.net
<?php

                  $options['host'] = "localhost";
                  $options['port'] = 5984;

                  $co...
<?php

                  $options['host'] = "localhost";
                  $options['port'] = 5984;

                  $co...
- MongoDB - http://www.mongodb.org
       - Tokyo Tyrant - http://1978th.net/tokyotyrant/
       - Cassandra - http://cass...
hadoop.apache.org




Data Mining With Hive


gareth rushgrove | morethanseven.net
CREATE TABLE u_data (
              userid INT,
              movieid INT,
              rating INT,
              unixtim...
LOAD DATA LOCAL INPATH 'data.txt' OVERWRITE
            INTO TABLE u_data;

            SELECT COUNT(1) FROM u_data;




H...
add FILE weekday_mapper.py;

            INSERT OVERWRITE TABLE u_data_new
            SELECT
              TRANSFORM (use...
- Hadoop - http://hadoop.apache.org
       - Hive - http://wiki.apache.org/hadoop/Hive/
       - Pig - http://hadoop.apach...
cukes.info




Testing with Cucumber


gareth rushgrove | morethanseven.net
Feature: google.co.uk
                To broaden their knowledge
                A user should be able
                To ...
Feature: google.co.uk
                To broaden their knowledge
                A user should be able
                To ...
puppetlabs.com




Server Provisioning with Puppet


gareth rushgrove | morethanseven.net
class baseclass {
                $packagelist = ["sudo", "openssh-server"]
                package { $packagelist: ensure...
node 'example.com' inherits basenode {
                $packagelist = ["nginx"]
                package { $packagelist: en...
- Chef - http://wiki.opscode.com/display/chef/Home/




Also See


gareth rushgrove | morethanseven.net
Conclusions


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/batega/1596898776/
1
Know What's Possible


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/docman/36125185/
2
Look for Opportunities


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/kgregory/500456103/
3
Experiment


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/seeminglee/3967329241/
4
Manage Complexity


gareth rushgrove | morethanseven.net   http://www.flickr.com/photos/30890318@N06/3510161637/
Questions?


gareth rushgrove | morethanseven.net   http://flickr.com/photos/psd/102332391/
Upcoming SlideShare
Loading in...5
×

You're Going To Need A Bigger Toolbox

2,837

Published on

Talk from Design It Build It conference in Newcastle. Convincing people to look outside their current development tools for inspiration

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,837
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
60
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "You're Going To Need A Bigger Toolbox"

  1. 1. You’re Going To Need A Bigger Toolbox DIBI 28th April 2010 gareth rushgrove | morethanseven.net http://www.flickr.com/photos/booleansplit/2376359338/
  2. 2. morethanseven.net Gareth Rushgrove gareth rushgrove | morethanseven.net
  3. 3. We Used To Just Build Websites gareth rushgrove | morethanseven.net
  4. 4. Probably Simple Websites gareth rushgrove | morethanseven.net
  5. 5. Maybe the odd Ecommerce Site gareth rushgrove | morethanseven.net
  6. 6. Then We Built Web Applications gareth rushgrove | morethanseven.net
  7. 7. Now We Just Build. gareth rushgrove | morethanseven.net
  8. 8. Your Toolbox is Growing gareth rushgrove | morethanseven.net http://www.flickr.com/photos/gordonr/42555739
  9. 9. 1 Embrace Heterogeneous Environments gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/
  10. 10. 1 No Development Tool Silver Bullet gareth rushgrove | morethanseven.net http://www.flickr.com/photos/mk1971/2548492513/
  11. 11. We Build Websites With... gareth rushgrove | morethanseven.net http://www.flickr.com/photos/freefoto/3436970425/
  12. 12. PHP, MySQL, Apache gareth rushgrove | morethanseven.net
  13. 13. .NET, MSSQL, IIS gareth rushgrove | morethanseven.net
  14. 14. Java, Oracle, Tomcat gareth rushgrove | morethanseven.net
  15. 15. Lots of Others gareth rushgrove | morethanseven.net
  16. 16. Just One Stack? gareth rushgrove | morethanseven.net http://www.flickr.com/photos/harrygoldenfeld/4263710200/
  17. 17. The Guardian Java, Python, Oracle, App Engine gareth rushgrove | morethanseven.net
  18. 18. LastFM PHP, C++, Java, Hadoop, Python gareth rushgrove | morethanseven.net
  19. 19. GitHub Ruby, Erlang, MySQL, Redis, Sinatra gareth rushgrove | morethanseven.net
  20. 20. Twitter Ruby, Scala, Java, C/C++ gareth rushgrove | morethanseven.net
  21. 21. Facebook PHP, Erlang, C, MySQL, Cassandra gareth rushgrove | morethanseven.net
  22. 22. 2 2. Know When Your Stack is out of its Depth gareth rushgrove | morethanseven.net http://www.flickr.com/photos/dk_spook/2421009077/
  23. 23. What Should You Know About? gareth rushgrove | morethanseven.net http://www.flickr.com/photos/takomabibelot/220265100/
  24. 24. wiki.nginx.org Serve Web Pages With Nginx gareth rushgrove | morethanseven.net
  25. 25. “Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache. Chris Lea Why Nginx gareth rushgrove | morethanseven.net
  26. 26. server { listen 80; server_name www.example.com; location / { root /var/www/example.com; } } Nginx Example gareth rushgrove | morethanseven.net
  27. 27. http { upstream php { server localhost:8002; } upstream python { server localhost:8003; } } server { server_name www.example.com; location / { proxy_pass http://python; } location ~ /basket/* { proxy_pass http://php; } } Nginx Example gareth rushgrove | morethanseven.net
  28. 28. - Thin - http://code.macournoyer.com/thin/ - Mongrel - http://github.com/fauna/mongrel/ - Spawning - http://pypi.python.org/pypi/Spawning/ - Unicorn - http://github.com/defunkt/unicorn/ Also See gareth rushgrove | morethanseven.net
  29. 29. memcached.org Caching with Memcached gareth rushgrove | morethanseven.net
  30. 30. from django.core.cache import cache key = "/about/" content = cache.get(key) if not content: # expensive query to get content cache.set(key, content, 300) Memcached Example gareth rushgrove | morethanseven.net
  31. 31. - Squid - http://www.squid-cache.org - Varnish - http://varnish-cache.org Also See gareth rushgrove | morethanseven.net
  32. 32. lucene.apache.org/solr/ Search with Solr gareth rushgrove | morethanseven.net
  33. 33. http://solr:8983/solr/products/select/ ?q=colour:red &sort=price%20desc &rows=60 &wt=json Solr Example gareth rushgrove | morethanseven.net
  34. 34. - Xapian - http://xapian.org - Sphinx - http://sphinxsearch.com - Nutch - http://lucene.apache.org/nutch/ - Whoosh - http://whoosh.ca Also See gareth rushgrove | morethanseven.net
  35. 35. rabbitmq.com Asynchronous Processing with RabbitMQ gareth rushgrove | morethanseven.net
  36. 36. require 'carrot' q = Carrot.queue('carrot', :durable => true) 10.times do |num| q.publish(num.to_s) end RabbitMQ Enqueue Example gareth rushgrove | morethanseven.net
  37. 37. puts "count: #{q.message_count}" while msg = q.pop(:ack => true) puts msg q.ack end Carrot.stop RabbitMQ Dequeue Example gareth rushgrove | morethanseven.net
  38. 38. - Apache ActiveMQ - http://activemq.apache.org - Beanstalk - http://kr.github.com/beanstalkd/ - Stomp Server - http://stomp.codehaus.org - MemcacheQ - http://memcachedb.org/memcacheq/ Also See gareth rushgrove | morethanseven.net
  39. 39. couchdb.apache.org Data Storage With CouchDB gareth rushgrove | morethanseven.net
  40. 40. <?php $options['host'] = "localhost"; $options['port'] = 5984; $couch = new CouchSimple($options); $resp = $couch->send("PUT", "/test"); $resp = $couch->send("PUT", "/test/123", '{"_id":"123","data":"Foo"}'); ?> CouchDB Example gareth rushgrove | morethanseven.net
  41. 41. <?php $options['host'] = "localhost"; $options['port'] = 5984; $couch = new CouchSimple($options); $resp = $couch->send("GET", "/test/_all_docs"); $resp = $couch->send("GET", "/test/123"); $resp = $couch->send("DELETE", "/test/"); ?> CouchDB Example gareth rushgrove | morethanseven.net
  42. 42. - MongoDB - http://www.mongodb.org - Tokyo Tyrant - http://1978th.net/tokyotyrant/ - Cassandra - http://cassandra.apache.org - Redis - http://code.google.com/p/redis/ Also See gareth rushgrove | morethanseven.net
  43. 43. hadoop.apache.org Data Mining With Hive gareth rushgrove | morethanseven.net
  44. 44. CREATE TABLE u_data ( userid INT, movieid INT, rating INT, unixtime STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY 't' STORED AS TEXTFILE; Hive Create Example Cucumber DSL Example gareth rushgrove | morethanseven.net
  45. 45. LOAD DATA LOCAL INPATH 'data.txt' OVERWRITE INTO TABLE u_data; SELECT COUNT(1) FROM u_data; Hive Load DSL Example Cucumber Example gareth rushgrove | morethanseven.net
  46. 46. add FILE weekday_mapper.py; INSERT OVERWRITE TABLE u_data_new SELECT TRANSFORM (userid, movieid, rating, unixtime) USING 'python weekday_mapper.py' AS (userid, movieid, rating, weekday) FROM u_data; SELECT weekday, COUNT(1) FROM u_data_new GROUP BY weekday; Hive Map Reduce Example Cucumber DSL Example gareth rushgrove | morethanseven.net
  47. 47. - Hadoop - http://hadoop.apache.org - Hive - http://wiki.apache.org/hadoop/Hive/ - Pig - http://hadoop.apache.org/pig/ - Dumbo - http://lastfm.com/dumbo/ - Disco - http://discoproject.org Also See gareth rushgrove | morethanseven.net
  48. 48. cukes.info Testing with Cucumber gareth rushgrove | morethanseven.net
  49. 49. Feature: google.co.uk To broaden their knowledge A user should be able To search for things Scenario: Searching for things Given I visit "http://www.google.co.uk" When I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org" Cucumber DSL Example gareth rushgrove | morethanseven.net
  50. 50. Feature: google.co.uk To broaden their knowledge A user should be able To search for things Scenario: Searching for things Given I visit "http://www.google.co.uk" When I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org" 1 scenario (1 failed) 4 steps (1 failed, 2 skipped, 1 passed) 0m0.332s Cucumber Results Example gareth rushgrove | morethanseven.net
  51. 51. puppetlabs.com Server Provisioning with Puppet gareth rushgrove | morethanseven.net
  52. 52. class baseclass { $packagelist = ["sudo", "openssh-server"] package { $packagelist: ensure => installed } service { sshd: name => "ssh", enable => true, ensure => running } } Puppet Class Example gareth rushgrove | morethanseven.net
  53. 53. node 'example.com' inherits basenode { $packagelist = ["nginx"] package { $packagelist: ensure => installed } service { "nginx": ensure => running, require => Package["nginx"] } } Puppet Node Example gareth rushgrove | morethanseven.net
  54. 54. - Chef - http://wiki.opscode.com/display/chef/Home/ Also See gareth rushgrove | morethanseven.net
  55. 55. Conclusions gareth rushgrove | morethanseven.net http://www.flickr.com/photos/batega/1596898776/
  56. 56. 1 Know What's Possible gareth rushgrove | morethanseven.net http://www.flickr.com/photos/docman/36125185/
  57. 57. 2 Look for Opportunities gareth rushgrove | morethanseven.net http://www.flickr.com/photos/kgregory/500456103/
  58. 58. 3 Experiment gareth rushgrove | morethanseven.net http://www.flickr.com/photos/seeminglee/3967329241/
  59. 59. 4 Manage Complexity gareth rushgrove | morethanseven.net http://www.flickr.com/photos/30890318@N06/3510161637/
  60. 60. Questions? gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×