MPI, Erlang and the web

4,273 views

Published on

slides from my talk at LCA2010 in Wellington about how to scale web services and why parallel programming and open standards matter

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
4,273
On SlideShare
0
From Embeds
0
Number of Embeds
39
Actions
Shares
0
Downloads
45
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

MPI, Erlang and the web

  1. parallel programming why does it matter to web development
  2. who i am • german expat in NZ • text editor addict since age 14 • CTO ideegeo Group Limited • erlounge Wellington
  3. erlounge Wellington • diverse crowd • more about parallel programming • once a month
  4. todays topics • message passing infrastructure • erlang • open standards
  5. scale web based projects • one server • separate functionality (some server) • load balance (some more server)
  6. loadbalance • web servers is easy • databases is harder • business logic often not possible
  7. the problem • shared state • shared data • waiting for other processes
  8. in general terms
  9. global locking
  10. sounds familiar? looks like a parallel programming issue to me
  11. i tried to solve ... • scalability issues • scale to many servers • asynchronous processes • process queues • callback queues
  12. i wrote • a simple perl based environment • my own message queue • my own messaging format
  13. what i actually did without knowing it ...
  14. i wrote a MPI ok, a quite basic message passing infrastructure but it was one
  15. and to my big surprise
  16. this thing worked better as expected
  17. ... but ...
  18. i found erlang
  19. erlang ... • is a functional language • has a built in MPI • share nothing approach
  20. feels like heaven on earth
  21. issues • not compatible to my existing stuff • feel the pain of custom protocols • rewrite or compat layer or ...
  22. the usual thing happened
  23. ... i left it aside ...
  24. ... meanwhile ...
  25. i started to play with • CouchDB • RabbitMQ • ejabberd
  26. all erlang and open standards based
  27. the more I played the more I wished I could use them
  28. the inevitable happened
  29. iteration two
  30. AMQP
  31. open standard • backed by major players • around for some years • RabbitMQ (erlang based)
  32. JSON
  33. open format • used by many • small footprint • easy to read for humans and computers
  34. erlang
  35. fast and small • compared to perl/php/ruby/... • scales linear to 32 cores • distributing processes is easy
  36. perl
  37. still in the system • for compatibility • for string processing • for stuff where it makes sense
  38. [your lang here]
  39. language independent • JSON is available in most languages • AMQP is available in all major languages • pick the one that makes sense
  40. internet architectures • scaling to many servers • loadbalancing • reducing global locks
  41. multicore programming • scaling to many cores • loadbalancing • reducing global locks
  42. do we solve simmilar problems here?
  43. where to find me • @norbu09 • http://norbu09.org • http://github.com/norbu09 • http://slideshare.net/norbu09
  44. thanks • http://www.flickr.com/photos/dranidis/ • http://www.flickr.com/photos/samuellkim/ • http://www.flickr.com/photos/tbcave/

×