Successfully reported this slideshow.

Ducksboard - A real-time data oriented webservice architecture

11

Share

1 of 66
1 of 66

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Ducksboard - A real-time data oriented webservice architecture

  1. 1. Ducksboard a real-time data oriented webservice architecture aitor@ducksboard.com @aitorciki on twitter
  2. 2. What is Ducksboard?
  3. 3. What is Ducksboard? ● it's a web-based dashboard...
  4. 4. What is Ducksboard? ● it's a web-based dashboard... ● to display SaaS / your own metrics...
  5. 5. What is Ducksboard? ● it's a web-based dashboard... ● to display SaaS / your own metrics... ● with heavy focus on UX...
  6. 6. What is Ducksboard? ● it's a web-based dashboard... ● to display SaaS / your own metrics... ● with heavy focus on UX... ● huge bet on REAL real-time updates...
  7. 7. What is Ducksboard? ● it's a web-based dashboard... ● to display SaaS / your own metrics... ● with heavy focus on UX... ● huge bet on REAL real-time updates... ● and much more in the near future!
  8. 8. Architectural Goals
  9. 9. Architectural Goals ● data grabber / visualizer independence
  10. 10. Architectural Goals ● data grabber / visualizer independence ● real-time updates (push vs pull)
  11. 11. Architectural Goals ● data grabber / visualizer independence ● real-time updates (push vs pull) ● scalability (web scale, bitches!)
  12. 12. Architectural Goals ● data grabber / visualizer independence ● real-time updates (push vs pull) ● scalability (web scale, bitches!) ● eat your own duck food
  13. 13. Architecture Concepts
  14. 14. Architecture Concepts ● Back-end (Twisted)
  15. 15. Architecture Concepts ● Back-end (Twisted) ● Parser: obtains data from external services
  16. 16. Architecture Concepts ● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data
  17. 17. Architecture Concepts ● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data ● API: provides read access to stored data
  18. 18. Architecture Concepts ● Front-end
  19. 19. Architecture Concepts ● Front-end ● web application
  20. 20. Architecture Concepts ● Front-end ● web application ● mobile web application (not yet available)
  21. 21. Architecture Concepts ● Front-end ● web application ● mobile web application (not available) ● mobile / desktop apps?
  22. 22. Common Components
  23. 23. Common Components ● PostgreSQL (the “DB”)
  24. 24. Common Components ● PostgreSQL (the “DB”) ● stores application configuration
  25. 25. Common Components ● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries
  26. 26. Common Components ● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries ● starts / stops components (PL/Python!)
  27. 27. Common Components ● RabbitMQ (the “queue”)
  28. 28. Common Components ● RabbitMQ (the “queue”) ● handles RT updates
  29. 29. Common Components ● RabbitMQ (the “queue”) ● handles RT updates ● simplifies publisher / consumer scenarios
  30. 30. Architecture Overview
  31. 31. Backend Components: Parser
  32. 32. Backend Components: Parser ● a parser per service and user
  33. 33. Backend Components: Parser ● a parser per service and user ● many API models supported (RT preferred)
  34. 34. Backend Components: Parser ● a parser per service and user ● many API models supported (RT preferred) ● configuration stored in the DB
  35. 35. Backend Components: Parser ● a parser per service and user ● many API models supported (RT preferred) ● configuration stored in the DB ● obtained data sent to the gateway for storage
  36. 36. Backend Components: Gateway
  37. 37. Backend Components: Gateway ● stores data in the DB for persistence
  38. 38. Backend Components: Gateway ● stores data in the DB for persistence ● pushes data to the queue for RT notifications
  39. 39. Backend Components: Gateway ● stores data in the DB for persistence ● pushes data to the queue for RT notifications ● receives updates from all the parsers
  40. 40. Backend Components: Gateway ● stores data in the DB for persistence ● pushes data to the queue for RT notifications ● receives updates from all the parsers ● exposes webservice for custom metrics
  41. 41. Backend Components: API
  42. 42. Backend Components: API ● gets historic data from the DB and ...
  43. 43. Backend Components: API ● gets historic data from the DB and ... ● … RT updates from the queue
  44. 44. Backend Components: API ● gets historic data from the DB and ... ● … RT updates from the queue ● exposes HTTP interface for data retrieval
  45. 45. Backend Components: API ● gets historic data from the DB and ... ● … RT updates from the queue ● exposes HTTP interface for data retrieval ● exposes Websocket server for RT updates
  46. 46. Frontend Components: WebApp
  47. 47. Frontend Components: WebApp ● Django application
  48. 48. Frontend Components: WebApp ● Django application ● Nginx + uWSGI
  49. 49. Frontend Components: WebApp ● Django application ● Nginx + uWSGI ● uses the DB as a normal webapp (SQLAlchemy)
  50. 50. Frontend Components: WebApp ● Django application ● Nginx + uWSGI ● uses the DB as a normal webapp (SQLAlchemy) ● Celery handles async operations
  51. 51. Frontend Components: WebApp ● Backbone.js empowers the UI
  52. 52. Frontend Components: WebApp ● Backbone.js empowers the UI ● Raphaël draws the widgets
  53. 53. Frontend Components: WebApp ● Backbone.js empowers the UI ● Raphaël draws the widgets ● a little bit of jQuery everywhere :)
  54. 54. Frontend Components: WebApp ● Backbone.js empowers the UI ● Raphaël draws the widgets ● a little bit of jQuery everywhere :) ● Websocket or Flash fallback for RT updates
  55. 55. Architecture Overview
  56. 56. Scale (or die trying)
  57. 57. Scale (or die trying) ● every component is standalone (no code deps)
  58. 58. Scale (or die trying) ● every component is standalone (no code deps) ● components talk using HTTP (Nginx + DNS win!)
  59. 59. Scale (or die trying) ● every component is standalone (no code deps) ● components talk using HTTP (Nginx + DNS win!) ● RabbitMQ allows m-gateways / n-apis scenarios
  60. 60. Security
  61. 61. Security ● HTTPS everywhere
  62. 62. Security ● HTTPS everywhere ● pgcrypto to provide credentials encryption
  63. 63. Security ● HTTPS everywhere ● pgcrypto to provide credentials encryption ● bcrypted webapp passwords
  64. 64. Security ● HTTPS everywhere ● pgcrypto to provide credentials encryption ● bcrypted webapp passwords ● relational integrity ensures data sanity
  65. 65. Well, that's it! Questions?
  66. 66. Thanks for listening! and buying ;)

×