Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ducksboarda real-time data orientedwebservice architecture             aitor@ducksboard.com                @aitorciki on t...
What is Ducksboard?
What is Ducksboard?●    its a web-based dashboard...
What is Ducksboard?●    its a web-based dashboard...●    to display SaaS / your own metrics...
What is Ducksboard?●    its a web-based dashboard...●    to display SaaS / your own metrics...●    with heavy focus on UX...
What is Ducksboard?●    its a web-based dashboard...●    to display SaaS / your own metrics...●    with heavy focus on UX....
What is Ducksboard?●    its a web-based dashboard...●    to display SaaS / your own metrics...●    with heavy focus on UX....
Architectural Goals
Architectural Goals●    data grabber / visualizer independence
Architectural Goals●    data grabber / visualizer independence●    real-time updates (push vs pull)
Architectural Goals●    data grabber / visualizer independence●    real-time updates (push vs pull)●    scalability (web s...
Architectural Goals●    data grabber / visualizer independence●    real-time updates (push vs pull)●    scalability (web s...
Architecture Concepts
Architecture Concepts●    Back-end (Twisted)
Architecture Concepts●    Back-end (Twisted)     ●         Parser: obtains data from external services
Architecture Concepts●    Back-end (Twisted)     ●         Parser: obtains data from external services     ●         Gatew...
Architecture Concepts●    Back-end (Twisted)     ●         Parser: obtains data from external services     ●         Gatew...
Architecture Concepts●    Front-end
Architecture Concepts●    Front-end     ●         web application
Architecture Concepts●    Front-end     ●         web application     ●         mobile web application (not yet available)
Architecture Concepts●    Front-end     ●         web application     ●         mobile web application (not available)    ...
Common Components
Common Components●    PostgreSQL (the “DB”)
Common Components●    PostgreSQL (the “DB”)     ●         stores application configuration
Common Components●    PostgreSQL (the “DB”)     ●         stores application configuration     ●         stores data for h...
Common Components●    PostgreSQL (the “DB”)     ●         stores application configuration     ●         stores data for h...
Common Components●    RabbitMQ (the “queue”)
Common Components●    RabbitMQ (the “queue”)     ●         handles RT updates
Common Components●    RabbitMQ (the “queue”)     ●         handles RT updates     ●   simplifies publisher / consumer scen...
Architecture Overview
Backend Components: Parser
Backend Components: Parser●    a parser per service and user
Backend Components: Parser●    a parser per service and user●    many API models supported (RT preferred)
Backend Components: Parser●    a parser per service and user●    many API models supported (RT preferred)●    configuratio...
Backend Components: Parser●    a parser per service and user●    many API models supported (RT preferred)●    configuratio...
Backend Components: Gateway
Backend Components: Gateway●    stores data in the DB for persistence
Backend Components: Gateway●    stores data in the DB for persistence●    pushes data to the queue for RT notifications
Backend Components: Gateway●    stores data in the DB for persistence●    pushes data to the queue for RT notifications●  ...
Backend Components: Gateway●    stores data in the DB for persistence●    pushes data to the queue for RT notifications●  ...
Backend Components: API
Backend Components: API●    gets historic data from the DB and ...
Backend Components: API●    gets historic data from the DB and ...●    … RT updates from the queue
Backend Components: API●    gets historic data from the DB and ...●    … RT updates from the queue●    exposes HTTP interf...
Backend Components: API●    gets historic data from the DB and ...●    … RT updates from the queue●    exposes HTTP interf...
Frontend Components: WebApp
Frontend Components: WebApp●    Django application
Frontend Components: WebApp●    Django application●    Nginx + uWSGI
Frontend Components: WebApp●    Django application●    Nginx + uWSGI●    uses the DB as a normal webapp (SQLAlchemy)
Frontend Components: WebApp●    Django application●    Nginx + uWSGI●    uses the DB as a normal webapp (SQLAlchemy)●    C...
Frontend Components: WebApp●    Backbone.js empowers the UI
Frontend Components: WebApp●    Backbone.js empowers the UI●    Raphaël draws the widgets
Frontend Components: WebApp●    Backbone.js empowers the UI●    Raphaël draws the widgets●    a little bit of jQuery every...
Frontend Components: WebApp●    Backbone.js empowers the UI●    Raphaël draws the widgets●    a little bit of jQuery every...
Architecture Overview
Scale (or die trying)
Scale (or die trying)●    every component is standalone (no code deps)
Scale (or die trying)●    every component is standalone (no code deps)●    components talk using HTTP (Nginx + DNS win!)
Scale (or die trying)●    every component is standalone (no code deps)●    components talk using HTTP (Nginx + DNS win!)● ...
Security
Security●    HTTPS everywhere
Security●    HTTPS everywhere●    pgcrypto to provide credentials encryption
Security●    HTTPS everywhere●    pgcrypto to provide credentials encryption●    bcrypted webapp passwords
Security●    HTTPS everywhere●    pgcrypto to provide credentials encryption●    bcrypted webapp passwords●    relational ...
Well, thats it! Questions?
Thanks for listening!     and buying ;)
Upcoming SlideShare
Loading in …5
×

Ducksboard - A real-time data oriented webservice architecture

6,161 views

Published on

Talk by @aitorciki at BcnDevCon'11.

Published in: Technology, Business
  • Be the first to comment

Ducksboard - A real-time data oriented webservice architecture

  1. 1. Ducksboarda real-time data orientedwebservice architecture aitor@ducksboard.com @aitorciki on twitter
  2. 2. What is Ducksboard?
  3. 3. What is Ducksboard?● its a web-based dashboard...
  4. 4. What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...
  5. 5. What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...
  6. 6. What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...● huge bet on REAL real-time updates...
  7. 7. What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...● huge bet on REAL real-time updates...● and much more in the near future!
  8. 8. Architectural Goals
  9. 9. Architectural Goals● data grabber / visualizer independence
  10. 10. Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)
  11. 11. Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)● scalability (web scale, bitches!)
  12. 12. Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)● scalability (web scale, bitches!)● eat your own duck food
  13. 13. Architecture Concepts
  14. 14. Architecture Concepts● Back-end (Twisted)
  15. 15. Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services
  16. 16. Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data
  17. 17. Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data ● API: provides read access to stored data
  18. 18. Architecture Concepts● Front-end
  19. 19. Architecture Concepts● Front-end ● web application
  20. 20. Architecture Concepts● Front-end ● web application ● mobile web application (not yet available)
  21. 21. Architecture Concepts● Front-end ● web application ● mobile web application (not available) ● mobile / desktop apps?
  22. 22. Common Components
  23. 23. Common Components● PostgreSQL (the “DB”)
  24. 24. Common Components● PostgreSQL (the “DB”) ● stores application configuration
  25. 25. Common Components● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries
  26. 26. Common Components● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries ● starts / stops components (PL/Python!)
  27. 27. Common Components● RabbitMQ (the “queue”)
  28. 28. Common Components● RabbitMQ (the “queue”) ● handles RT updates
  29. 29. Common Components● RabbitMQ (the “queue”) ● handles RT updates ● simplifies publisher / consumer scenarios
  30. 30. Architecture Overview
  31. 31. Backend Components: Parser
  32. 32. Backend Components: Parser● a parser per service and user
  33. 33. Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)
  34. 34. Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)● configuration stored in the DB
  35. 35. Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)● configuration stored in the DB● obtained data sent to the gateway for storage
  36. 36. Backend Components: Gateway
  37. 37. Backend Components: Gateway● stores data in the DB for persistence
  38. 38. Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications
  39. 39. Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications● receives updates from all the parsers
  40. 40. Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications● receives updates from all the parsers● exposes webservice for custom metrics
  41. 41. Backend Components: API
  42. 42. Backend Components: API● gets historic data from the DB and ...
  43. 43. Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue
  44. 44. Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue● exposes HTTP interface for data retrieval
  45. 45. Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue● exposes HTTP interface for data retrieval● exposes Websocket server for RT updates
  46. 46. Frontend Components: WebApp
  47. 47. Frontend Components: WebApp● Django application
  48. 48. Frontend Components: WebApp● Django application● Nginx + uWSGI
  49. 49. Frontend Components: WebApp● Django application● Nginx + uWSGI● uses the DB as a normal webapp (SQLAlchemy)
  50. 50. Frontend Components: WebApp● Django application● Nginx + uWSGI● uses the DB as a normal webapp (SQLAlchemy)● Celery handles async operations
  51. 51. Frontend Components: WebApp● Backbone.js empowers the UI
  52. 52. Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets
  53. 53. Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets● a little bit of jQuery everywhere :)
  54. 54. Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets● a little bit of jQuery everywhere :)● Websocket or Flash fallback for RT updates
  55. 55. Architecture Overview
  56. 56. Scale (or die trying)
  57. 57. Scale (or die trying)● every component is standalone (no code deps)
  58. 58. Scale (or die trying)● every component is standalone (no code deps)● components talk using HTTP (Nginx + DNS win!)
  59. 59. Scale (or die trying)● every component is standalone (no code deps)● components talk using HTTP (Nginx + DNS win!)● RabbitMQ allows m-gateways / n-apis scenarios
  60. 60. Security
  61. 61. Security● HTTPS everywhere
  62. 62. Security● HTTPS everywhere● pgcrypto to provide credentials encryption
  63. 63. Security● HTTPS everywhere● pgcrypto to provide credentials encryption● bcrypted webapp passwords
  64. 64. Security● HTTPS everywhere● pgcrypto to provide credentials encryption● bcrypted webapp passwords● relational integrity ensures data sanity
  65. 65. Well, thats it! Questions?
  66. 66. Thanks for listening! and buying ;)

×