1.
Ducksboarda real-time data orientedwebservice architecture aitor@ducksboard.com @aitorciki on twitter

2.
What is Ducksboard?

3.
What is Ducksboard?● its a web-based dashboard...

4.
What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...

5.
What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...

6.
What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...● huge bet on REAL real-time updates...

7.
What is Ducksboard?● its a web-based dashboard...● to display SaaS / your own metrics...● with heavy focus on UX...● huge bet on REAL real-time updates...● and much more in the near future!

8.
Architectural Goals

9.
Architectural Goals● data grabber / visualizer independence

10.
Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)

11.
Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)● scalability (web scale, bitches!)

12.
Architectural Goals● data grabber / visualizer independence● real-time updates (push vs pull)● scalability (web scale, bitches!)● eat your own duck food

13.
Architecture Concepts

14.
Architecture Concepts● Back-end (Twisted)

15.
Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services

16.
Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data

17.
Architecture Concepts● Back-end (Twisted) ● Parser: obtains data from external services ● Gateway: stores data ● API: provides read access to stored data

18.
Architecture Concepts● Front-end

19.
Architecture Concepts● Front-end ● web application

20.
Architecture Concepts● Front-end ● web application ● mobile web application (not yet available)

21.
Architecture Concepts● Front-end ● web application ● mobile web application (not available) ● mobile / desktop apps?

22.
Common Components

23.
Common Components● PostgreSQL (the “DB”)

24.
Common Components● PostgreSQL (the “DB”) ● stores application configuration

25.
Common Components● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries

26.
Common Components● PostgreSQL (the “DB”) ● stores application configuration ● stores data for historic queries ● starts / stops components (PL/Python!)

27.
Common Components● RabbitMQ (the “queue”)

28.
Common Components● RabbitMQ (the “queue”) ● handles RT updates

29.
Common Components● RabbitMQ (the “queue”) ● handles RT updates ● simplifies publisher / consumer scenarios

30.
Architecture Overview

31.
Backend Components: Parser

32.
Backend Components: Parser● a parser per service and user

33.
Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)

34.
Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)● configuration stored in the DB

35.
Backend Components: Parser● a parser per service and user● many API models supported (RT preferred)● configuration stored in the DB● obtained data sent to the gateway for storage

36.
Backend Components: Gateway

37.
Backend Components: Gateway● stores data in the DB for persistence

38.
Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications

39.
Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications● receives updates from all the parsers

40.
Backend Components: Gateway● stores data in the DB for persistence● pushes data to the queue for RT notifications● receives updates from all the parsers● exposes webservice for custom metrics

41.
Backend Components: API

42.
Backend Components: API● gets historic data from the DB and ...

43.
Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue

44.
Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue● exposes HTTP interface for data retrieval

45.
Backend Components: API● gets historic data from the DB and ...● … RT updates from the queue● exposes HTTP interface for data retrieval● exposes Websocket server for RT updates

46.
Frontend Components: WebApp

47.
Frontend Components: WebApp● Django application

48.
Frontend Components: WebApp● Django application● Nginx + uWSGI

49.
Frontend Components: WebApp● Django application● Nginx + uWSGI● uses the DB as a normal webapp (SQLAlchemy)

50.
Frontend Components: WebApp● Django application● Nginx + uWSGI● uses the DB as a normal webapp (SQLAlchemy)● Celery handles async operations

51.
Frontend Components: WebApp● Backbone.js empowers the UI

52.
Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets

53.
Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets● a little bit of jQuery everywhere :)

54.
Frontend Components: WebApp● Backbone.js empowers the UI● Raphaël draws the widgets● a little bit of jQuery everywhere :)● Websocket or Flash fallback for RT updates

55.
Architecture Overview

56.
Scale (or die trying)

57.
Scale (or die trying)● every component is standalone (no code deps)

58.
Scale (or die trying)● every component is standalone (no code deps)● components talk using HTTP (Nginx + DNS win!)

59.
Scale (or die trying)● every component is standalone (no code deps)● components talk using HTTP (Nginx + DNS win!)● RabbitMQ allows m-gateways / n-apis scenarios

60.
Security

61.
Security● HTTPS everywhere

62.
Security● HTTPS everywhere● pgcrypto to provide credentials encryption

63.
Security● HTTPS everywhere● pgcrypto to provide credentials encryption● bcrypted webapp passwords

64.
Security● HTTPS everywhere● pgcrypto to provide credentials encryption● bcrypted webapp passwords● relational integrity ensures data sanity

65.
Well, thats it! Questions?

66.
Thanks for listening! and buying ;)