Developing a Service Dashboard: keeping an eye on things

4,174 views

Published on

This session walks the user through the process of developing a custom service dashboard for the Blackboard Learn platform, which was deployed in production December 2011. It displays metrics such as disk space, database tables, processor load, connections, the number of users, etc., automatically updating.
The author shares the measures used and how they were obtained, the APIs used to display the content and manage access, and the use of ajax and Google charts to provide live updates. Some time is spent explaining the design philosophy so that viewers aren’t dazzled by an array of blinking lights.
It concludes showing how we have incorporated other monitoring tools into the dashboard and our plans for the future.

Delivered at BbWorld 2012 in New Orleans

Published in: Education, Technology
  • Be the first to comment

Developing a Service Dashboard: keeping an eye on things

  1. Developing a Service Dashboard: keeping an eye on things 12th July 2012 Dr Malcolm Murray
  2. Bb: a complex system to manage
  3. People need a simple dashboard
  4. What should it look like?
  5. Complexity 5
  6. A complex system to manage F5
  7. Many interfacesToo many interfaces
  8. What should it do? Monitor Measure Alert Email Report Make Coffee?
  9. Measurement 9
  10. What should we measure?Disk SpaceTable SizeLoadUsers
  11. Database MeasuresDisks usedTable SizeTrends
  12. Database Measures
  13. Detail from the Sparkline
  14. Application ServersDisk spaceLoadConnections
  15. Users
  16. Digg Effect or DOS?
  17. Whose Online Now?With thanks to Santo Nucifora at Seneca College
  18. Implementation 18
  19. A lot of shell scriptscd /local/bboard/blackboard/contentdf -P . | grep -v 1024-blocks | awk {print "insert into dur_dashboard_data (when,name, space, capacity, used, available) values(sysdate,?duocontent?,?"$1"?,?"$2"?,?"$3"?,?"$4"?);"}|sed "s^?^^g" >> /local/home/bbuser/sc/intotable.sqlFilesystem 1024-blocks Used Available Capacity Mounted on/dev/sda1 16246428 3231304 12176772 21% /tmpfs 32995944 2401084 30594860 8% /dev/shm/dev/mapper/vg0-s01 51606140 3914360 45070340 8% /s01/dev/mapper/vg0-s02 51606140 14266484 34718216 30% /s02/dev/mapper/vg0-data01 258030980 181684776 63239004 75% /data01/dev/mapper/vg0-data02 258030980 88587208 156336572 37% /data02/dev/mapper/vg0-data03 309637120 284903840 9005280 97% /data03ssh bbuser@duoapp1 w10:23:05 up 3 days, 23:44, 0 users, load average: 0.02, 0.04, 0.00USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
  20. Database Tablespacesspool /local/home/bbuser/sc/tablespacereturn;SELECT Total.name "Tablespace Name", nvl(Free_space, 0) Free_space, nvl(total_space-Free_space, 0) Used_space, total_spaceFROM (select tablespace_name, sum(bytes/1024) Free_Space from sys.dba_free_space dfs group by tablespace_name ) Free, (select b.name, sum(bytes/1024) TOTAL_SPACE from sys.v_$datafile a, sys.v_$tablespace B where a.ts# = b.ts# group by b.name ) TotalWHERE Free.Tablespace_name(+) = Total.nameORDER BY Total.name/spool off;
  21. Database TablespacesTablespace Name FREE_SPACE USED_SPACE TOTAL_SPACE------------------------------ ---------- ---------- -----------BBADMIN_DATA 48384 2816 51200BBADMIN_INDX 18048 2432 20480BB_BB60_DATA 1766336 102734880 104501216BB_BB60_INDX 1128512 29519808 30648320BB_BB60_STATS_DATA 3545344 76160752 79706096BB_BB60_STATS_INDX 6362880 141035728 147398608CMS_DATA 4736 81664 86400CMS_DOC_DATA 42368 467712 510080CMS_DOC_INDX 23040 453184 476224CMS_FILES_COURSES_DATA 139392 2779776 2919168CMS_FILES_COURSES_INDX 93760 1867072 1960832CMS_FILES_INST_DATA 204928 4090752 4295680CMS_FILES_INST_INDX 414720 3681280 4096000CMS_FILES_LIBRARY_DATA 41600 299904 341504CMS_FILES_LIBRARY_INDX 9728 177664 187392CMS_FILES_ORGS_DATA 44160 430592 474752CMS_FILES_ORGS_INDX 13376 249792 263168CMS_FILES_USERS_DATA 28672 557056 585728CMS_FILES_USERS_INDX 25408 503424 528832CMS_INDX 3136 60096 63232SYSAUX 75392 1235328 1310720SYSTEM 7616 832064 839680UNDOTBS1 4676480 341120 5017600USERS 1445056 1880384 3325440
  22. Stored in custom tables
  23. Is there a better way?Almost definitely! e.g. ZabbixMore details available:http://tinyurl.com/bbzabbix
  24. Simplicity 24
  25. Deployment Target
  26. Must work from my desk Hey folks, duoapp2 is in trouble… … and you need to tidy your desk
  27. 28
  28. 29
  29. 30
  30. 31
  31. Favourite?1 23 5 4
  32. 33
  33. Learn from othersStephen Few Edward Tufte
  34. Careful use of colour
  35. Informative Graphics 1991.1.1 65 months 2004.4.28 low high Euro foreign exchange $1.1608 1.1907 .8252 1.2858 Euro foreign exchange ¥121.32 130.17 89.30 140.31 Euro foreign exchange £0.7111 0.6665 .5711 0.7235Edward Tufte’s Sparklines
  36. Simple Code  <span class="sparkline"> 10 14 15 4.5 3.4 16 </span> http://code.google.com/p/js-sparklines/
  37. Deployment 38
  38. Light touch on production 3 million visits Oct – Dec 2011
  39. Update only the key metrics
  40. JavaScript Graphinghttp://code.google.com/apis/chart/
  41. Easy live reconfiguration
  42. Tailor to the Live System
  43. Use Meaningful Labels
  44. Tabs provide easy links
  45. What next? Better integration with the F5 More on NetApps disk usage Java Memory Utilization Number of downloads per user Decide what to make public!
  46. F5 Reporting
  47. Summary 48
  48. SummaryKeep it simpleLight touch on system being monitoredAllow dynamic reconfigurationManage access using tabs & rolesLearn from others Slides available at: http://db.tt/rp2D88Nt
  49. @malcolmmurray malcolm.murray@durham.ac.uk malcolm.murray@gmail.com We value your feedback!Please fill out a session evaluation. 50

×