10 minute lightning talk delivered at Agilasverige (Agile Sweden) 2011

1. What’s happening in production
Mårten Gustafson
@martengustafson
http://marten.gustafson.pp.se/
marten.gustafson@gmail.com

2. Hi!

6. which services are utilized?

7. most?

8. least?

9. most memory?

10. highest latency?

11. does it work?

12. http://www.picpop.com/

13. what’s happening in
production?

14. right. this. second.

15. measure!

16. everything!

17. in real time!

18. it’s cheap...

19. ...and simple!

20. meter = metrics.meter("results",
"records",
TimeUnit.SECONDS)
...calculate result...
meter.mark(results.size())

21. counter = Metrics.newCounter(
getClass(),
"requests")
...receive request...
counter.inc()

22. counters

23. timers

24. averages

25. etc...

26. how?

27. develop something, or...

28. ...for java & scala use:
https://github.com
/codahale/metrics

29. JMX

30. Servlet
=
HTTP + JSON

31. ete.v2","application_status":"Ok","counter_tot":0,"request_counters":[{"name":"Autocomple
{"name":"AutocompleteService","counter":0},{"name":"AutocompleteService: Queries exec
"AutocompleteService","status":"Ok"}]},{"name":"geo","application_status":"Ok","counter_to
0},{"name":"PointConverter-deprecated","counter":0}],"monitored_services":[{"name":"Point
ication_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhone","c
monitored_services":[{"name":"RouteService","status":"Ok"}]},{"name":"image","application
:[{"name":"ImageService","counter":19}],"monitored_services":[{"name":"ImageService","sta
n_status":"Ok","counter_tot":0,"request_counters":[{"name":"LocalityResolverService-cityN
e-countyCode","counter":0},{"name":"LocalityResolverService-countyName","counter":0}],"
erService","status":"Ok"}]},{"name":"sms.v2","application_status":"Ok","counter_tot":0,"requ
s","counter":0},{"name":"SmsService","counter":0}],"monitored_services":[{"name":"SmsServ
":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-forecasts","counter":0
ame":"WeatherService-forecast","counter":0}],"monitored_services":[{"name":"WeatherServi
plication_status":"Ok","counter_tot":3,"request_counters":[{"name":"AutocompleteService:r
ervice","counter":3}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}]}
unter_tot":55,"request_counters":[{"name":"PoiService:chunks","counter":47},{"name":"PoiS
":"PoiService:poi","counter":0}],"monitored_services":[{"name":"PoiService","status":"Ok"}]},
ication_status":"Ok","counter_tot":12244,"request_counters":[{"name":"memory.allocated"
2749},{"name":"memory.max","counter":4079},{"name":"memory.used","counter":1329},{"
:"JvmMonitor","status":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot
r-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts","counter":0},{"nam
ame":"WeatherService-total","counter":0},{"name":"WeatherService-everythirdhour-forecas
ange-forecast","counter":0},{"name":"WeatherService-oneperday-forecast","counter":0}],"
[{"name":"WeatherService","status":"Ok"}]}]}

32. ete.v2","application_status":"Ok","counter_tot":0,"request_counters":[{"name":"Autocomple
{"name":"AutocompleteService","counter":0},{"name":"AutocompleteService: Queries exec
"AutocompleteService","status":"Ok"}]},{"name":"geo","application_status":"Ok","counter_to
0},{"name":"PointConverter-deprecated","counter":0}],"monitored_services":[{"name":"Point
ication_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhone","c
monitored_services":[{"name":"RouteService","status":"Ok"}]},{"name":"image","application
:[{"name":"ImageService","counter":19}],"monitored_services":[{"name":"ImageService","sta
n_status":"Ok","counter_tot":0,"request_counters":[{"name":"LocalityResolverService-cityN
e-countyCode","counter":0},{"name":"LocalityResolverService-countyName","counter":0}],"
erService","status":"Ok"}]},{"name":"sms.v2","application_status":"Ok","counter_tot":0,"requ
s","counter":0},{"name":"SmsService","counter":0}],"monitored_services":[{"name":"SmsServ
":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-forecasts","counter":0
ame":"WeatherService-forecast","counter":0}],"monitored_services":[{"name":"WeatherServi
cation_status":"ok","counter_tot":3,"request_counters":[{"name":"AutocompleteServ
ervice","counter":3}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}]}
unter_tot":55,"request_counters":[{"name":"PoiService:chunks","counter":47},{"name":"PoiS
":"PoiService:poi","counter":0}],"monitored_services":[{"name":"PoiService","status":"Ok"}]},
ication_status":"Ok","counter_tot":12244,"request_counters":[{"name":"memory.allocated"
2749},{"name":"memory.max","counter":4079},{"name":"memory.used","counter":1329},{"
:"JvmMonitor","status":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot
r-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts","counter":0},{"nam
ame":"WeatherService-total","counter":0},{"name":"WeatherService-everythirdhour-forecas
ange-forecast","counter":0},{"name":"WeatherService-oneperday-forecast","counter":0}],"
[{"name":"WeatherService","status":"Ok"}]}]}

33. ete.v2","application_status":"Ok","counter_tot":0,"request_counters":[{"name":"Autocomple
{"name":"AutocompleteService","counter":0},{"name":"AutocompleteService: Queries exec
"AutocompleteService","status":"Ok"}]},{"name":"geo","application_status":"Ok","counter_to
0},{"name":"PointConverter-deprecated","counter":0}],"monitored_services":[{"name":"Point
ication_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhone","c
monitored_services":[{"name":"RouteService","status":"Ok"}]},{"name":"image","application
:[{"name":"ImageService","counter":19}],"monitored_services":[{"name":"ImageService","sta
n_status":"Ok","counter_tot":0,"request_counters":[{"name":"LocalityResolverService-cityN
e-countyCode","counter":0},{"name":"LocalityResolverService-countyName","counter":0}],"
erService","status":"Ok"}]},{"name":"sms.v2","application_status":"Ok","counter_tot":0,"requ
s","counter":0},{"name":"SmsService","counter":0}],"monitored_services":[{"name":"SmsServ
":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-forecasts","counter":0
ame":"WeatherService-forecast","counter":0}],"monitored_services":[{"name":"WeatherServi
plication_status":"Ok","counter_tot":3,"request_counters":[{"name":"AutocompleteService:r
ervice","counter":3}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}]}
unter_tot":55,"request_counters":[{"name":"PoiService:chunks","counter":47},{"name":"PoiS
":"PoiService:poi","counter":0}],"monitored_services":[{"name":"PoiService","status":"Ok"}]},
ication_status":"Ok","counter_tot":12244,"request_counters":[{"name":"memory.allocated"
9},{"name":"memory.max","counter":4079},{"name":"memory.used","counter": 132
:"JvmMonitor","status":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot
r-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts","counter":0},{"nam
ame":"WeatherService-total","counter":0},{"name":"WeatherService-everythirdhour-forecas
ange-forecast","counter":0},{"name":"WeatherService-oneperday-forecast","counter":0}],"
[{"name":"WeatherService","status":"Ok"}]}]}

34. ete.v2","application_status":"Ok","counter_tot":0,"request_counters":[{"name":"Autocomple
{"name":"AutocompleteService","counter":0},{"name":"AutocompleteService: Queries exec
"AutocompleteService","status":"Ok"}]},{"name":"geo","application_status":"Ok","counter_to
0},{"name":"PointConverter-deprecated","counter":0}],"monitored_services":[{"name":"Point
ication_status":"Ok","counter_tot":1,"request_counters":[{"name":"RouteService-iPhone","c
RouteService","counter":1}],"monitored_services":[{"name":"RouteService","status":"Ok"}]},
us":"Ok","counter_tot": 19,"request_counters":[{"name":"ImageService","counter":19
"}]},{"name":"locality","application_status":"Ok","counter_tot":0,"request_counters":[{"name
":"LocalityResolverService-countyCode","counter":0},{"name":"LocalityResolverService-cou
calityResolverService","status":"Ok"}]},{"name":"sms.v2","application_status":"Ok","counter
s","counter":0},{"name":"SmsService","counter":0}],"monitored_services":[{"name":"SmsServ
":"Ok","counter_tot":0,"request_counters":[{"name":"WeatherService-forecasts","counter":0
ame":"WeatherService-forecast","counter":0}],"monitored_services":[{"name":"WeatherServi
plication_status":"Ok","counter_tot":3,"request_counters":[{"name":"AutocompleteService:r
ervice","counter":3}],"monitored_services":[{"name":"AutocompleteService","status":"Ok"}]}
unter_tot":55,"request_counters":[{"name":"PoiService:chunks","counter":47},{"name":"PoiS
":"PoiService:poi","counter":0}],"monitored_services":[{"name":"PoiService","status":"Ok"}]},
ication_status":"Ok","counter_tot":12244,"request_counters":[{"name":"memory.allocated"
2749},{"name":"memory.max","counter":4079},{"name":"memory.used","counter":1329},{"
:"JvmMonitor","status":"Ok"}]},{"name":"weather.v2","application_status":"Ok","counter_tot
r-forecast","counter":0},{"name":"WeatherService-rectangle-forecasts","counter":0},{"nam
ame":"WeatherService-total","counter":0},{"name":"WeatherService-everythirdhour-forecas
ange-forecast","counter":0},{"name":"WeatherService-oneperday-forecast","counter":0}],"
[{"name":"WeatherService","status":"Ok"}]}]}

35. data is good

36. visualized data
is better

37. visualize

41. visualization = insight

42. insight = knowledge

43. knowledge = safety

44. does it work?

45. No.

46. does it work?

47. Yes.

48. which services are utilized?

50. available memory?

52. 1 screen
+
1 computer
=
visualization!

53. why not more?

54. think build server...

55. ...only bigger...

56. ...and more

57. what’s happening in
production?

59. visualizing data in
real time gives
insight and safety