Successfully reported this slideshow.
Your SlideShare is downloading. ×

Visualizing code bases for CraftConf 2016 / Full Stack Meetup Budapest

Visualizing code bases for CraftConf 2016 / Full Stack Meetup Budapest

Download to read offline

Code lines are added at high speed? Too many developers? How do you see the big picture? What processes are going on with your large code base? This presentation will show how to leverage existing code analysis tools (Gource, D3, Git and others) to make at least some sense out of millions of code lines and their history. During his consulting work, author often meets unfamiliar and at the same time large code bases that need quick analysis and input for decision making. That’s where visualizations come to into play by helping mining important knowledge directly from the code statistics.

Code lines are added at high speed? Too many developers? How do you see the big picture? What processes are going on with your large code base? This presentation will show how to leverage existing code analysis tools (Gource, D3, Git and others) to make at least some sense out of millions of code lines and their history. During his consulting work, author often meets unfamiliar and at the same time large code bases that need quick analysis and input for decision making. That’s where visualizations come to into play by helping mining important knowledge directly from the code statistics.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Visualizing code bases for CraftConf 2016 / Full Stack Meetup Budapest

  1. 1. 01
  2. 2. Hello! 02
  3. 3. About me 03
  4. 4. Andrey Adamovich Java/Groovy developer DevOps guy, automation junkie Co­author of Groovy 2 Cookbook Co­organizer of @latcraft and @devternity Coach at @devchampions Twitter: @codingandrey • • • • • • 04
  5. 5. Let's start! 05
  6. 6. Background Find new clients Face big code bases Need quick analysis Need quick results • • • • 06
  7. 7. Code size Google: 2 billions LOC Facebook: 61 million LOC Me: from 20K to 2M LOC • • • 07
  8. 8. I hate reading... 08
  9. 9. Reading the code I want this to work This just works Working is the definition of this block Work! You, stupid! The work should be here 01. 02. 03. 04. 05. 09
  10. 10. It's life, Jim, but not as we know it! 10
  11. 11. Code is data! 11
  12. 12. Well, big data 12
  13. 13. Snapshot 13
  14. 14. Temporal 14
  15. 15. First steps 15
  16. 16. Count the lines! 16
  17. 17. Size does matter Gives you an estimate (on how much reading is needed). Most of the code bases are polyglot. Ratio between languages can tell something. Ratio between test code, comments, blank lines is also interesting. • • • 17
  18. 18. cloc 18
  19. 19. Usage cloc ‐‐help cloc ‐‐write‐lang‐def=lang.defs 01. 02. 19
  20. 20. Usage cloc ‐‐csv       ‐‐quite       ‐‐progress‐rate=0      ‐‐ignored=files.ignored       ‐‐exclude‐dir=test,build      ‐‐read‐lang‐def=lang.defs       ‐‐out=data.csv      . 01. 02. 03. 04. 05. 06. 07. 08. 20
  21. 21. Language definitions Gradle     filter remove_matches ^s*//     filter remove_inline //.*$     filter call_regexp_common C     extension gradle     3rd_gen_scale 4.10 01. 02. 03. 04. 05. 06. 21
  22. 22. Where are the pictures? 22
  23. 23. We have stats, let's plot them!23
  24. 24. Excel? Probably not. 24
  25. 25. Pie charts are boring! 25
  26. 26. Infographics! 26
  27. 27. d3.js 27
  28. 28. d3.js Javascript library for data visualizations Tons of examples Many libraries built on top of d3.js Several books • • • • 28
  29. 29. Example 29
  30. 30. Demo 30
  31. 31. Many alternatives vis.js raphael.js sigma.js many, many, many more • • • • 31
  32. 32. Tableau Public 32
  33. 33. Tableau Public 33
  34. 34. Tableau Public 34
  35. 35. infogr.am 35
  36. 36. infogr.am 36
  37. 37. Temporal analysis 37
  38. 38. GitHub 38
  39. 39. GitHub 39
  40. 40. FishEye 40
  41. 41. Gource Software projects are displayed by Gource as an animated tree with the root directory of the project at its centre.  Directories appear as branches with files as leaves.  Developers can be seen working on the tree at the times they contributed to the project. “ 41
  42. 42. Launch gource gource ‐s 0.1 ‐1280x720  gource ‐‐output‐custom‐log log1.txt  gource ‐s 0.1 ‐1280x720 log1.txt 01. 02. 03. 42
  43. 43. Log format 1444125624|Luke Daley|M|/ratpack‐core/.../WiretapPublisher.java 1444125624|Luke Daley|M|/ratpack‐core/.../YieldingPublisher.java 1444306114|Stian Lindhom|M|/ratpack‐manual/.../13‐http.md 1444312172|Andrey Antukh|M|/ratpack‐core/.../WebSocketEngine.java 01. 02. 03. 04. 43
  44. 44. Demo 44
  45. 45. Code Maat Code Maat is a command line tool used to mine and analyze data from version­control systems “ 45
  46. 46. Launch Maat git log ‐‐pretty=format:'[%h] %aN %ad %s'      ‐‐date=short ‐‐numstat ‐‐after=YYYY‐MM‐DD 01. 02. 46
  47. 47. Launch Maat maat ‐l git.log ‐c git ‐a summary  maat ‐l ratpack_evo.log ‐c git ‐a revisions > ratpack_freqs.csv python scripts/merge_comp_freqs.py ratpack_freqs.csv ratpack_lines 01. 02. 03. 47
  48. 48. Analysis types abs­churn, age, author­churn, authors, communication, coupling, entity­churn, entity­effort, entity­ownership, fragmentation, identity, main­dev, main­dev­by­revs, messages, refactoring­main­dev, revisions, soc, summary 48
  49. 49. Demo 49
  50. 50. Let's talk big data 50
  51. 51. ElasticSearch Distributed, scalable, and highly available Real­time search and analytics capabilities Sophisticated RESTful API • • • 51
  52. 52. Index data $ curl ‐XPUT 'http://localhost:9200/gitlog/commit/123345' ‐d '{   "commitId" : "123345",   "timestamp" : "2009‐11‐15T14:12:12",   "message" : "git into es" }' 01. 02. 03. 04. 05. 52
  53. 53. Kibana Flexible analytics and visualization platform Real­time summary and charting of streaming data Intuitive interface for a variety of users Instant sharing and embedding of dashboards • • • • 53
  54. 54. Kibana 54
  55. 55. Tag Cloud kibana plugin ‐i tagcloud  ‐u https://github.com/stormpython/tagcloud/archive/master.zip 01. 02. 55
  56. 56. Demo 56
  57. 57. Structure 57
  58. 58. Structure101 58
  59. 59. Demo 59
  60. 60. Graphs are everywhere! 60
  61. 61. jQAssistant jQAssistant is a QA tool which allows the definition and validation of project specific rules on a structural level.  It is built upon the graph database Neo4j and can easily be plugged into the build process to automate detection of constraint violations and generate reports about user defined concepts and metrics. “ 61
  62. 62. Query your code MATCH   (class:Class)‐[:DECLARES]‐>(method:Method) RETURN   class.fqn, count(method) as Methods ORDER BY   Methods DESC LIMIT 20 01. 02. 03. 04. 05. 06. 07. 62
  63. 63. neo4j 63
  64. 64. Demo 64
  65. 65. That's it! 65
  66. 66. Conclusion Extract data from your code! Visualize it! Search for new facts and knowledge! • • • 66
  67. 67. Reading material 67
  68. 68. Your Code as a Crime Scene 68
  69. 69. Metrics and Models in SQE 69
  70. 70. Software Metrics and Metrology 70
  71. 71. D3 Cookbook 71
  72. 72. D3 Tips & Tricks 72
  73. 73. Links: images http://abstrusegoose.com/432 http://camarenaphoto.tumblr.com/post/112238079516/its­life­jim­but­ not­as­we­know­it­spock http://technology.ie/big­data­looks­like/ http://www.informationisbeautiful.net/visualizations/million­lines­of­ code/ http://githut.info/ http://emmanueloga.com/2013/10/07/Graphs­are­Everywhere­An­ overview­of­GraphConnect­San­Francisco­2013.html • • • • • • 73
  74. 74. Links: tools https://github.com/AlDanial/cloc http://gource.io/ http://wettel.github.io/codecity­wof.html https://github.com/adamtornhill/code­maat http://d3js.org/ http://visjs.org/ • • • • • • 74
  75. 75. Links: tools http://www.sonarqube.org/ https://www.atlassian.com/pt/software/fisheye/overview https://www.elastic.co/products/elasticsearch https://www.elastic.co/products/kibana http://jqassistant.org/ http://neo4j.com/ • • • • • • 75
  76. 76. Links: tools https://github.com/ThoughtWorksStudios/saikuro_treemap https://www.youtube.com/watch?v=iiIytERhV9o • • 76
  77. 77. Köszönöm! 77
  78. 78. Thank you! 78
  79. 79. Questions? 79

×