Software Analytics with Jupyter, Pandas, jQAssistant, and Neo4j [Ne...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Software Analytics
with Jupyter, Pandas,
jQAssistant and Neo4j
Identifying Problems in Software Development
with Data Anal...

Markus Harrer
Software Development Analyst
Key Activities
Java Development, Data Analysis in Software
Development
Areas of...

Agenda
1. Motivation
2. Sofware Analytics
3. My impl of Software Analytics
4. Examples & Demos
5. Summary
6. Q&A

Motivation
Everything wrong with Software Development

Meanwhile in the pub…

Symptom Fixing

Lack of
Communication$

Politics

Why is software development
still so crazy?

WALL OF IGNORANCE
Janelle Klein: IDEAFLOW - How to Measure the PAIN in Software Development. Leanpub

WALL OF IGNORANCE
RISK
VISIBILITY
Janelle Klein: IDEAFLOW - How to Measure the PAIN in Software Development. Leanpub

RISK
DATA ANALYSIS
VISIBILITY
My wife

RISK
DATA ANALYSIS
VISIBILITY
Me

Software Analytics
Sober Problem Solving with Data Analysis based on Software Data

Software Analytics is...
“... analytics on software data
for managers and software engineers
with the aim of empowering so...

Frequency
Questions
Use standard tools
for everyday‘s questions
Use Software Analytics to
tackle high-risk problems
Risk/V...

Types of Software Data
Communitychrono-
logical
Runtimestatic
=> Problems are interconnected, so should be the data source...

Tackling problems –
automated,
data-driven and
reproducible.
MyGuideline
Software Analytics
= Data Science on Software Data

Why does it work now?
• Domain-Driven Design brings business language into code
• Data Science enables problem analysis fo...

My impl of Software Analytics
How can Developers use the Power of Data Analysis in their Daily Work?

What can you do today?
• Visualize developer contributions over time
• Identify unused, error-prone or abandoned code
• Cr...

Choose known tools
or tools for plan B*
Python
Neo4j, Pandas, Spark
* want to learn / profit from in near future
on a suit...

Notebookan open dialog with data
Context
Idea
Analysis
Conclusion
Problem
Context documented
Ideas, assumptions and
heuris...

Notebook-Driven Data Analysis

Python
Data Scientist's Best Friend: Easy, effective, fast programming
language
Pandas
Pragmatic Data Analysis Framework: ...

Advanced Tooling: jQAssistant & Neo4j
+ =
scan document validate
https://jqassistant.org/

Advanced Tooling: jQAssistant & Neo4j
Main Ideas
• Scan software structures
• Store data in Neo4j database
• Execute queri...

jQAssistant – Use Cases
Living,
self-validating
architecture
documentation

jQAssistant – Use Cases
Java Class
Business‘ Subdomain
Living,
self-validating
architecture
documentation
+
Find design &
...

Neo4j Schema for Software Data
Node Labels
File
Class
Method
Commit
Relationship Types
CONTAINS
DEPENDS_ON
INVOKES
CONTAIN...

Cypher Query
Example
Spring PetClinic
“Give me all database objects”
MATCH
(t:Type)-[:ANNOTATED_BY]->()-[:OF_TYPE]->(a:Typ...

Toolchain
Python, Jupyter
XML/Graph
Tables
Text
Data
Pandas
jQAssistant
Input
Pandas,
Neo4j
Analysis
matplotlib
xlsx
E
ppt...

Examples
The complete Toolchain in Action

Example JaCoCo  Pandas  D3
Production Coverage
1. Measure code coverage in
production
2. Calculate ratio of covered
line...

Example Git  Pandas  D3
Knowledge Island*
1. Take Git log with numstats
2. Calculate proportional
contributions for each...

Example jQAssistant  Neo4j  Pandas  D3
Dependency Analysis between Bounded Contexts
https://www.feststelltaste.de/a-gra...

Example jQAssistant  Neo4j  Pandas  D3
Dependency Analysis between Bounded Contexts
MATCH
(s1:Subdomain)<-[:BELONGS_TO]...

Example JProfiler  jQAssistant  Neo4j  Pandas
Mining performance hotspots
1. Record Call Trees
2. Identify which parts ...

Example jQAssistant  Neo4j  Pandas
Recursive Method Calls
MATCH (m:Method)-[:INVOKES*]->(m)
RETURN m

Example jQAssistant  Neo4j  Pandas
Recursive Method Calls to Database
MATCH (m:Method)-[:INVOKES*]->(m)
-[:INVOKES]->(db...

Example jQAssistant  Neo4j  Pandas
Identify possible Race Conditions
public class OwnerController {
...
private static i...

Summary

Summary
• Tooling for data analysis in software development is here!
• First analyses are easy to do using tools you alrea...

Links
Markus Harrer
• Blog: https://feststelltaste.de
• Twitter: https://twitter.com/feststelltaste
• SlideShare: https://...

Q&A
Questions and Answers

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1011447 views
Storytelling For The Web: Integrate... by Chiara Aliotta 919244 views
Artificial Intelligence, Data and C... by OECD Directorate ... 820562 views
How to Leverage AI to Boost Employe... by SocialHRCamp 342424 views
2024 State of Marketing Report – by... by Marius Sescu 195380 views
Everything You Need To Know About C... by Expeed Software 218594 views

View on Slideshare

1 of 48 Ad

View on Slideshare

1 of 48 Ad