Software Analytics for Pragmatists [DevOps Camp 2017]

•

0 likes•1,168 views

Talk at DevOps Camp 2017, Nürnberg, 13.05.2017 Each step in the development or use of software leaves valuable, digital tracks. The analysis of this "software data" (such as runtime measures, log files or commits) refines our gut feeling to facts with sound evidence. I'll show how questions that arise in software development can be answered automated, data-driven and reproducible. I demonstrate the interaction of open source analysis tools (such as jQAssistant, Neo4j, Pandas, and Jupyter) for the analysis of data from different sources (such as JProfiler, Jenkins, and Git). Together, we have a look at how we can develop solutions to optimize performance, identify build breaker or make knowledge gaps in our source code visible.

Software

“Without data you're
just another person
with an opinion.“
W. Edwards Deming

Motivation Software Analytics
“In software engineering
there is much we are seeing,
but little we are learning.”
Tim Menzies

Motivation
Why?
•Make problems visible
+ Improve clarity and understanding
•Drive decisions
+ Raise money
+ Raise more money for further analysis
•Support Continuous Learning
+ Master challenges
+ Thrive for improvement steadily

Claim
Solving problems –
automated,
data-centric and
reproducible.

Automation
•Data retrieval
•Analysis
•Visualization
DevOps style

Types of software artifacts and meta data
chronologicalcommunity
runtimestatic

Data is dirty
Time for cleaning up data
up to 80%

Data + Automation + Traceability
Replication

Code Book
Example:
Analyzing performance bottlenecks

Pipeline
Data Mining
• NumPy
• scikit-learn
• SciPy
Visualization
• matplotlib
• plot.ly
• Bokeh
• python-pptx
...

Pipeline
XML/Graph
Tabellen
matplotlib
Pandas,
...
Pandas
jQAssistant,
Neo4j
Text
xlsx
E
pptx
P
Python Jupyter
Input
Pre-
processing
Analysis
Output
D3

Pipeline
ZIP
GZ
*.class
JAR, WAR, EAR
MANIFEST.MF
*.properties
XSD
YAML
XML
application.xml
web.xml
beans.xml
JaCoCo
FindBugs
CheckStyle
pom.xml
surefire-reports.xml RDBMS Schema
M2 Repository
DBCSV
Excel
BigQuery
Inputs
HDFStore
Web
JSON
Git
Pandas
jQAssistant

Examples
Demos
•Performance Bottlenecks
•Build Breaker Analysis
•Mining Knowledge Islands
•Git Log Analysis

What‘s the
value of the information
related to the
effort for the analysis
of the information?
Always ask the question

References
Leek, Jeff: The Elements of Data Analytic Style.
LeanPub, 2015.
McKinney, Wes: Python For Data Analysis, O’Reilly,
2012.
Mens, Tom; Serebrenik; Cleve, Anthony:Evolving
Software Systems. Springer, 2014.
Mens, Tom; Demeyer, Serge: Software Evolution.
Springer, 2008.
Shull, Forrest; Singer, Janice; Sjøberg, Dag I.K.: Guide
to Advanced Empirical Software Engineering. Springer,
2008.
Tornhill, Adam:Your Code As a Crime Scene.
Pragmatic Programmers, 2015.

Relationships are highly predictive of behavior, yet most data science models overlook this information because it's difficult to extract network structure for use in machine learning (ML). With graphs, relationships are embedded in the data itself, making it practical to add these predictive capabilities to your existing practices. That’s why we’re presenting and demoing the use of graph-native ML to make breakthrough predictions. This will cover: - Different approaches to graph feature engineering, from queries and algorithms to embeddings - How ML techniques leverage everything from classical network science to deep learning and graph convolutional neural networks - How to generate representations of your graph using graph embeddings, create ML models for link prediction or node classification, and apply these models to add missing information to an existing graph/incoming data - Why no-code visualization and prototyping is important

Experiments With Knowledge Graphs in Fisheries & Oceans Canada

Neo4j

Bridging the Gap: Analyzing Data in and Below the Cloud

Inside Analysis

The Briefing Room with Dean Abbott and Tableau Software Live Webcast July 23, 2013 http://www.insideanalysis.com Today’s desire for analytics extends well beyond the traditional domain of Business Intelligence. That’s partly because business users are realizing the value of mixing and matching all kinds of data, from all kinds of sources. One emerging market driver is Cloud-based data, and the desire companies have to analyze this data cohesively with their on-premise data sets. Register for this episode of The Briefing Room to learn from Analyst Dean Abbott, who will explain how the ability to access data in the cloud can play a critical role for generating business value from analytics. He’ll be briefed by Ellie Fields of Tableau Software who will tout Tableau’s latest release, which includes native connectors to cloud-based applications like Salesforce.com, Amazon Redshift, Google Analytics and BigQuery. She’ll also demonstrate how Tableau can combine cloud data with other data sources, including spreadsheets, databases, cubes and even Big Data.

Neo4j Graph Use Cases, Bruno Ungermann, Neo4j

Neo4j

Webinar - Product Matching - Palombo (20160428)

Turi, Inc.

Graphs in Life Sciences

Neo4j

Neo4j Graph Data Science - Webinar

Neo4j

Let’s tackle problems in software development in an automated, data-driven and reproducible way! As developers, we often feel that there might be something wrong with the way we develop software. Unfortunately, a gut feeling alone isn’t sufficient for the complex, interconnected problems in software systems. We need solid, understandable arguments to gain budgets for improvement projects or to defend us against political decisions. Though, we can help ourselves: Every step in the development or use of software leaves valuable, digital traces. With clever analysis, these data can show us root causes of problems in our software and deliver new insights – understandable for everybody. If concrete problems and their impact are known, developers and managers can create solutions and take sustainable actions aligned to existing business goals. In this meetup, I talk about the analysis of software data by using a digital notebook approach. This allows you to express your gut feelings explicitly with the help of hypotheses, explorations and visualizations step by step. I show the collaboration of open source analysis tools (Jupyter, Pandas, jQAssistant and, of course, Neo4j) to inspect problems in Java applications and their environment. We have a look at performance hotspots, knowledge loss and worthless code parts – completely automated from raw data up to visualizations for management. Participants learn how they can translate their unsafe gut feelings into solid evidence for obtaining budgets for dedicated improvement projects with the help of data analysis.

Software Engineering for Data Scientists

Domino Data Lab

Although both disciplines are unique in their own ways, Software Engineering and Data Science make heavy use of programing languages to do their respective jobs. Data Science is a relatively new discipline and many of its practitioners have not previously been professional software engineers. There are a few techniques that Data Scientists can leverage from Software Engineering in order to make their tooling and environments, faster to design, more easily debugged and most importantly, clearer to read. This talk will be going over some practical tips that anyone can use to help better understand their code; give clarity around cloud environments, their uses and drawbacks and finally briefly touching on the Software Development Lifecycle.

Building Data Science Teams

EMC

This session describes the roles and skill sets required when building a Data Science team, and starting a data science initiative, including how to develop Data Science capabilities, select suitable organizational models for Data Science teams, and understand the role of executive engagement for enhancing analytical maturity at an organization. Objective 1: Understand the knowledge and skills needed for a Data Science team and how to acquire them. After this session you will be able to: Objective 2: Learn about the different organizational models for forming a Data Science team and how to choose the best for your organization. Objective 3: Understand the importance of Executive support for Data Science initiatives and role it plays in their successful deployment.

Introduction to Data Science

Francis Michael Bautista

Data Analytics in Real World (May 2016)

geetachauhan

JavaZone 2018 - A Practical(ish) Introduction to Data Science

Mark West

Code: https://github.com/markwest1972/titanic Video: https://vimeo.com/289705893 Data Science has been described as the sexiest job of the 21st Century. But what is Data Science? And what has Machine Learning got to do with all of this? In this talk I will share insights and knowledge that I have gained from building up a Data Science department from scratch. This talk will be split into three sections: 1. I’ll begin by defining what Data Science is, how it is related to Machine Learning and share some tips for introducing Data Science to your organisation. 2. Next up we’ll run through some commonly used Machine Learning algorithms used by Data Scientists, along with examples for use cases where these algorithms can be applied. 3. The final third of the talk will be a demonstration of how you can quickly get started with Data Science and Machine Learning using Python and the Open Source scikit-learn Library.

Neo4j 4 Overview

Neo4j

How do You Graph

Ben Krug

Worst Practices in Artificial Intelligence

William Tsoi

Applied Data Science Course Part 1: Concepts & your first ML model

Dataiku

New opportunities for connected data : Neo4j the graph database

Cédric Fauvet

Introduction to Data Science (Data Summit, 2017)

Caserta

At DBTA's 2017 Data Summit in New York, NY, Caserta Founder & President, Joe Caserta, and Senior Architect, Bill Walrond, gave a pre-conference workshop presenting the ins and outs of data science. Data scientist has been dubbed the "sexiest" job of the 21st century, but it requires an understanding of many different elements of data analysis. This presentation dives into the fundamentals of data exploration, mining, and preparation, applying the principles of statistical modeling and data visualization in real-world applications.

A field guide to the Financial Times, Rhys Evans, Financial Times

Neo4j

Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Neo4j

Webinar - Fraud Detection - Palombo (20160428)

Turi, Inc.

Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...

Neo4j

The Big Data Dream Team

Accenture Analytics

Data Scientist vs Data Analyst vs Data Engineer - Role & Responsibility, Skil...

Simplilearn

In this presentation, we will decode the basic differences between data scientist, data analyst and data engineer, based on the roles and responsibilities, skill sets required, salary and the companies hiring them. Although all these three professions belong to the Data Science industry and deal with data, there are some differences that separate them. Every person who is aspiring to be a data professional needs to understand these three career options to select the right one for themselves. Now, let us get started and demystify the difference between these three professions. We will distinguish these three professions using the parameters mentioned below: 1. Job description 2. Skillset 3. Salary 4. Roles and responsibilities 5. Companies hiring This Master’s Program provides training in the skills required to become a certified data scientist. You’ll learn the most in-demand technologies such as Data Science on R, SAS, Python, Big Data on Hadoop and implement concepts such as data exploration, regression models, hypothesis testing, Hadoop, and Spark. Why be a Data Scientist? Data scientist is the pinnacle rank in an analytics organization. Glassdoor has ranked data scientist first in the 25 Best Jobs for 2016, and good data scientists are scarce and in great demand. As a data scientist you will be required to understand the business problem, design the analysis, collect and format the required data, apply algorithms or techniques using the correct tools, and finally make recommendations backed by data. Simplilearn's Data Scientist Master’s Program will help you master skills and tools like Statistics, Hypothesis testing, Clustering, Decision trees, Linear and Logistic regression, R Studio, Data Visualization, Regression models, Hadoop, Spark, PROC SQL, SAS Macros, Statistical procedures, tools and analytics, and many more. The courseware also covers a capstone project which encompasses all the key aspects from data extraction, cleaning, visualisation to model building and tuning. These skills will help you prepare for the role of a Data Scientist. Who should take this course? The data science role requires the perfect amalgam of experience, data science knowledge, and using the correct tools and technologies. It is a good career choice for both new and experienced professionals. Aspiring professionals of any educational background with an analytical frame of mind are most suited to pursue the Data Scientist Master’s Program, including: IT professionals Analytics Managers Business Analysts Banking and Finance professionals Marketing Managers Supply Chain Network Managers Those new to the data analytics domain Students in UG/ PG Analytics Programs Learn more at https://www.simplilearn.com/big-data-and-analytics/senior-data-scientist-masters-program-training

Data+Science : A First Course

Arnab Majumdar

Product Management in the Era of Data Science

Mandar Parikh

My slide-deck from a webinar on the same topic for the Institute of Product Leadership, April 4th, 2017 What does it take to build killer products in the “AI-first” era? What makes for a great Data Science-driven product and how do great Product Managers leverage Data Science to drive value for customers? Find out how to avoid the pitfalls of hype-chasing Data Science tactics. Learn how to work with Data Science and Engineering to build a compelling product and solve real problems. Mandar takes a practitioner’s approach to present his recipe for success for building Data Science-driven products that drive enduring value for customers.

Analytics-Enabled Experiences: The New Secret Weapon

Databricks

Tracking and analyzing how our individual products come together has always been an elusive problem for Steelcase. Our problem can be thought of in the following way: “we know how many Lego pieces we sell, yet we don’t know what Lego set our customers buy.” The Data Science team took over this initiative, which resulted in an evolution of our analytics journey. It is a story of innovation, resilience, agility and grit. The effects of the COVID-19 pandemic on corporate America shined the spotlight on office furniture manufacturers to solve for ways on which the office can be made safe again. The team would have never imagined how relevant our work on product application analytics would become. Product application analytics became an industry priority overnight. The proposal presented this year is the story of how data science is helping corporations bring people back to the office and set the path to lead the reinvention of the office space. After groundbreaking milestones to overcome technical challenges, the most important question is: What do we do with this? How do we scale this? How do we turn this opportunity into a true competitive advantage? The response: stop thinking about this work as a data science project and start to think about this as an analytics-enabled experience. During our session we will cover the technical elements that we overcame as a team to set-up a pipeline that ingests semi-structured and unstructured data at scale, performs analytics and produces digital experiences for multiple users. This presentation will be particularly insightful for Data Scientists, Data Engineers and analytics leaders who are seeking to better understand how to augment the value of data for their organization

What's hot

Data science team, a practice to setup

Omid Mogharian

Software Analytics with Jupyter, Pandas, jQAssistant, and Neo4j [Neo4j Online...

Markus Harrer

Software Engineering for Data Scientists

Domino Data Lab

Building Data Science Teams

EMC

Introduction to Data Science

Francis Michael Bautista

Data Analytics in Real World (May 2016)

geetachauhan

JavaZone 2018 - A Practical(ish) Introduction to Data Science

Mark West

Neo4j 4 Overview

Neo4j

How do You Graph

Ben Krug

Worst Practices in Artificial Intelligence

William Tsoi

Applied Data Science Course Part 1: Concepts & your first ML model

Dataiku

New opportunities for connected data : Neo4j the graph database

Cédric Fauvet

Introduction to Data Science (Data Summit, 2017)

Caserta

A field guide to the Financial Times, Rhys Evans, Financial Times

Neo4j

Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Neo4j

Webinar - Fraud Detection - Palombo (20160428)

Turi, Inc.

Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...

Neo4j

The Big Data Dream Team

Accenture Analytics

Data Scientist vs Data Analyst vs Data Engineer - Role & Responsibility, Skil...

Simplilearn

Data+Science : A First Course

Arnab Majumdar

What's hot (20)

Data science team, a practice to setup

Software Analytics with Jupyter, Pandas, jQAssistant, and Neo4j [Neo4j Online...

Software Engineering for Data Scientists

Building Data Science Teams

Introduction to Data Science

Data Analytics in Real World (May 2016)

JavaZone 2018 - A Practical(ish) Introduction to Data Science

Neo4j 4 Overview

How do You Graph

Worst Practices in Artificial Intelligence

Applied Data Science Course Part 1: Concepts & your first ML model

New opportunities for connected data : Neo4j the graph database

Introduction to Data Science (Data Summit, 2017)

A field guide to the Financial Times, Rhys Evans, Financial Times

Neo4j GraphTalk Amsterdam - Next Generation Solutions using Neo4j

Webinar - Fraud Detection - Palombo (20160428)

Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...

The Big Data Dream Team

Data Scientist vs Data Analyst vs Data Engineer - Role & Responsibility, Skil...

Data+Science : A First Course

Similar to Software Analytics for Pragmatists [DevOps Camp 2017]

Product Management in the Era of Data Science

Mandar Parikh

Analytics-Enabled Experiences: The New Secret Weapon

Databricks

Gse uk-cedrinemadera-2018-shared

cedrinemadera

Choosing which big data, nosql or database technology to use

mark madsen

Basic overview of how to evaluate and match workloads to the various database technologies available, from NoSQL to relational. Workloads have different characteristics. If you don’t understand them you can end up implementing the wrong solution for the problem you have. The video from this presentation is available at https://bloorgroup.webex.com/bloorgroup/lsr.php?AT=pb&SP=EC&rID=4953842&rKey=d03b10ecd9163770

Data science 101 Masterclass

Ben Keen

Data Scientist Salary, Skills, Jobs And Resume | Data Scientist Career | Data...

Simplilearn

This presentation about "Data Science Engineer Career, Salary, and Resume" will help you understand who is a Data Science Engineer, the salary of a Data Science Engineer, Data Science Engineer Skillset and Data Science Engineer Resume. Data science is a systematic way to analyze a massive amount of data and extract information from them. Data Science can answer a lot of questions, as well. Data Science is mainly required for better decision making, predictive analysis, and pattern recognition. Below are topics that we will be discussing in this presentation: 1. Introduction to Data Science 2. Who is a Data Science Engineer 3. Data Science Engineer Skillset 4. Data Science Engineer job roles 5. Data Science Engineer salary trends 6. Data Science Engineer Resume Why learn Data Science? Data Scientists are being deployed in all kinds of industries, creating a huge demand for skilled professionals. The data scientist is the pinnacle rank in an analytics organization. Glassdoor has ranked data scientist first in the 25 Best Jobs for 2016, and good data scientists are scarce and in great demand. As a data, you will be required to understand the business problem, design the analysis, collect and format the required data, apply algorithms or techniques using the correct tools, and finally make recommendations backed by data. You can gain in-depth knowledge of Data Science by taking our Data Science with python certification training course. With Simplilearn’s Data Science certification training course, you will prepare for a career as a Data Scientist as you master all the concepts and techniques. Those who complete the course will be able to: 1. Gain an in-depth understanding of data science processes, data wrangling, data exploration, data visualization, hypothesis building, and testing. You will also learn the basics of statistics. Install the required Python environment and other auxiliary tools and libraries 2. Understand the essential concepts of Python programming such as data types, tuples, lists, dicts, basic operators and functions 3. Perform high-level mathematical computing using the NumPy package and its large library of mathematical functions Perform scientific and technical computing using the SciPy package and its sub-packages such as Integrate, Optimize, Statistics, IO, and Weave 4. Perform data analysis and manipulation using data structures and tools provided in the Pandas package 5. Gain expertise in machine learning using the Scikit-Learn package Data Science with python is recommended for: 1. Analytics professionals who want to work with Python 2. Software professionals looking to get into the field of analytics 3. IT professionals interested in pursuing a career in analytics 4. Graduates looking to build a career in analytics and data science Learn more at https://www.simplilearn.com/big-data-and-analytics/python-for-data-science-training

Operationalizing Machine Learning

AgileThought

Data and data scientists are not equal to money david hoyle

Institute of Contemporary Sciences

A lot of companies make the mistake of thinking that just hiring Data Scientists will lead to increased revenue or increased profit. For a company’s investment in Data Science to be successful the Data Scientists need to work on the right problems, with the right people, and with the right tools. In this presentation, I will talk about the lessons I have learned, and mistakes made in applying Data Science in commercial settings over the last 10 years. I will highlight what processes can increase the chances of Data Science investment being successful.

Data Modeling for Security, Privacy and Data Protection

Karen Lopez

Thinkful DC - Intro to Data Science

TJ Stalcup

From c# Into Machine Learning

Dev Raj Gautam

When we hear the Word Machine Learning we think of Self Driving Car and Advanced Medical Solutions. This brings the awe-inspiring of Huge and Complex Data, Advanced Statistics, Algebra and Sophisticated Solutions & we get scared to Build Solutions in Machine Learning. Machine Learning solutions are not that Hard to develop and the same time not that easy to make them perfect. This slide decks will provide insight and demos of How a Software Engineer can start Developing Machine Learning Solutions easily and Eventually master the Knowledge of Machine Learning.

What Managers Need to Know about Data Science

Annie Flippo

Lessons learned from over 25 Data Virtualization implementations

Denodo

The Right Data Warehouse: Automation Now, Business Value Thereafter

Inside Analysis

The Briefing Room with Dr. Robin Bloor and WhereScape Live Webcast on April 1, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7b23b14b532bd7be60a70f6bd5209f03 In the Big Data shuffle, everyone is looking at Hadoop as “the answer” to collect interesting data from a new set of sources. While Hadoop has given organizations the power to gather more information assets than ever before, the question still looms: which data, regardless of source, structure, volume and all the rest, are significant for affecting business value – and how do we harness it? One effective approach is to bolster the data warehouse environment with a solution capable of integrating all the data sources, including Hadoop, and automating delivery of key information into the rights hands. Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor as he explains how a rapidly changing information landscape impacts data management. He will be briefed by Mark Budzinski of WhereScape, who will tout his company’s data warehouse automation solutions. Budzinski will discuss how automation can be the cornerstone for closing the gap between those responsible for data management and the people driving business decisions. Visit InsideAnlaysis.com for more information.

Bespoke Data Insights at New Finance

Karthik Shashidhar

Designing for Data Security by Karen Lopez

Karen Lopez

Intel Faster Risk Oct08 - Andrew Parry

mikeohara

DiscoverText Product Overview

Stuart Shulman

An Agile Approach to Machine Learning

Randy Shoup

Machine learning has become an important tool in the modern software toolbox, and high-performing organizations are increasingly coming to rely on data science and machine learning as a core part of their business. eBay introduced machine learning to its commerce search ranking and drove double-digit increases in revenue. Stitch Fix built a multibillion dollar clothing retail business in the US by combining the best of machines with the best of humans. And WeWork is bringing machine-learned approaches to the physical office environment all around the world. In all cases, algorithmic techniques started simple and slowly became more sophisticated over time. This talk will use these examples to derive an agile approach to machine learning, and will explore that approach across several different dimensions. We will set the stage by outlining the kinds of problems that are most amenable to machine-learned approaches as well as describing some important prerequisites, including investments in data quality, a robust data pipeline, and experimental discipline. Next, we will choose the right (algorithmic) tool for the right job, and suggest how to incrementally evolve the algorithmic approaches we bring to bear. Most fancy cutting-edge recommender systems in the real world, for example, started out with simple rules-based techniques or basic regression. Finally, we will integrate machine learning into the broader product development process, and see how it can help us to accelerate business results

Data Science-Why?What?How? By Hari Prasad

Hari Prasad

Similar to Software Analytics for Pragmatists [DevOps Camp 2017] (20)

Product Management in the Era of Data Science

Analytics-Enabled Experiences: The New Secret Weapon

Gse uk-cedrinemadera-2018-shared

Choosing which big data, nosql or database technology to use

Data science 101 Masterclass

Data Scientist Salary, Skills, Jobs And Resume | Data Scientist Career | Data...

Operationalizing Machine Learning

Data and data scientists are not equal to money david hoyle

Data Modeling for Security, Privacy and Data Protection

Thinkful DC - Intro to Data Science

From c# Into Machine Learning

What Managers Need to Know about Data Science

Lessons learned from over 25 Data Virtualization implementations

The Right Data Warehouse: Automation Now, Business Value Thereafter

Bespoke Data Insights at New Finance

Designing for Data Security by Karen Lopez

Intel Faster Risk Oct08 - Andrew Parry

DiscoverText Product Overview

An Agile Approach to Machine Learning

Data Science-Why?What?How? By Hari Prasad

More from Markus Harrer

Datenanalysen in der Softwareentwicklung (IMPROVE Workshop Wien)

Markus Harrer

Softwareentwickler haben bei ihren altgedienten Anwendungssystemen oft das Bauchgefühl, dass irgendetwas komisch läuft. Das Management lässt sich aber nur mit Zahlen-Daten-Fakten von dringenden Verbesserungsarbeiten überzeugen. Genau dafür stelle ich den Bereich "Software Analytics" vor, dessen Vorgehen und Methoden darauf abzielen, Daten aus der Softwareentwicklung so aufzubereiten, dass sie von Managern zur Entscheidungsfindung herangezogen werden können. Ich bringe hierzu einen digitalen Notizbuchansatz sowie Werkzeuge für die schnelle Durchführung von nachvollziehbaren Datenanalysen mit. Damit können ganz individuelle Problemursachen in Softwaresystemen Schritt für Schritt herausgearbeitet und explizit dargestellt werden. Ich zeige das Zusammenspiel von Open-Source-Analysewerkzeugen (wie Jupyter, Pandas, jQAssistant, Neo4j und D3) zur Untersuchung von Java-Anwendungen und deren Umgebung (Git, JaCoCo, Profiler, Logfiles etc.). Im Live-Coding-Teil sehen wir uns einige Auswertungen zu Race-Conditions, Wissenslücken oder wertlosen Codeteilen an – von den Rohdaten bis zu den Visualisierungen.

Software Analytics - Datenanalysen in der Softwareentwicklung (BigDataMeetup)

Markus Harrer

Softwareentwickler haben bei ihren altgedienten Anwendungssystemen oft das Bauchgefühl, dass irgendetwas komisch läuft. Das Management lässt sich aber nur mit Zahlen-Daten-Fakten von dringenden Verbesserungsarbeiten überzeugen. In diesem Meetup stellt Markus Harrer den Bereich "Software Analytics" vor, dessen Vorgehen und Methoden darauf abzielen, Daten aus der Softwareentwicklung so aufzubereiten, dass sie von Managern zur Entscheidungsfindung herangezogen werden können. Markus bringt hierzu einen digitalen Notizbuchansatz sowie Werkzeuge für die schnelle Durchführung von nachvollziehbaren Datenanalysen mit. Damit können ganz individuelle Problemursachen in Softwaresystemen Schritt für Schritt herausgearbeitet und explizit dargestellt werden. Markus zeigt das Zusammenspiel von Open-Source-Analysewerkzeugen (wie Jupyter, Pandas, jQAssistant, Neo4j und D3) zur Untersuchung von Java-Anwendungen und deren Umgebung (Git, JaCoCo, Profiler, Logfiles etc.). Im Live-Coding-Teil sehen wir uns einige Auswertungen zu Race-Conditions, Wissenslücken, wertlosen Codeteilen sowie die Optimierung des fachlichen Schnitts einer Anwendung an – von den Rohdaten bis zu den Visualisierungen.

Datenanalysen in der Softwareentwicklung mit Software Analytics

Markus Harrer

Philosophy screws it all up (Pecha Kucha) [Java Forum Stuttgart 2017]

Markus Harrer

Wie Erkenntnistheorie unsere Softwareentwicklung beeinflusst und wie wir damit umgehen können... Softwareentwicklungsprojekte scheitern ab und an weder aus technischen, noch fachlichen – ja nicht einmal aus politischen Gründen. Sind anderweitige Gründe im Spiel, versenkt diese Art von Softwareprojektabbrüchen meist auch noch extrem viel Geld. Es ist unheimlich, dass wir diese Fälle nicht wirklich begründen können und daher vor allem nicht herausbekommen, woran es wohl wirklich gelegen hat. Es müssen somit noch andere, grundsätzliche Aspekte eine Rolle spielen, welche nicht sofort offensichtlich sind. In diesem Pecha-Kucha-Vortrag ergründe ich mit Hilfe der philosophischen Disziplin der Erkenntnistheorie – der Theorie des Wissens – einige der nicht augenscheinlichen, jedoch immer vorhandenen Probleme der Softwareentwicklung. Die vorgestellten „erkenntnistheoretischen Dilemmata“ begegnen uns immer, wenn wir es mit Wissen zu tun haben. Softwareentwicklung ist die Implementierung von Wissen in Code. Daher sind auch wir – ob wir es wollen oder nicht – immer von diesen Dilemmata betroffen. Es gibt jedoch Hoffnung: Sind diese Grundprobleme erst einmal bekannt, können wir sie nutzen, um unsere eigene Arbeit – unser Vorgehen, unserer eingesetzten Methoden und Praktiken in der Softwareentwicklung – nach objektiven Kriterien zu beurteilen und zu verbessern. Konkret zeige ich, wie sich die normative Wissenschaft der Erkenntnistheorie ganz praktisch als „Messwerkzeug“ verwenden lässt. Dazu stelle ich ihre Einflussnahme auf softwaretechnische Praktiken anhand von Praxisbeispielen vor. Meine Kernziele sind es, die Dilemmata kurz und prägnant darzustellen, handhabbar zu machen und praxisorientierte Lösungsansätze zum Umgang mit ihnen aufzuzeigen.

Architektur und Code im Einklang [JUG Nürnberg]

Markus Harrer

28.06.2017, Java User Group Nürnberg Umsetzung von Architekturkonzepten und Designvorgaben mit jQAssistant direkt im Quellcode überprüfen jQAssistant ist ein Werkzeug zur statischen Analyse von Softwareartefakten. Mit jQAssistant ist es möglich, Architektur- und Designkonzepte als einen Satz von Regeln zu definieren. Die Einhaltung dieser Regeln kann anschließend bei jedem Bau einer Software automatisiert überprüft werden. Dadurch kann sichergestellt werden, dass eine vorgegebene Architektur sowie Designvorgaben auch wirklich im Quellcode umgesetzt wurden. In meinem Vortrag stelle ich die Grundlagen von jQAssistant vor und zeige an einem Beispiel, wie sich mit dem Werkzeug eine sich selbst prüfende Architekturdokumentation erstellen lässt.

Architektur und Code im Einklang [DeveloperCamp 2017]

Markus Harrer

18.05.2017, DeveloperCamp 2017, Nürnberg Umsetzung von Architekturkonzepten und Designvorgaben mit jQAssistant direkt im Quellcode überprüfen jQAssistant ist ein Werkzeug zur statischen Analyse von Softwareartefakten. Mit jQAssistant ist es möglich, Architektur- und Designkonzepte als einen Satz von Regeln zu definieren. Die Einhaltung dieser Regeln kann anschließend bei jedem Bau einer Software automatisiert überprüft werden. Dadurch kann sichergestellt werden, dass eine vorgegebene Architektur sowie Designvorgaben auch wirklich im Quellcode umgesetzt wurden. In meinem Vortrag stelle ich die Grundlagen von jQAssistant vor und zeige an einem Beispiel, wie sich mit dem Werkzeug eine sich selbst prüfende Architekturdokumentation erstellen lässt.

Nachvollziehbare, datengetriebene, automatisierte Analysen der Softwareentwic...

Markus Harrer

Session am 17.05.2017, DeveloperCamp 2017, Nürnberg Jeder Schritt in der Entwicklung oder Verwendung von Software hinterlässt wertvolle, digitale Spuren. Die Auswertung dieser „Softwaredaten“ (wie etwa Laufzeitmessungen, Logdateien oder Commits) lässt unser Bauchgefühl zu Fakten und stichhaltigen Beweisen werden. Ich zeige, wie sich Fragen aus der Softwareentwicklung datengetrieben, automatisiert und nachvollziehbar beantworten lassen. Dazu demonstriere ich das Zusammenspiel von Open-Source-Analysewerkzeugen (wie jQAssistant, Neo4j, Pandas und Jupyter) zur Auswertung von Daten aus unterschiedlichen Quellen (wie Profiler, Jenkins und Git). Gemeinsam sehen wir uns u. a. an, wie wir Lösungen zur Optimierung der Performance erarbeiten, Build-Breaker identifizieren oder Wissenslücken im Quellcode offenlegen können.

Einsatzmöglichkeiten der automatisierten Analyse von Artefakten und Metadaten...

Markus Harrer

Erkenntnistheoretische Beurteilung von Extreme Programming

Markus Harrer

English abstract: Agile software development projects fail from time to time for reasons that can be explained neither technically nor functionally. So other aspects must be playing an important reason why those projects fail. In this article, I take another view on agile development practices with the help of the philosophical discipline of epistemology to show the inherent human problems. These problems occur in every methodology where humans try to model real world phenomenon. Further, these epistemological dilemmas can be used to evaluate the problems of methods and procedures used in the software development. I take a deeper look at some of the practices used in Extreme Programming (XP). The goal is to find out what epistemological dilemmas are particularly well treated by XP practices and which ones can be very dangerous for projects using XP practices

An interactive form-based mobile software system with a sample application in...

Markus Harrer

Presentation about my bachelor's thesis back in early 2009. The motivation for this paper was that mobile devices have been entering the workplace at businesses that were previously not supposed to be "digitalized" thus leading to overwhelmed staff and software systems suffering from major acceptance drawbacks. My primary research questions were to find a way to design and realize a software system that can support "interactive forms." Interactive forms are a structured game-like way of entering or gathering data in a computer. They have a well-designed and visually appealing user interface combined with a motivation system (e.g. achievements or high scores). Other goals of my thesis were to create a software development process and a software architecture that support a workflow for the interdisciplinary design of interactive forms. In my thesis I initially investigated and gathered domain-specific requirements for elderly care, which is a (nearly) non-digitized domain, to better understand the problems that a caretaker has to deal with every day. I then analyzed the specific needs of interaction designers and software developers and looked for domain expert centric, iterative development processes that are similar for interaction designers and software developers. This was followed by an examination of existing software architectures that can support most of the requirements previously determined. Finally, I implemented a prototype that demonstrated the feasibility of interactive forms and their underlying software architecture and development process.

Erkenntnistheoretische Beurteilung von Extreme Programming

Markus Harrer

Agile Softwareentwicklungsprojekte scheitern immer wieder aus Gründen, welche weder technisch noch fachlich erklärbar sind. Es müssen also andere Aspekte eine Rolle spielen. Aus der philosophischen Disziplin der Erkenntnistheorie lassen sich menschliche Grundprobleme ableiten, die bei jeder Erstellung von Softwaresystemen auftreten. Diese erkenntnistheoretischen Dilemmata können dazu dienen, verwendete Vorgehen und Methoden in der Softwareentwicklung objektiv nach menschlichen Faktoren zu beurteilen. In dieser Arbeit werden ausgewählte Techniken von Extreme Programming (XP) mit Hilfe der Erkenntnistheorie untersucht. Ziel ist es herauszufinden, welche Dilemmata von XP besonders gut behandelt werden und welche sich gefährlich auf den Projektverlauf auswirken können.

More from Markus Harrer (11)

Datenanalysen in der Softwareentwicklung (IMPROVE Workshop Wien)

Software Analytics - Datenanalysen in der Softwareentwicklung (BigDataMeetup)

Datenanalysen in der Softwareentwicklung mit Software Analytics

Philosophy screws it all up (Pecha Kucha) [Java Forum Stuttgart 2017]

Architektur und Code im Einklang [JUG Nürnberg]

Architektur und Code im Einklang [DeveloperCamp 2017]

Nachvollziehbare, datengetriebene, automatisierte Analysen der Softwareentwic...

Einsatzmöglichkeiten der automatisierten Analyse von Artefakten und Metadaten...

Erkenntnistheoretische Beurteilung von Extreme Programming

An interactive form-based mobile software system with a sample application in...

Erkenntnistheoretische Beurteilung von Extreme Programming

Recently uploaded

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Globus

The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

How Recreation Management Software Can Streamline Your Operations.pptx

wottaspaceseo

Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.

OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam

takuyayamamoto1800

OpenMetadata Community Meeting - 5th June 2024

OpenMetadata

The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features. * How to run your own data quality framework * What is the performance impact of running data quality frameworks * How to run the test cases in your own ETL pipelines * How the Incident Manager is integrated * Get notified with alerts when test cases fail Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E

Top 7 Unique WhatsApp API Benefits | Saudi Arabia

Yara Milbes

Discover the transformative power of the WhatsApp API in our latest SlideShare presentation, "Top 7 Unique WhatsApp API Benefits." In today's fast-paced digital era, effective communication is crucial for both personal and professional success. Whether you're a small business looking to enhance customer interactions or an individual seeking seamless communication with loved ones, the WhatsApp API offers robust capabilities that can significantly elevate your experience. In this presentation, we delve into the top 7 distinctive benefits of the WhatsApp API, provided by the leading WhatsApp API service provider in Saudi Arabia. Learn how to streamline customer support, automate notifications, leverage rich media messaging, run scalable marketing campaigns, integrate secure payments, synchronize with CRM systems, and ensure enhanced security and privacy.

GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)

Alina Yurenko

Developing Distributed High-performance Computing Capabilities of an Open Sci...

Globus

COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.

APIs for Browser Automation (MoT Meetup 2024)

Boni García

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Aftab Hussain

Understanding variable roles in code has been found to be helpful by students in learning programming -- could variable roles help deep neural models in performing coding tasks? We do an exploratory study. - These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia

Vitthal Shirke Microservices Resume Montevideo

Vitthal Shirke

Globus Compute wth IRI Workflows - GlobusWorld 2024

Globus

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

Quarkus Hidden and Forbidden Extensions

Max Andersen

Understanding Globus Data Transfers with NetSage

Globus

NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Cracking the code review at SpringIO 2024

Paco van Beckhoven

Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production. Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process? In this session we will cover: - The Art of Effective Code Reviews - Streamlining the Review Process - Elevating Reviews with Automated Tools By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Google

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-fusion-buddy-review AI Fusion Buddy Review: Key Features ✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini ✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique! ✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs! ✅Fully automated AI articles bulk generation! ✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more. ✅With one keyword or URL, generate complete websites, landing pages, and more… ✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7. ✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches. ✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all! ✅Save over $5000 per year and kick out dependency on third parties completely! ✅Brand New App: Not available anywhere else! ✅ Beginner-friendly! ✅ZERO upfront cost or any extra expenses ✅Risk-Free: 30-Day Money-Back Guarantee! ✅Commercial License included! See My Other Reviews Article: (1) AI Genie Review: https://sumonreview.com/ai-genie-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIFusionBuddyReview, #AIFusionBuddyFeatures, #AIFusionBuddyPricing, #AIFusionBuddyProsandCons, #AIFusionBuddyTutorial, #AIFusionBuddyUserExperience #AIFusionBuddyforBeginners, #AIFusionBuddyBenefits, #AIFusionBuddyComparison, #AIFusionBuddyInstallation, #AIFusionBuddyRefundPolicy, #AIFusionBuddyDemo, #AIFusionBuddyMaintenanceFees, #AIFusionBuddyNewbieFriendly, #WhatIsAIFusionBuddy?, #HowDoesAIFusionBuddyWorks

Recently uploaded (20)

Globus Connect Server Deep Dive - GlobusWorld 2024

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

May Marketo Masterclass, London MUG May 22 2024.pdf

How Recreation Management Software Can Streamline Your Operations.pptx

OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam

OpenMetadata Community Meeting - 5th June 2024

Top 7 Unique WhatsApp API Benefits | Saudi Arabia

GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)

Developing Distributed High-performance Computing Capabilities of an Open Sci...

APIs for Browser Automation (MoT Meetup 2024)

BoxLang: Review our Visionary Licenses of 2024

A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Vitthal Shirke Microservices Resume Montevideo

Globus Compute wth IRI Workflows - GlobusWorld 2024

Quarkus Hidden and Forbidden Extensions

Understanding Globus Data Transfers with NetSage

openEuler Case Study - The Journey to Supply Chain Security

Cracking the code review at SpringIO 2024

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Software Analytics for Pragmatists [DevOps Camp 2017]

1. Software Analytics for Pragmatists Solving problems – automated, data-centric and reproducible Markus Harrer software analytics clean code

2. code

3. “Without data you're just another person with an opinion.“ W. Edwards Deming

4. Motivation Software Analytics “In software engineering there is much we are seeing, but little we are learning.” Tim Menzies

5. State of the Art

6. State of the Art

7. Questions from daily work

8. Motivation Why? •Make problems visible + Improve clarity and understanding •Drive decisions + Raise money + Raise more money for further analysis •Support Continuous Learning + Master challenges + Thrive for improvement steadily

9. Claim Solving problems – automated, data-centric and reproducible.

10. Claim Solving problems – automated, data-centric and reproducible.

11. Automation •Data retrieval •Analysis •Visualization DevOps style

12. Claim Solving problems – automated, data-centric and reproducible.

13. Types of software artifacts and meta data chronologicalcommunity runtimestatic

14. Data is dirty Time for cleaning up data up to 80%

15. Claim Solving problems – automated, data-centric and reproducible.

16. Data + Automation + Traceability Replication

17. Code Book Example: Analyzing performance bottlenecks

18. Claim Solving problems – automated, data-centric and reproducible.

19. Pipeline Data Mining • NumPy • scikit-learn • SciPy Visualization • matplotlib • plot.ly • Bokeh • python-pptx ...

20. Pipeline XML/Graph Tabellen matplotlib Pandas, ... Pandas jQAssistant, Neo4j Text xlsx E pptx P Python Jupyter Input Pre- processing Analysis Output D3

21. Pipeline ZIP GZ *.class JAR, WAR, EAR MANIFEST.MF *.properties XSD YAML XML application.xml web.xml beans.xml JaCoCo FindBugs CheckStyle pom.xml surefire-reports.xml RDBMS Schema M2 Repository DBCSV Excel BigQuery Inputs HDFStore Web JSON Git Pandas jQAssistant

22. Examples Demos •Performance Bottlenecks •Build Breaker Analysis •Mining Knowledge Islands •Git Log Analysis

23. Experience Learning by Doing!

24. What‘s the value of the information related to the effort for the analysis of the information? Always ask the question

25. References Leek, Jeff: The Elements of Data Analytic Style. LeanPub, 2015. McKinney, Wes: Python For Data Analysis, O’Reilly, 2012. Mens, Tom; Serebrenik; Cleve, Anthony:Evolving Software Systems. Springer, 2014. Mens, Tom; Demeyer, Serge: Software Evolution. Springer, 2008. Shull, Forrest; Singer, Janice; Sjøberg, Dag I.K.: Guide to Advanced Empirical Software Engineering. Springer, 2008. Tornhill, Adam:Your Code As a Crime Scene. Pragmatic Programmers, 2015.

26. Your Questions

Software Analytics for Pragmatists [DevOps Camp 2017]

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Software Analytics for Pragmatists [DevOps Camp 2017]

Similar to Software Analytics for Pragmatists [DevOps Camp 2017] (20)

More from Markus Harrer

More from Markus Harrer (11)

Recently uploaded

Recently uploaded (20)

Software Analytics for Pragmatists [DevOps Camp 2017]