Data has always been used in every company irrespective of its domain to improve the operational efficiency and the products themselves. However, analyzing and extracting information from “Big Data” is the next revolution in technology, since previously unknown nuggets of information are now made visible. In fact, over 90% of the data available in the world has been generated in the last two years. “Big Data” analytics has become the next hot topic for most companies - from financial institutions to technology companies to service providers. Likewise in software engineering, data collected about the development of software, the operation of the software in the field, and the users feedback on software have been used before. However, collecting and analyzing this information across hundreds of thousands or millions of software projects gives us the unique ability to reason about the ecosystem at large, and software in general. At no time in history has there been easier access to extremely powerful computational resources as it is today, thanks to the advances in cloud computing, both from the technology and business perspectives. Therefore, it is easier today than ever before to analyze big data. In this technical briefing, we will present the state-of-the-art with respect to the research carried out in the area of big data analytics in software engineering research. We will present the research along three dimensions: 1) What are the software engineering problems being solved? Examples of problems include: How much source code is newly written and how much is reused from past projects? Can we recommend best practices to developers by observing the development of software among hundreds of thousands of software projects? 2) What are the datasets that are being used? Examples of my datasets include: all the mobile apps in the Google Play store, all of the world's Open Source projects, and hundreds of gigabytes of execution logs. Such large datasets provide us with a unique view into the SE field. 3) What are the tools and techniques available to analyze the large datasets? We intend to present generic software solutions that have been applied to big datasets in other areas of research, and the tools and techniques created by software engineering researchers. In the end we will present the challenges inherently present in large datasets - volume, variety, velocity, and veracity. Such challenges often complicate the analysis of the data and can invalidate the interpretation of the results. We will conclude with the future opportunities that are present in big data analytics for software engineering research.

1. Big(ger) Data
Open Source
Software
Github
Apache
Sourceforge
App Store Data
App Store
Google
App
Windows
store
Execution
Logs
Amazon
Microsoft
Stackoverflow
TopCoder
Big(ger) Data in Software Engineering
Meiyappan Nagappan, Mehdi Mirakhorli
Rochester Institute of Technology

2. Nagappan & Mirakhorli ICSE 2015
Meiyappan Nagappan
Dept of Software Engineering
Rochester Institute of Technology
mei@se.rit.edu
http://mei-nagappan.com
Mehdi Mirakhorli
Dept of Software Engineering
Rochester Institute of Technology
mehdi@se.rit.edu
http://www.se.rit.edu/~mehdi/
Speakers

3. Our Research Collaborations
Nagappan & Mirakhorli ICSE 2015

4. Nagappan & Mirakhorli ICSE 2015

5. Sam Malek
George Mason
University
Rick Kazman
SEI
Yuanfang Cai
University of Drexel
Patrick Maeder
University of Illmenue
Bob Hanmer
Alcatel Lucent
Muhammad Ali Babar
University of Adelaide
Robert L. Nord
SEI
Jane Cleland-Huang
DePaul University
Nagappan & Mirakhorli ICSE 2015

6. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Agenda
Nagappan & Mirakhorli ICSE 2015

7. What are you passionate about?

8. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Nagappan & Mirakhorli ICSE 2015

9. BIG DATA in SE
Nagappan & Mirakhorli ICSE 2015

10. BIG(ger) DATA in SE
Nagappan & Mirakhorli ICSE 2015

11. BIG(ger) DATA in SE
SE Datasets that are several orders of
magnitude BIGGERNagappan & Mirakhorli ICSE 2015

12. All Android Apps in Google Play
Nagappan & Mirakhorli ICSE 2015

13. So Why BIG(ger) Data in SE Now?
Nagappan & Mirakhorli ICSE 2015

14. Access to Data
Why BIG(ger) Data in SE Now?
Nagappan & Mirakhorli ICSE 2015

15. Computing PowerAccess to Data
Why BIG(ger) Data in SE Now?
Nagappan & Mirakhorli ICSE 2015

16. But, Big DATA => Big CHALLENGES
Nagappan & Mirakhorli ICSE 2015

17. But, Big DATA => Big CHALLENGES
Volume
Nagappan & Mirakhorli ICSE 2015

18. But, Big DATA => Big CHALLENGES
Volume
Velocity
Nagappan & Mirakhorli ICSE 2015

19. But, Big DATA => Big CHALLENGES
Volume
Velocity
Variety
Nagappan & Mirakhorli ICSE 2015

20. But, Big DATA => Big CHALLENGES
Volume
Veracity/ NoiseVelocity
Variety
Nagappan & Mirakhorli ICSE 2015

21. But why should SE Research
adopt BIG(ger) Data?
Nagappan & Mirakhorli ICSE 2015

22. World of Code
Nagappan & Mirakhorli ICSE 2015

23. FSE13: Extracted Case Study Subject
Systems from All Research Papers
Nagappan & Mirakhorli ICSE 2015

24. TLOC
Language Type
# Dev
Churn
#
Commits
AgeActivity
Diversity in ICSE/FSE
Nagappan & Mirakhorli ICSE 2015

25. World of Code
What Area
do SE
Studies
Cover?
Nagappan & Mirakhorli ICSE 2015

26. Patterns:
 Data brings
knowledge
 Can you find new
patterns?
Why is Big(ger) Data useful?
Generalizability:
A limited set of
projects examined.
Results are valid in
the context
Nagappan & Mirakhorli ICSE 2015

27. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Nagappan & Mirakhorli ICSE 2015

28. 100’s of GBs of Execution Logs per Day
App Store Data
100+ 300K+6.9M+
All the Open Source Projects in
the World
Crowd sourced Data
Datasets
Nagappan & Mirakhorli ICSE 2015

29. Sourcerer provides a collection of tools for automated
crawling, parsing and fingerprinting of open source applications.
Sourcerer
Repositories: Apache, Java.net, Google
Code and Sourceforge.
Collected Info:
– Versioned source code across multiple
releases
– documentation(if available)
– Projects’ metadata
– a coarse-grained structural analysis of
each project.
Size: Over 20,000
open source systems.
Download:
http://www.ics.uci.ed
u/~lopes/datasets/
lopes@ics.uci.edu
Sushil Bajracharya, Joel Ossher, and Cristina Lopes. 2014. Sourcerer: An infrastructure for large-scale
collection and analysis of open-source code. Sci. Comput. Program. 79 (January 2014), 241-259.
Usage data of Koders.com
sourcerer-maven-aug12 containing 2,232 projects from the
Maven Central repository (~80GB).
Nagappan & Mirakhorli ICSE 2015

30. Domain-specific language and infrastructure for software
repository mining.
Boa
• Boa project has collected
source code of 23K java
projects (only subversion)
• Meta-data of 600K
projects.
• Offers a domain specific
language to query the data,
it is primarily useful for
replicating the existing
research where the
concepts are known and
well understood
Nagappan & Mirakhorli ICSE 2015

31. Ghtorrent
Github - http://ghtorrent.org/
Create a scalable, queriable, offline mirror of data
offered through the Github REST API.
Every two months, the project releases the collected data.
Nagappan & Mirakhorli ICSE 2015

32. Apache
Apache - http://svn-dump.apache.org/
Download 250 Apache Projects, in 24
categories (domain)
Nagappan & Mirakhorli ICSE 2015

33. TeraPromise
https://terapromise.csc.ncsu.edu/
http://openscience.us/repo/ Nagappan & Mirakhorli ICSE 2015

34. 2014 2015 2016
TeraPromise
Oursummary. Andotherrelatedbooks
The MSR
community
and others
Perspective on
Data Science
for Software
Engineering
Tim Menzies
Laurie Williams
Thomas
Zimmermann
Nagappan & Mirakhorli ICSE 2015

35. StackOverflow
http://blog.stackoverflow.com/2014/01/stack-exchange-cc-
data-now-hosted-by-the-internet-archive/
http://2013.msrconf.org/challenge.php#challenge_data
anonymized dump of all user-contributed Stack
Exchange content since 2009.
Nagappan & Mirakhorli ICSE 2015

36. Google Play
Nagappan & Mirakhorli ICSE 2015

37. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Nagappan & Mirakhorli ICSE 2015

38. Hadoop
Nagappan & Mirakhorli ICSE 2015

39. Who uses Hadoop?
40
Amazon
Facebook
Google
IBM
Intel Research
Joost
Last.fm
New York Times
PowerSet
Veoh
Yahoo!
Nagappan & Mirakhorli ICSE 2015

40. Hadoop’s Major Subsystems
• HDFS is designed for
large, streaming reads
of files.
• Files in HDFS are write
once.
Nagappan & Mirakhorli ICSE 2015

41. 1. Read: Sequentially read a lot of data
2. Map: Extract something you care about
3. Group by key: Sort and Shuffle
Map-Reduce Example
Depending on
the problem,
you only define
map and
reduce
functions.
4. Reduce: Aggregate,
summarize, filter or
transform
5. Write the result
Nagappan & Mirakhorli ICSE 2015

42. Data-Mining Libraries
A framework for building scalable
algorithms, many new Scala +
Spark (H2O in progress)
algorithms, and Mahout's mature
Hadoop MapReduce algorithms.
Dimensionality Reduction
Singular Value Decomposition
Lanczos Algorithm
Stochastic SVD
PCA
Nagappan & Mirakhorli ICSE 2015

43. Data-Mining Libraries
Parallel Computing Toolbox™ supports
solving computationally and data-intensive
problems using multicore processors, GPUs,
and computer clusters.
http://it.mathworks.com/pr
oducts/parallel-computing/
Mr.LDA is an open-source package for
flexible, scalable, multilingual topic modeling
using variational inference in MapReduce.
http://arxiv.org/pdf/1502.07989
v1.pdf
Collection of Different Statistical Methods
and Computing for Big Data
Mr.LDA
https://github.com/lintool/Mr.
LDA
Rhadoop
https://github.com/Revolution
Analytics/RHadoop/
A collection of R packages that allow users to
manage and analyze data with Hadoop.
Nagappan & Mirakhorli ICSE 2015

44. Data-Mining Libraries
Is Hadoop THE SOLUTION?
Nagappan & Mirakhorli ICSE 2015

45. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Nagappan & Mirakhorli ICSE 2015

46. SE problems using Big Data
(to name a few)
Nagappan & Mirakhorli ICSE 2015

47. Big Data Analytics Applications
Assisting Developers of Big Data Analytics Applications
When Deploying on Hadoop Clouds
Code Evolution Analysis
Clone Detection
Log Analysis
Nagappan & Mirakhorli ICSE 2015

48. Mobile Apps
API Change and
Fault Proneness:
A Threat to
Success of
Android Apps
An Examination
of the Current
Rating System
used in Mobile
App Stores
On the
Relationship
between the
Number of Ad
Libraries in an
Android App and
its Rating
Nagappan & Mirakhorli ICSE 2015

49. Programming Languages
A Large-Scale
Empirical Study of
the Relationship
Between Build
Technology and
Build Maintenance
A large scale
study of
programming
languages and
code quality in
github
An empirical
study of goto in C
code
Nagappan & Mirakhorli ICSE 2015

50. Big(ger) Data Analysis in
Requirement Engineering Domain
On-demand Feature Recommendations Derived from
Mining Public Product Descriptions
Nagappan & Mirakhorli ICSE 2015

51. Big(ger) Data Analysis in
Requirement Engineering Domain
On-demand Feature Recommendations Derived from
Mining Public Product Descriptions
Nagappan & Mirakhorli ICSE 2015

52. Big(ger) Data Analysis in
Requirement Engineering Domain
On-demand Feature Recommendations Derived from
Mining Public Product Descriptions
Nagappan & Mirakhorli ICSE 2015

53. Big(ger) Data Analysis in Software
Architecture Domain
Variability Points and Design Pattern Usage in
Architectural Tactics
Learn from millions of
open source developers.
How to implement high
level design decision
(fault detection) using low
level implementation
techniques (design
patterns)?
Nagappan & Mirakhorli ICSE 2015

54. Big(ger) Data Analysis and Rapid
Development
Nagappan & Mirakhorli ICSE 2015

55. Why Big(ger) Data
in software
engineering
• Introduction
• Defining concepts
• One Minute
Madness activity
State of art in
empirical SE and
large datasets
• Summary
• Advances
• Challenges
Public datasets
• Repositories
• Properties
• Accessibility
Tools and
techniques to
analyze large
datasets
• Infrastructure
• Languages
• Techniques
• Example
Agenda for the Technical Briefing of Big(ger) Data in Software Engineering
Nagappan & Mirakhorli ICSE 2015

56. Our Research Manifesto
Developer Maintainer
Operator Manager
Assist various Stakeholders to
build better SoftwareNagappan & Mirakhorli ICSE 2015