This is the SeCold presentation at MSR 2012 Conference. More info at secold.org Paper Title: A Linked Data Platform for Mining Software Repositories Paper Abstract: The mining of software repositories involves the extraction of both basic and value-added information from existing software repositories. The repositories will be mined to extract facts by different stakeholders (e.g. researchers, managers) and for various purposes. To avoid unnecessary pre-processing and analysis steps, sharing and integration of both basic and value-added facts are needed. In this research, we introduce SeCold, an open and collaborative platform for sharing software datasets. SeCold provides the first online software ecosystem Linked Data platform that supports data extraction and on-the-fly inter-dataset integration from major version control, issue tracking, and quality evaluation systems. In its first release, the dataset contains about two billion facts, such as source code statements, software licenses, and code clones from 18 000 software projects. In its second release the SeCold project will contain additional facts mined from issue trackers and versioning systems. Our approach is based on the same fundamental principle as Wikipedia: researchers and tool developers share analysis results obtained from their tools by publishing them as part of the SeCold portal and therefore make them an integrated part of the global knowledge domain. The SeCold project is an official member of the Linked Data dataset cloud and is currently the eighth largest online dataset available on the Web.

1. A Linked Data Platform for
Mining Software Repositories
Iman Keivanloo
Christopher Forbes
Aseel Hmood
Mostafa Erfani
Christopher Neal
George Peristerakis
Juergen Rilling
MSR 2012 June 2

2. SeCold is a “Wikipedia of source code
related facts” produced from over
1,000,000 open source projects.
SeCold main objectives:
(1) establish the fundamental framework
(2) perform data analysis
SeCold 2.0 is an ongoing research project
(currently in its second year)
MSR 2012 2

3. Software Analysis Story
Issue Tracker
Source Code
Mailing List
Versioning Control Some output
…
Some analysis
MSR 2012 3

4. Software Analysis Story
Issue Tracker
Source Code
Mailing List
Versioning Control Some output
…
Structured
Extraction Internal Data
Process Analysis Process
Raw Representation Structured
Data Output
[Source Code Analysis: A Roadmap, FOSE’07]
MSR 2012 4

5. Issue Tracker
Source Code
Mailing List
Versioning Control
…
Sharing
[Source code analysis: a roadmap, FOSE’07]
[Fostering synergies: how … ICSE-SUITE’10]
MSR 2012 5

6. Integration
Alignment
Internal Analysis Output
Data Process
Inter-dataset Analysis
Issue Tracker Internal Analysis Output
Data Process
Source Code
Mailing List
Versioning Control
… Internal Analysis Output
Data Process
Internal Analysis Output
Data Process
MSR 2012 6

7. How to align?
The Challenge
Dataset A Dataset B
MSR 2012 7

8. History of Data Sharing
8

9. Linked Data is about being …
Online a URL for each fact!
Standard uses HTTP, XML, HTML and …
Open usable for both human and machines
NOT Static data and schema are editable
Graph-based graph of triples vs. XML (tree)
Integrating integrated/linked on the fly
MSR 2012 9

10. A Linked Data Platform for
SeCold Project
Mining Software Repositories
1- Vocabulary Set
(aka Schema, Data Model, Ontology)
Source Code Ecosystem Ontology Family (SECON)
SOCON, VERON, METON, ISSUEON, LICENSON, CLON
MSR 2012 10

11. A Linked Data Platform for
SeCold Project
Mining Software Repositories
2- URL/ID Generation Schema
A URL for each piece of fact (e.g. var. def. stmt)
http://aseg.cs.concordia.ca/secold/page/type/java/DatasetChangeInfo
Integration Challenge
Several ways to generate URLs (e.g. random )
REPRODUCIBLE IDENTIFIERS
MSR 2012 11

12. A Linked Data Platform for
SeCold Project
Mining Software Repositories
3- Baseline Data Publication
General Information ( ~2,000,000 triples)
Source Code (~2,000,000,000 triples)
Issue Tracker ( ~30,000,000 triples)
Version Control ( ~700,000,000 triples)
~1 MILLION PROJECTS
MSR 2012 12

13. SeCold
LinkedData Cloud (LOD)
SeCold:
Among the 9 largest
Media datasets in the cloud
Publication
Triple
Circle size
count
Government Very large >1B
Large 1B-10M
Medium 10M-500k
Small 500k-10k
Life Science Very small <10k
[Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/, as of Sept 2011]
MSR 2012 13

14. secold.org
14

15. Showcase #1 (Similar Code Search)
MSR 2012 15

16. Showcase #2 –Part1 (Copyright violation detection)
Se Clone [SeClone … ICPC’11& WCRE’11]
Line level fingerprints
Clone (Type 1,2 and 3)
Internal Analysis Output
Data Process
Source Code of
25K projects Upload
Ninka [A sentence-matching …, ASE’10]
License per file
Internal Analysis Output
Data Process
MSR 2012 16

17. Showcase #2 –Part2 (Copyright violation detection)
e … ICPC’11& WCRE’11]
Line level fingerprints
Clone (Type 1,2 and 3)
s Output
Copyright violation detection:
select ?fileA ?fileB where {
Upload ?fileA testxi ?fingerprint .
?fileB testxi ? fingerprint .
?fileA hasLicense ?la .
?fileB hasLicense ?lb .
-matching …, ASE’10]
Filter (?la != ?lb) }
License per file
Output
MSR 2012 17

18. Showcase #3 (Statistical Analysis)
Apache 2, 9.70%
2009 GPL
2, 12%
LGPL 2.1, 8.80%
BSD, 3% PHP, 0.08% Sleepycat, 0.06%
Mozilla PL 1.1, 0.13%
Mozilla PL 1.0, 2.60% Artistic, 0.02%
All Rights
MIT, 0.92% Nokos, 0.01%
Reserved, 13%
Shareware, 0.00%
Apache 1, 0.65%
No License , 46% Other, 0.00568
Patented, 0%
BSD, 0.27%
2012 Apache 2
9%
All Rights Mozilla PL 1.1
Nokos
Reserved 0%
LGPL 2.1 0%
14%
12% BSD PHP
3% Apache 1 0%
GPL 2 Mozilla PL 1.0 0% Sleepycat
17% 1% 0%
Other MIT Artistic
1% 0% 0%
Shareware
No License 0%
42% Patented
0%
MSR 2012 18

19. MSR 2012 19