The document discusses analyzing GitHub commit data with R. It covers obtaining GitHub data from sources like the GitHub Archive and API, exploring and processing the data with R basics and libraries, and visualizing the results. The agenda includes an introduction to R, data exploration and processing techniques in R like handling different data sources and formats, and how to deal with big data using tools that connect R to Hadoop. Examples demonstrate retrieving active repository data, processing multiple files, plotting, and detecting anomalies.

1. Analysing GitHub commits with R
Barbara Fusinska
@BasiaFusinska
barbarafusinska.com

2. About me
Programmer
Manager
Sweet tooth

3. Goals
Analysing
GitHub data
Methods of data
exploration & analysis
Learn R

4. Agenda
• Data analysis
• R basics
• Data exploration & processing
• Big Data
• Outside GitHub:
– Data visualisation
– Libraries & tools

5. Data analysis process
Raw Data
Processed
Data
Data Analysis
& Visualization
Exploratory
Data Analysis

6. What do we want to find out?

7. GitHut Visualisation
http://githut.info/

8. GitHub Archive
https://www.githubarchive.org/

9. Google BigQuery
https://cloud.google.com/bigquery/what-is-bigquery

10. GitHub API
https://developer.github.com/v3/

11. DEMO: R basics
• RStudio
• Data types
• Data filtering

12. DEMO: Active repositiories
• What is an active repository?
• Where can we get data from?
• How to get useful information?
• How to handle missing data?

13. GitHub Archive CreateEvent

14. GitHub Archive PushEvent

15. GitHub Archive PullRequestEvent

16. DEMO: Processing various data
sources
• Active repositories – from Create, Push and
PullRequest events
• Missing language information

17. Big Data in R
• What’s _Big_Data_ anyway?
• R processes data in memory
• Bring down only the data
you need
• Streaming the data from
database

18. R, Hadoop, and How They Work
Together
• Hadoop Streaming – utilities available as R scripts
• ORCH (Oracle R Connector for Hadoop) - provides access to a
Hadoop cluster from R
• RHIPE (R and Hadoop Integrated Programming Environment)
– techniques designed for analysing large sets of data
• Rhadoop - R packages that allow users to manage and analyse
data with Hadoop
https://blog.udemy.com/r-hadoop/

19. Azure Machine Learning Studio
http://channel9.msdn.com/Blogs/Windows-Azure/R-in-Azure-ML-Studio

20. DEMO: Processing multiple files
• Reading files line by line
• Reading multiple files

21. DEMO: Plotting in R
http://www.harding.edu/fmccown/r/

22. Libraries & Tools
• DataScience
• JSON processing
• Http requests
• Summarising
• Plotting

23. DEMO: Downsampling
https://github.com/javiljoen/LTTB

24. DEMO: Anomalies detection
https://github.com/twitter/AnomalyDetection

25. To summarize…
• GitHub Archive, GitHub API
• R language – RStudio, types, data exploration,
I/O operations, etc.
• Big Data in R
• 3rd party & build-in libraries
• Visualization

26. Thank you
Barbara Fusinska
@BasiaFusinska
barbarafusinska.com
https://github.com/BasiaFusinska/RTalk