Presentation given at CodeMash 2016

1. Analysing GitHub commits with R
@BasiaFusinska
barbarafusinska.com
barbara.fusinska@gmail.com

2. About me
Programmer
Math enthusiast
Sweet tooth
@BasiaFusinska
• @BasiaFusinska
• barbarafusinska.com
• barbara.fusinska@gmail.com
https://github.com/BasiaFusinska/RTalk

3. Goals

4. Agenda
• Data analysis
• Capturing GitHub data
• Using R in Data Analysis
• Azure ML

5. Data analysis process
Raw Data
Processed
Data
Data Analysis
& Visualization
Exploratory
Data Analysis
Data
Capture

6. What do we want to find out?

7. Language statistics

8. GitHut Visualisation
http://githut.info/

9. Data Capture

10. GitHub Archive
https://www.githubarchive.org/

12. Event Data
Data Source
(Events/Logs/Files)
Store & reorganize
Query

13. Google BigQuery
https://cloud.google.com/bigquery/what-is-bigquery

14. GitHub API
https://developer.github.com/v3/

15. Why R?
• Ross Ihaka & Robert Gentleman
• Name:
– First letter of names
– Play on the name of S
– S-PLUS – commercial alternative
• Open source
• Nr 1 for statistical computing

16. Development environment

17. R Environment
• R project
– console environment
– http://www.r-project.org/
• IDE
– Any editor
– RStudio
http://www.rstudio.com/products/rstudio/download/

18. RStudio
Editor
Console
Environment
variables
Plots
Files
Help
Packages

19. R Basics

21. Filtering

23. Goal: Language distribution

25. Distribution of active repositories per
language

26. What is an active repository?

27. Source of truth

28. GitHub Archive CreateEvent

29. GitHub Archive PushEvent

30. GitHub Archive PullRequestEvent

32. Task: Reading Pull Requests
1. Read the file line by line and
extract only pull request events
2. Extract id and language
information
3. Count and visualise language
distribution
Data: 1h GitHub Archive Events
from 01-01-2015, 3 PM

33. Reading Events

34. Read Pull Requests

35. Unique data

36. Language information

37. Language information output

39. Missing information

40. Omitting information

41. Plotting

42. Now everything is sorted…

43. Goal: Analyze ACTUAL active
repositories

44. Missing data

45. Language information
• Active repositories – Create, Push and
PullRequest events
• Missing language information:
– Google BigQuery
– GitHub API
• Process various data sources

46. Google BigQuery

47. Different sources of data
• GitHub Archive:
– id,
– url in a form:
https://api.github.com/repos/:name
– (rare cases) language
• Google BigQuery:
– no id,
– url in a form:
https://github.com/:name
– language

48. Task: Reading Active Repositories
1. Read the file line by line and extract only
create, push and pull request events
2. Extract id and url information
3. Read Google BigQuery data from saved file
4. Combine repositories data and Google data
base on the same url and fill in missing
language information
5. Count and visualise language distribution

49. Read GitHub Archive

50. Read Google Data

51. Repository data

52. Various data sources

53. Combining data

54. Sorted data

55. Active repositories per languages

56. Goal: Using GitHub API

57. Task: Retrieve language info for
repository
• GET
/repos/:owner/:repo/languages
• Owner: BasiaFusinska
• Repo: RWorkshop

58. GitHub API from R

59. Task: Calling GitHub Search
• GET
/search/repositories
• Querying: q parameter
• Paging: page parameter

60. GitHub API Search

61. Digression

62. Data…

63. Big Data in R
• What’s _Big_Data_ anyway?
• R processes data in memory
• Bring down only the data
you need
• Streaming the data from
database

64. Azure Machine Learning Studio

65. Azure ML R Script

66. Datasets

67. Reading pull requests experiments

68. Analyzing the whole day

69. R Script

70. Pull requests - visualization

71. To summarize…
• Data science – not a rocket science, shaving
the yak
• Different sources – different truths
• Capturing & storing data
• Data science UI – visualization is the key
• Desktop - hypothesis, development
• Cloud – production

72. What’s next?
• Data Exploration in R, workshop
• basiafusinska.com, blog
• katacoda.com, interactive learning platform

73. Thank you
barbara.fusinska@gmail.com
@BasiaFusinska
barbarafusinska.com
https://github.com/BasiaFusinska/RTalk