Presentation given at VoxxedDays Vienna 2016

1. Getting started with R when
analysing GitHub events
Barbara Fusinska
barbarafusinska.com

2. About me
Programmer
Math enthusiast
Sweet tooth
@BasiaFusinska
https://github.com/BasiaFusinska/RTalk

3. Agenda
• R ecosystem
• R basics
• Analysing GitHub events
• Data sources
• Code… a lot of code

4. Why R?
• Ross Ihaka & Robert Gentleman
• Name:
• First letter of names
• Play on the name of S
• S-PLUS – commercial alternative
• Open source
• Nr 1 for statistical computing

5. R Environment
• R project
• console environment
• http://www.r-project.org/
• IDE
• Any editor
• RStudio
http://www.rstudio.com/products/rstudio/download/

6. RStudio
Editor
Console
Environment
variables
Plots
Files
Help
Packages

7. R Basics

8. Basics - Types
> myChar <- "a"
> myChar
[1] "a"
> typeof(myChar)
[1] "character"
> myNum <- 10
> myNum
[1] 10
> typeof(myNum)
[1] "double"
> # Dynamic
> myNum <- "some text"
> typeof(myNum)
[1] "character"

9. Vectors
> myVector <- c("a", "b", "c")
> myVector
[1] "a" "b" "c"
> typeof(myVector)
[1] "character"
myVector <- 1:10
myVector <- double(0)
myVector <- c(2, 5:10, 20)
myVector <- letters[1:5]
myVector[5]

10. Lists
> myList <- list("a", "b", "c")
> myList
[[1]]
[1] "a"
[[2]]
[1] "b"
[[3]]
[1] "c"
> typeof(myList)
[1] "list"

11. Named elements
> myVector <- c(a="a", b="b", c="c")
> myVector
a b c
"a" "b" "c"
> myList <- list(a="a", b="b", c="c")
> myList
$a
[1] "a"
$b
[1] "b"
$c
[1] "c"

12. Accessing element
> myVector[1]
a
"a"
> myVector[[1]]
[1] "a"
> myVector['a']
a
"a"
> myVector[['a']]
[1] "a"
> myList[1]
$a
[1] "a"
> myList[[1]]
[1] "a"
> myList['a']
$a
[1] "a"
> myList[['a']]
[1] "a"
> myList$a
[1] "a"

13. Data frames
> dataFrame <- data.frame(col1=c(1,2,3), col2=c(4,5,6))
> dataFrame
col1 col2
1 1 4
2 2 5
3 3 6
> typeof(dataFrame)
[1] "list"

14. Summary
> summary(dataFrame)
col1 col2
Min. :1.0 Min. :4.0
1st Qu.:1.5 1st Qu.:4.5
Median :2.0 Median :5.0
Mean :2.0 Mean :5.0
3rd Qu.:2.5 3rd Qu.:5.5
Max. :3.0 Max. :6.0

15. Summary statistics
mean(dataFrame$col1)
max(dataFrame$col1)
min(dataFrame$col1)
sum(dataFrame$col1)
median(dataFrame$col1)
quantile(dataFrame$col1)

16. Filtering vectors and lists
> a <- 1:10
> a[a > 4]
[1] 5 6 7 8 9 10
> select <- function(x) { x > 4}
> a[select(a)]
[1] 5 6 7 8 9 10
> Filter(select, a)
[1] 5 6 7 8 9 10

17. Filtering data frames
dataFrame <- data.frame(
age=c(20, 15, 31, 45, 17),
gender=c('F', 'F', 'M', 'M', 'F'),
smoker=c(TRUE, TRUE, FALSE, TRUE, FALSE))
> dataFrame
age gender smoker
1 20 F TRUE
2 15 F TRUE
3 31 M FALSE
4 45 M TRUE
5 17 F FALSE

18. Filtering by rows
> dataFrame$age[
dataFrame$gender == 'F']
[1] 20 15 17
> dataFrame[2:4, ]
age gender smoker
2 15 F TRUE
3 31 M FALSE
4 45 M TRUE
> dataFrame[
dataFrame$age < 30, ]
age gender smoker
1 20 F TRUE
2 15 F TRUE
5 17 F FALSE
> dataFrame[
dataFrame$gender == 'M', ]
age gender smoker
3 31 M FALSE
4 45 M TRUE

19. Filtering by columns
> dataFrame[, 3]
[1] TRUE TRUE FALSE TRUE FALSE
> dataFrame[, c(1,3)]
age smoker
1 20 TRUE
2 15 TRUE
3 31 FALSE
4 45 TRUE
5 17 FALSE
> dataFrame[, c(3,2)]
smoker gender
1 TRUE F
2 TRUE F
3 FALSE M
4 TRUE M
5 FALSE F
> dataFrame[, c('age', 'smoker')]
age smoker
1 20 TRUE
2 15 TRUE
3 31 FALSE
4 45 TRUE
5 17 FALSE

20. Goal: Language distribution

21. https://www.githubarchive.org/

22. Google BigQuery

23. Language information
• Only Pull Requests event types
have language information
• Data source – 1h events from
01.01.2015 3 PM
• ~11k events
• ~500 pull requests

24. Gender bias?
• 4,037,953 GitHub user
profiles
• 1,426,121 identified
(35.3%)
http://arstechnica.com/information-technology/2016/02/data-analysis-
of-github-contributions-reveals-unexpected-gender-bias/
Open Closed
Women 8,216 111,011
Men 150,248 2,181,517

26. Reading data from files - csv
> sizes <- read.csv(sizesFile)
> sizes
category length width
1 B 20.0 3.0
2 A 23.0 3.6
3 B 75.0 18.0
4 B 44.0 10.0
5 C 2.5 6.0
6 B 7.2 27.0
7 A 45.8 34.0
8 C 12.0 2.0
9 A 5.0 13.0
10 A 68.0 14.5

27. Reading data from files - lines
> lines <- readLines(sizesFile)
> lines
[1] "category,length,width" "B,20,3"
[3] "A,23,3.6" "B,75,18"
[5] "B,44,10" "C,2.5,6"
[7] "B,7.2,27" "A,45.8,34"
[9] "C,12,2" "A,5,13"
[11] "A,68,14.5"

28. Writing data to csv file
write.csv(sizes, file=outputFile)
write.csv(sizes, file=outputFile, row.names = FALSE)

29. Applying operation across elements
> myVector <- c(1, 4, 9, 16, 25)
> sapply(myVector, sqrt)
[1] 1 2 3 4 5
> lapply(myVector, sqrt)
[[1]]
[1] 1
[[2]]
[1] 2
[[3]]
[1] 3
[[4]]
[1] 4
[[5]]
[1] 5

30. Read GitHub Archive events
library("rjson")
readEvents <- function(file, eventNames) {
lines <- readLines(file)
jsonEvents <- lapply(lines, fromJSON)
specificEvents <- Filter(
function(e) { e$type %in% eventNames },
jsonEvents)
return(specificEvents)
}

31. Missing data
# Missing values
> a <- c(1,2,NA,3,4,5)
> a
[1] 1 2 NA 3 4 5
# Checking if missing data
> is.na(a)
[1] FALSE FALSE TRUE FALSE
FALSE FALSE
> anyNA(a)
[1] TRUE
# Setting missing values
> is.na(a) <- c(2,4)
> a
[1] 1 NA NA NA 4 5
# Setting null values
> a <- NULL
> is.null(a)
[1] TRUE

32. Read pull requests
pullRequestEvents <- readEvents(fileName,"PullRequestEvent")
select <- function(x) {
id <- x$payload$pull_request$base$repo$id
language <- x$payload$pull_request$base$repo$language
if (!is.null(language)) {
c(ID=id, Language=language)
} else {
c(ID=id, Language="")
}
}
pullRequests <- sapply(pullRequestEvents, select)

33. Some solutions
for(x in pullRequests) {
# version 1
rbind(dataFrame, x)
#version 2
idColumn <- c(idColumn, x[“ID”,])
languageColumn <- c(languageColumn, x[“Language”,])
}
# version 2
dataFrame <- data.frame(
id=idColumn,
language=languageColumn)

34. Prepare data
reposLanguages <- data.frame(
id=pullRequests["ID",],
language=pullRequests["Language",])
head(reposLanguages)
summary(reposLanguages)

35. Little look on the data
> head(reposLanguages)
id language
1 3542607 C++
2 10391073 Python
3 28668460 Python
4 28608107 Ruby
5 5452699 JavaScript
6 19777872 C#
> summary(reposLanguages)
id language
28648149: 12 Ruby : 66
28688863: 8 PHP : 55
20413356: 5 Python : 53
28668553: 5 : 51
10160141: 4 JavaScript: 47
206084 : 4 C++ : 30
(Other) :436 (Other) :172

36. Duplicated data
> myData <- c(1,2,3,4,3,2,5,6)
> duplicated(myData)
[1] FALSE FALSE FALSE FALSE TRUE TRUE FALSE FALSE
> anyDuplicated(myData)
[1] 5
> unique(myData)
[1] 1 2 3 4 5 6

37. Unique repositories data
> reposLanguages <- unique(reposLanguages)
> summary(reposLanguages)
id language
25994257: 2 Python : 36
28528325: 2 JavaScript: 35
10126031: 1 Ruby : 35
10160141: 1 PHP : 34
10344201: 1 : 27
10391073: 1 Java : 22
(Other) :297 (Other) :116

38. Distribution tables
> collection <-
c('A','C','B','C','B','C')
> oneWayTable <-
table(collection)
> oneWayTable
collection
A B C
1 2 3
> attributes(oneWayTable)
$dim
[1] 3
$dimnames
$dimnames$collection
[1] "A" "B" "C"

39. Language distribution
> languages <- table(reposLanguages$language)
> head(languages)
ActionScript Bluespec C
27 1 1 9
C# C++
11 20
> languages <- sort(languages, decreasing=TRUE)
> head(languages)
Python JavaScript Ruby PHP
36 35 35 34 27
Java
22

40. Recognised languages
reposLanguages <-
reposLanguages[reposLanguages$language != "",]
languages <- table(reposLanguages$language)
languages <- sort(languages, decreasing=TRUE)

41. Language names
> languagesNames <- names(languages)
> languagesNames
[1] "Python" "JavaScript" "Ruby"
[4] "PHP" "Java" "C++"
[7] "CSS" "C#" "C"
[10] "Go" "Shell" "CoffeeScript”
[13] "Objective-C" "Puppet" "Scala"
[16] "Lua" "Rust" "Clojure"
[19] "Emacs Lisp" "Haskell" "Julia"
[22] "Makefile" "Perl" "VimL"
[25] "ActionScript" "Bluespec" "DM"
[28] "Elixir" "F#" "Haxe"
[31] "Matlab" "Swift" "TeX"
[34] ""

42. Plotting languages2Display <- languages[languages > 5]
barplot(languages2Display)

43. Summary
• GitHub Archive
• Introduction to R
• Data types
• Filtering
• I/O
• Applying operations
• Missing values & duplicates
• Binding data
• Distribution tables
• Plotting (barplot)

44. Thank you
barbara.fusinska@gmail.com
@BasiaFusinska
barbarafusinska.com
https://github.com/BasiaFusinska/RTalk
Questions?