R packages for importing spreadsheets and complex tables: tidyxl and unpivotr. Presented to Edinbr, the Edinburgh R User Group on 19 October 2016. By Duncan Garmonsway. https://github.com/nacnudus/tidyxl, https://github.com/nacnudus/unpivotr.

1. Unpivot the impossible
tidyxl & unpivotr
Duncan Garmonsway
@nacnudus

5. But seriously, check out data.linz.govt.nz

9. Take-home message
• Importing spreadsheets is easy
• Try it at home
• Help make it better

10. Grammar of spreadsheets
• Value
• Content
• Formula
• Formatting
• Conditional formatting
• Position

11. Grammar of spreadsheets
• Value
• Content
• Formula
• Formatting
• Conditional formatting
• Position
content formula
A1
A1 =A1

12. Grammar of spreadsheets
• Value
• Content
• Formula
• Formatting
• Conditional formatting
• Position
content formatted
0.9 1
1 1/01/1900

13. Grammar of spreadsheets
• Value
• Content
• Formula
• Formatting
• Position
Attribute Colour
Name Brown
Attribute Colour
Name Brown

14. Normalisation
header1 header2
value1 value2
row col character
1 1 header1
1 2 header2
2 1 value1
2 2 value2

15. Normalisation
header1 header2
value1 value2
row col character
1 1 header1
1 2 header2
2 1 value1
2 2 value2

16. Normalisation
row col value numFmt character date format id
1 1 1 General content 2
1 2 2 General formatted 2
2 1 0.9 0.0 3
2 2 0.9 0 4
3 1 1 0 5
3 2 1 d/mm/yyyy 1/01/1900 6
content formatted
0.9 1
1 1/01/1900

17. Normalisation
row col value numFmt character date format id
1 1 1 General content 2
1 2 2 General formatted 2
2 1 0.9 0.0 3
2 2 0.9 0 4
3 1 1 0 5
3 2 1 d/mm/yyyy 1/01/1900 6
content formatted
0.9 1
1 1/01/1900

18. Normalisation
row col value numFmt character date format id
1 1 1 General content 2
1 2 2 General formatted 2
2 1 0.9 0.0 3
2 2 0.9 0 4
3 1 1 0 5
3 2 1 d/mm/yyyy 1/01/1900 6
content formatted
0.9 1
1 1/01/1900

19. Normalisation
row col value numFmt character date format id
1 1 1 General content 2
1 2 2 General formatted 2
2 1 0.9 0.0 3
2 2 0.9 0 4
3 1 1 0 5
3 2 1 d/mm/yyyy 1/01/1900 6
content formatted
0.9 1
1 1/01/1900

20. Normalisation
row col value numFmt character date format id
1 1 1 General content 2
1 2 2 General formatted 2
2 1 0.9 0.0 3
2 2 0.9 0 4
3 1 1 0 5
3 2 1 d/mm/yyyy 1/01/1900 6
content formatted
0.9 1
1 1/01/1900

21. Formats
• Number format
• Font
• Fill
• Border
• Alignment
• Protection

22. Formats
• Number format
• Font
• Fill
• Border
• Top
• Colour
• Linetype
• Weight
• Left
• Colour
• Linetype …

23. Formats
> x$local$number_format
[1] "General"
[2] "d-mmm-yy"
[3] "[$-1409]d mmmm yyyy;@"
[4] "yyyy mmmm dddd"
[5] "0.00"
[6] "General"

24. Formats
> x$local$font$color$theme
[1] "dk2"
[2] "lt2"
[3] "dk1"
[4] "hlink"
[5] "accent2"
[6] "accent3"
[7] "accent4"
[8] "accent5"
[9] "accent6"

25. Formats
theme
local
x$theme x$local

27. > print(cells)
# A tibble: 25,884 × 15
address row col content formula formula_type formula_ref formula_group type character
<chr> <int> <int> <chr> <chr> <chr> <chr> <int> <chr> <chr>
1 A1 1 1 82 <NA> <NA> <NA> NA s Officer-issued excess speed band
2 C2 2 3 2009 <NA> <NA> <NA> NA <NA> <NA>
3 M2 2 13 114 <NA> <NA> <NA> NA s Total
4 N2 2 14 2010 <NA> <NA> <NA> NA <NA> <NA>
5 X2 2 24 114 <NA> <NA> <NA> NA s Total
6 AI2 2 35 114 <NA> <NA> <NA> NA s Total
7 AJ2 2 36 2011 <NA> <NA> <NA> NA <NA> <NA>
8 AT2 2 46 114 <NA> <NA> <NA> NA s Total
9 BE2 2 57 114 <NA> <NA> <NA> NA s Total
10 BF2 2 58 2012 <NA> <NA> <NA> NA <NA> <NA>
# ... with 25,874 more rows, and 5 more variables: height <dbl>, width <dbl>, style_format_id <int>,
# local_format_id <int>, comment <chr>

28. tidyxl
• The row/col position is the data
• Like hadley/readxl (Rcpp, RapidXml)
• But tidy
• Higher level of normalisation

29. tidyxl
• .xlsx
• contents() Normalised data
• formats() Formatting
• data.frames and matrices
• tidytable()
• offset()
• extend()
• anchor()
• pad()
• join_header()
unpivotr

30. tidyxl
• .xlsx
• contents() Normalised data
• formats() Formatting
• data.frames and matrices
• tidytable()
• offset()
• extend()
• anchor()
• pad()
• join_header()
• N() E() S() W()
• NNW() NNE() ENE() ESE()
SSE() SSW() WSW() WNW()
ABOVE() BELOW()
LEFT() RIGHT()
unpivotr

34. Cell selection
anchorDec <-
cells %>%
filter(col == 3, character == "Dec") %>%
split(.$row, .$col)

36. Cell selection
header_area <-
x %>% # x is a single cell from anchorDec
offset(cells, rows = 2, cols = -1) %>%
extend_S(cells,
boundary = ~ is.na(lead(content))) %>%
select(row, col, area = character)

38. Cell selection
datacells <-
header_area %>%
offset(cells, cols = 1) %>%
extend_E(cells,
boundary = ~ col == max(col)) %>%
mutate(value = if_else(is.na(character),
content,
character)) %>%
select(row, col, value)

39. • join_header()
• N() E() S() W()
• NNW() NNE() ENE() ESE()
SSE() SSW() WSW() WNW()
ABOVE() BELOW()
LEFT() RIGHT()

42. Joining
datacells %>%
W(header_district) %>%
W(header_area) %>%
NNW(header_month) %>%
NNW(header_year)

43. Joining
> dplyr::glimpse(tidied)
Observations: 19,684
Variables: 7
$ row <int> 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18...
$ col <int> 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4...
$ value <chr> "84", "65", "136", "48", "118", "591", "334", "10...
$ district <chr> "Auckland", "Auckland", "Auckland", "Bay Of Plent...
$ area <chr> "Auckland Central Area", "Auckland East Area", "A...
$ month <chr> "Dec", "Dec", "Dec", "Dec", "Dec", "Dec", "Dec", ...
$ year <chr> "2009", "2009", "2009", "2009", "2009", "2009", "...

49. Similar work
• https://github.com/ONS-OpenData/ONSdatabaker inspiration for tidyxl and unpivotr
• https://github.com/rsheets Jenny Bryan and Rich Fitzjohn (jailbreakr: AI)
• https://github.com/jennybc/2016-06_spreadsheets Jenny Bryan’s talk
• https://www.stat.auckland.ac.nz/~joh024/Research/TableToLongForm/ Jimmy Oh (match pattern)

50. Acknowledgments
Cook’s map of NZ
http://mapsof.net/new-zealand/cook-chart-of-new-zealand
Middle Earth
http://www.r-chart.com/2016/10/map-of-middle-earth-map-above-was.html
https://github.com/jvangeld/ME-GIS
http://worlds.outercraft.com/forum/
http://lotr.wikia.com/wiki/File:Middle_earth_map.jpg
http://kilbeth.deviantart.com/art/Middle-earth-Map-18324387
NZ Police
http://www.police.govt.nz/about-us/publication/road-policing-driver-offence-data-january-2009-
march-2016

51. tidyxl & unpivotr
Duncan Garmonsway
@nacnudus