Master Class at the Nordiskt statistikermöte 2016. Stockholm, August 22, 2016.

1. Consuming
Nordic Statbank data
with JSON-stat
Xavier Badosa (@badosa)
Statistical Institute of Catalonia (Idescat)

2. Aluu
Hallå
Hallo
Halló
Hei
Hej

3. Consuming
Nordic Statbank data
with JSON-stat
Xavier Badosa (@badosa)
Statistical Institute of Catalonia (Idescat)

5. Consuming
Nordic Statbank data
with JSON-stat

6. JSON-stat.org
a simple light standard
for all kinds of data disseminators
}

8. http://www.stat2go.com

13. data
as an
infrastructure

14. API
NSO as a
platform
data
as an
infrastructure

15. The Software Staircase
access
processing
webpages

16. download
The Software Staircase
access
processing
webpages

17. webpages
download
API
The Software Staircase
access
processing

18. data
provider
developer

19. users
data
provider
developer

20. users
data
app

21. users
data
API
app

22. real
tim
e

23. lightweight

24. Statistics Sweden
AM0401DA
Population
aged 15-74
(LFS), thousands
by sex, age, labour
status and month
SDMX-ML XML
PX-JSON JSON
JSON-stat JSON
3,709 Kb
848 Kb
92 Kb

25. JavaScript Object Notation
JSON
http://json.org

26. "This is a text."
JS

27. 666
JS

28. 666
3.14159265
JS

29. 666
3.14159265
-299792458
JS

30. 666
3.14159265
-299792458
null
JS

31. 666
3.14159265
-299792458
null true false
JS

32. [ 1, 2, 3 ]
JS

33. [ 1, 2, 3 ]
[ "one", "two", "three" ]a =
JS

34. [ 1, 2, 3 ]
[ "one", "two", "three" ]a =
a[1]
0 1 2
JS

35. [ "one", "two", "three" ]
[ 1, 2, 3 ]
[ "Men", 301784 ]
JS

36. [ "one", "two", "three" ]
[ 1, 2, 3 ]
[ "Men", 301784 ]
[
[ "Men", 301784 ],
[ "Women", 284434 ]
]
JS

37. [
[ "Men", "0-4", "2015", "Sweden", 301784 ],
[ "Women", "0-4", "2015", "Sweden", 284434 ]
]
JS

38. [
{ "sex": "Men", "val": 301784 }
{"sex": "Women", "value": 3800}
]
a =
JS

39. [
{ "sex": "Men", "val": 301784 }
{"sex": "Women", "value": 3800}
]
a =
a["val"]
a.val
JS

40. [
{ "sex": "Men", "val": 301784 },
{ "sex": "Women", "val": 284434 }
]
JS

41. [
{ "sex": "Men", "val": 301784 },
{ "sex": "Women", "val": 284434 }
]
a =
JS

42. [
{ "sex": "Men", "val": 301784 },
{ "sex": "Women", "val": 284434 }
]
a =
a[1]
JS

43. [
{ "sex": "Men", "val": 301784 },
{ "sex": "Women", "val": 284434 }
]
a =
a[1]["sex"]
a[1].sex
JS

44. [
{
"concept": "Population",
"sex": "Men",
"age": "0-4",
"country": "Sweden",
"year": "2015",
"value": 301784
},
{
"concept": "Population",
"sex": "Women",
"age": "0-4",
"country": "Sweden",
"year": "2015",
"value": 284434
},
...
]
Array-of-objects pattern

47. Abbreviation

48. Cubic
Model
Describe data
in dimension terms

49. [
{
"concept": "Population",
"sex": "Men",
"age": "0-4",
"country": "Sweden",
"year": "2015",
"value": 301784
},
{
"concept": "Population",
"sex": "Women",
"age": "0-4",
"country": "Sweden",
"year": "2015",
"value": 284434
},
...
]

52. [
]
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
"value" :

53. value
note/source/updated
label

54. [
]
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
"value" :
"version" : "2.0", "class" : "dataset",
"label" : "Population by sex and age group. Canada. 2012",
"source" : "Statistics Canada, CANSIM, table 051-0001",
"updated" : "2012-09-27",
}
{

55. [
] ,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
"value" :
}
"id" : [ "country" , "year" , "age" , "concept" , "sex" ],
"size" : [ 1 , 1 , 20 , 2 , 3 ],
"dimension" : { … }
"version" : "2.0", "class" : "dataset",
"label" : "Population by sex and age group. Canada. 2012",
"source" : "Statistics Canada, CANSIM, table 051-0001",
"updated" : "2012-09-27",
{

56. "value" : [ … ],
}
"id" : [ "country" , "year" , "age" , "concept" , "sex" ],
"size" : [ 1 , 1 , 20 , 2 , 3 ],
"role" : { "time" : ["year"] , "geo" : ["country"] , "metric" : ["concept"] },
"dimension" : {
"country" : { … },
"year" : { … },
"age" : { … },
"concept" : { … },
"sex" : { … }
}
"version" : "2.0",
"class" : "dataset",
"label" : "Population by sex and age group. Canada. 2012",
"source" : "Statistics Canada, CANSIM, table 051-0001",
"updated" : "2012-09-27",
{

57. "value" : [ … ],
}
"id" : [ "country" , "year" , "age" , "concept" , "sex" ],
"size" : [ 1 , 1 , 20 , 2 , 3 ],
"role" : { "time" : ["year"] , "geo" : ["country"] , "metric" : ["concept"] },
"dimension" : {
"country" : { … },
"year" : { … },
"age" : { … },
"concept" : { … },
"sex" : { … }
}
"version" : "2.0",
"class" : "dataset",
"label" : "Population by sex and age group. Canada. 2012",
"source" : "Statistics Canada, CANSIM, table 051-0001",
"updated" : "2012-09-27",
{

58. "sex" : {
"label" : "sex",
"category" : {
"index" : ["T", "M", "F"],
"label" : {
"T" : "total",
"M" : "male",
"F" : "female"
}
}
}

59. [
]
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
?
The “unflattening” problem

60. function arr2num( arr, size ){
for(var i=0, num=0, mult=1, ndims=size.length; i<ndims; i++){
mult*=(i>0) ? size[ndims-i] : 1;
num+=mult*arr[ndims-i-1];
}
return num;
}
The “unflattening” problem

61. Method: Row-major order
In computing, row-major order and column-
major order describe methods for arranging
multidimensional arrays in linear storage such
as memory.
The “unflattening” problem

62. users
data
API
dev

63. users
dev
data
tools
API

65. JavaScript
R
Python
Python
Java

67. You could not step twice into the same river.
Everything changes and nothing stands still.
Heraclitus

68. Responses Version
bundle
0
(set of one or more datasets)

69. Responses Version
bundle
1dataset
dimension
collection

70. Responses Version
bundle
2dataset
dimension
collection

71. 2.0
2.0
2.0
2.0
JavaScript
R
Python
Python
Java

72. {
"version": "2.0", "class": "dataset",
"label": "Persons by region, sex, age and time",
"source": "Statistics Norway",
"updated": "2016-06-01T19:45:12Z",
"value": [ 26175, 26923, 27394, 29332, 30325, ... ],
"id": ["Region","Kjonn","Alder","Tid","ContentsCode"],
"size": [1, 2, 106, 31, 1],
"role": {"time": ["Tid"], "metric": ["ContentsCode"]},
"dimension": {...}
}
Version 2. Dataset

73. {
"your dataset id here": {
"label": "Persons by region, sex, age and time",
"source": "Statistics Norway",
"updated": "2016-06-01T19:45:12Z",
"value": [ 26175, 26923, 27394, 29332, 30325, ... ],
"dimension": {...}
}
}
Version 0/1. Bundle

74. {
"dataset": {
"label": "Persons by region, sex, age and time",
"source": "Statistics Norway",
"updated": "2016-06-01T19:45:12Z",
"value": [ 26175, 26923, 27394, 29332, 30325, ... ],
"dimension": {...}
}
}
Version 0/1. Bundle

76. PX-Web

77. PX-Web

78. JSON-stat

79. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA

80. user
dev
prov

81. interface
data
logic

82. interface
logic
retrieve
process
build
data

83. process
build
retrieve
JavaScript

84. process
build
retrieve
JavaScript

85. process
build
retrieve
JSON-stat
Javascript
Toolkit
JSON-stat
Javascript
Utilities Suite
https://json-stat.com
JavaScript

86. process
build
JavaScript
JSONP, XHR, Fetch
JJT
jQuery…
JJT
retrieve

87. process
build
JavaScript
JSONP, XHR, Fetch
JJT
jQuery…
JJT
Visual
retrieve

88. Introducing
the
JSON-stat
Javascript
Toolkit
json-stat.com

89. JSON-stat Tree
Dataset
Dimension
Category
Data
1. Chainable traversing methods

90. Dataset
Dimension
Category
Data
Metadata Path
class
label
length
id
1. Chainable traversing methods

91. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Dimension(3).Category(2).label
JSONstat(obj).Dataset("dataset").Dimension("sex").Category("F").label
"female"

92. Dataset
Dimension
Category
Data
Data Path
value
status
1. Chainable traversing methods

93. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
} ).value

94. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
"country" : "CA",
"year" : "2012",
"concept" : "POP",
"age" : "34",
"sex" : "F"
} ).value 1202.8

95. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
"country" : "CA",
"year" : "2012",
"concept" : "POP",
"age" : "34",
"sex" : "F"
} ).value 1202.8

96. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
"country" : "CA",
"year" : "2012",
"concept" : "POP",
"age" : "34",
"sex" : "F"
} )

97. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
"country" : "CA",
"year" : "2012",
"concept" : "POP",
"age" : "34",
"sex" : "F"
}, false )
[2406.3, 1203.5, 1202.8]
T M F

98. 1. Chainable traversing methods
JSONstat(obj).Dataset(0).Data( {
country : "CA",
year : "2012",
concept : "POP",
age : "34",
sex : "F"
}, false )
[2406.3, 1203.5, 1202.8]
T M F

99. 2. Chainable transformation methods
Dataset
toTable
Slice

100. JSONstat( obj ).Dataset(0).toTable( { type : "arrobj" } );
[
{
age : "Total",
concept : "Persons (thousands)",
country : "Canada",
sex : "Total",
year : "2012",
value : 34880.5
},
{
age : "Total",
concept : "Persons (thousands)",
country : "Canada",
sex : "Male",
year : "2012",
value : 17309.1
},
…
]
Array-of-objects pattern

101. json-stat.com/nsm

102. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA

104. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Download

106. process
build
JavaScript
JSONP, XHR, Fetch
JJT
jQuery…
JJT
Visual
retrieve

107. Demo

108. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API Dedicated Section

109. http://www.scb.se/en_/About-us/Open-data-API/API-for-the-Statistical-Database-/

110. Demo

111. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API JSONP

112. Demo

113. HTTP Verbs
GET
POST Requests that the server stores
the data enclosed in the body of
the request message.
Requests a representation of
the specified resource.
…

114. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GETJSONP

118. Demo

119. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GET SQueryJSONP

121. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GETJSONP SQuery

122. Demo

123. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GETJSONP SQuery

124. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
SBA
CORS SQuery

125. Cross-Origin
Resource Sharing
A web application using XHR/Fetch can
only make HTTP requests to its own domain.
Same-Origin Policy
CORS SQuery

126. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GETJSONP SQuery

127. Demo

128. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GETJSONP SQuery

129. Demo

130. Denmark
Norway
Sweden
Finland
Greenland
Iceland
Faroe Islands
ΩΩ
SBA
Down. API GET SQueryJSONP

131. ΩΩThank
You

132. Credits
“Soma” (blocks’ background)
by Dru! (CC BY-NC)
“Metal movable type”
by Willi Heidelbach (CC BY-SA)
“Portrait” (cubic head)
by Thomas Leth-Olsen (CC BY)
“River in Iceland”
by Kamil Porembiński (CC BY-SA)
“Silicon 0116 6336” (integrated circuit)
by Ross Elliott (CC BY)
“Sterile” (walking girl)
by Lee Nachtigal (CC BY)
“Cubes” (sculpture)
by Alex [Fino] LA (CC BY-SA)
“Railroad”
by Xavier Badosa (CC BY)
“Dartboard”
by Jacob Vance (CC BY-NC)
Icons by Visualpharm.com (Linkware)