Standards for statistical data dissemination: a wish list

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Standards for statistical data dissemination

a wish list
Xa...

For humans

For humans (naked)

For humans (naked)
(accessed and read by humans)

For humans (naked)
(accessed and read humans)

For humans with machines
(accessed by humans,
but read by machines)

For humans (naked)
(accessed and ...

For machines

For machines built by humans

For machines built by humans
(accessed and read by machines)

API D A T A

General-purpose
“Apps”

Third-party
“Apps”
that solve specific needs

Statistics as platform “O.S.”

How I stopped worrying about my
website and learned to love the whole
Internet
Matt McAlister
Director of Digita...

We don’t operate in the
eyeball market maximum data
exposure & reach
We operate in th...

E A B
C
H
A Big orgs
N
G Stats-savvy
E
B2B

E A B
C
H
A Big orgs
N
G Stats-savvy
E
B2B

Shared environment

D
I
S
S
E
M
I
N
A
T
...

like the ?
WWW

For machines built by humans

don’t make me think

KISS principle

respect the conventions

develop also
for the weakest link

is a platform

An example

is widely used

An example

2006

is widely used

2009 50,000 applications

is widely used

# mashups

is widely used

had more than
1 provider

is widely used

= is a standard

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(…) easy to implement even on mobi...

KISS principle

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(…) easy to imp...

develop for the weakest link

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(...

provide a useful service

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(…) e...

get close to the end users

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(…)...

langs / APIs ~ browsers / websites

“these APIs are simple enough for weekend
hackers to build interesting projects on,...

“these APIs are simple enough for weekend
hackers to build interesting projects on, and
(…) easy to implement even on mobi...

2 APIs

REST API 100 x 1.5 = 150 pp.
Streaming API 6 x 3 = 18 pp.
< 200 pp.

208 pp.

< 200 pp.
books.google.com

The Fifteen Minute Rule

A person of reasonable
ability should be able to
get from zero to ‘Hello World’
...

http://lcsd05.cs.tamu.edu/slides/keynote.pdf

18 pp.
1 PDF
18 pp.

18 pp.
1 PDF

143 pp.

161 pp.
1 PDF

143 pp.

161 pp.
ZIP 8 MSW

1,109 pp.
ZIP 8 MSW
46 pp.
341 pp.
269 pp.
68 pp.
176 pp.
34 pp.
4 pp.
10 pp.

1,109 pp.
Samples

1,278 pp.
ZIP 1 PDF

169 pp.

1,327 pp.
PDF

49 pp.

1,377 pp.
PDF

50 pp.

1,408 pp.
PDF

31 pp.

1,408 pp.

1,344 pp.

books.google.com

high barrier to entry

API

developers

E

C
H
A
N SDMX
G
E

Shared environment

E

C
H
A
N SDMX
G amazingly flexible
E

Shared environment

E

C
H
A
N SDMX
G amazingly complex
E

Shared environment

E

C
H
A
N SDMX
G amazingly complex
E

SDMX-ML is a meta-language

D
I
S TypePad WordPress
S
E
M
I
N
...

The Metcalfe-Bray Law
D The value of a
I ...

The Metcalfe-Bray Law
D The value of a
I ...

D
I Are we
S Stats provider ...

Are we
so special
that
we
...

Population Search API

Population Search API

Atom

Population Search API

Atom OpenSearch

Population Search API

Atom OpenSearch SDMX

Population Search API

Population Search API

Rectifications API

Rectifications API

Atom OpenSearch

Rectifications API

Atom OpenSearch

Rectifications API

Atom OpenSearch

Rectifications API

Atom OpenSearch

Rectifications API

Atom OpenSearch

Rectifications API

Atom OpenSearch

Atom
OpenSearch
http://www.opensearch.org

Open Data Protocol
http://www.odata.org

Google Data Protocol
http://code.googl...

Are we
so special
that
we
...

Are we
so special
SDMX? that
we
...

Are we
so special
SDMX? that
we
...

A Complex System That Works Is Invariably
Found To Have Evolved From A Simple System
That Worked.

...

The parallel proposition also appears to be true:

A Complex System Designed From Scratch
Never Works And Cannot Be Mad...

What’s common practice
in the API space?

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

“REST”
RESTish
...

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

<KeyFamilyRef>ALFS_SUMTAB</KeyFamilyRef>
<Series>
<SeriesKey>
<Value concept="LOCATIO...

<KeyFamilyRef>ALFS_SUMTAB</KeyFamilyRef>
<KeyFamilyRef>ALFS_SUMTAB</KeyFamilyRef> <Series>
...

<KeyFamilyRef>ALFS_SUMTAB</KeyFamilyRef> "KeyFamilyRef" : "ALFS_SUMTAB" ,
<Series> ...

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { "location" : "AUS" , "subject" : "YGTT01L1_ST" , ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { "location" : "AUS" , "subject" : "YGTT01L1_ST" , ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { "location" : "AUS" , "subject" : "YGTT01L1_ST" , ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { "location" : "AUS" , "subject" : "YGTT01L1_ST" , ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , a “data” format "Series" : { "Seri...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , a “data” format "Series" : { ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { ...$

$"KeyFamilyRef" : "ALFS_SUMTAB" , "Series" : { "SeriesKey" : { ...$

From document viewer

Natural to programmers

to application platform

Fragmented device market

Fragmented device market

Storage
Offline
Geolocation
Audio and Video
Canvas
Drag and Drop
…

JS APIs

SGML / DTD XML / XSD

JSONP callback
JSON with padding

C
O
R
S

http://caniuse.com/#feat=cors

Date: Wed, 22 Feb 2012 17:12:52 GMT
Server: Apache
Expires: Wed, 22 Feb 2012 18:12:52 GMT
Access-Control-Allow-Origin: *
C...

XML SDMX-ML
JSON
SDMX-JS?

XML
JSON

XML Xml2Json()

JSON
On-the-fly conversion

XML Xml2Json()

JSON
On-the-fly conversion
<text>Hello, World!</text>

XML Xml2Json()

JSON
On-the-fly conversion
<text>Hello, World!</text>

"text": "Hello, World!"

XML Xml2Json()

JSON
On-the-fly conversion
<text>Hello, World!</text>
<text>Goodbye, Life!</text>

"text": "...

XML Xml2Json()

JSON
On-the-fly conversion
<text>Hello, World!</text>
<text>Goodbye, Life!</text>

"text": [...

XML Xml2Json()

JSON
On-the-fly conversion
<text>Hello, World!</text>

"text": ["Hello, World!"]

XML
JSON
Predefined map

XML
JSON
Predefined map
namespaces?
elements/attributes?
all strings?

“XMLish”
JSON
Predefined map

${"@concept" : "OBS_STATUS", "@value" : "P"}}}, {"generic:Time" : "2011-04", "generic:ObsValue" : “XMLish” {"@val...$

natural-born

JSON

natural-born

JSON
“JSON-G” genuine

natural-born

JSON
“JSON-G”
“JSON-C”
compact/customized

Price of an apple?

Save as many bytes
as possible

Price of an apple?

Nutrition value
Allergies
Recipes
Price
Forbidden fruit in
the Garden of Eden

Flexibility
Nutrition value
Allergies
Recipes
Price
Forbidden fruit in
the Garden of Eden

detail=full
detail=dataonly
detail=serieskeysonly
detail=nodata

ResponseGroup=Images,Reviews

fields=id,name,picture

fields=title,teaserbyline,none

:(id,first-name,last-name,positions:(title))

fields=id,scope/type

fields=entry(id,gd:when)

fields=entry(@gd:*,id,gd:when)

fields=entry(title,link[@rel='self' or @rel='edit'],
media:group(media:thumbnail))

fields=entry[media:group/yt:duration/@seconds
<= 120 and yt:statistics/@viewCount > 20]

simple
flexible
RESTish
JSON
lightweight
developer-centric
...

simple
flexible
RESTish
JSON
lightweight
Semantic Web-ish?
develope...

I have a dream for the Web [in which computers] become
capable of analyzing all the data on the Web – the content,
links, ...

Statistical “Cube” Data. The group will produce a
vocabulary, compatible with SDMX, for expressing
some kinds of statistic...

The Mainstream
Relevance “Law”

The mainstream
relevance of a
communication
environment is
proportional to
the quantity of...

The Mainstream
Relevance “Law”

The mainstream
relevance of a
communication
environment is
proportional to
the quantity of...

The Mainstream
Relevance “Law”

TV
e-mail
Twitter
...

simple?
flexible
RESTish
JSON?
lightweight?
Semantic Web-ish?
develo...

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

Open APIs and the Semantic Web (John Musser, ProgrammableWeb)
http://www.slideshare.net/jmusser/j-musser-semtechjun2011

For some web developers the need to understand
the RDF data model and associated serializations and
query language (SPARQL...

Simplify + JSONify SDMX?

Reuse/adapt a generic data format?
semantic

Simplify + JSONify SDMX?

Reuse/adapt a generic data format?
semantic

Create a new simple standard for sta...

http://json-stat.org

http://json-stat.org

lightweight

http://json-stat.org

simple

http://json-stat.org

developer-centric

http://json-stat.org

flexible

http://code.google.com/p/json-stat/

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Latest population figure of Catalonia",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", "time...

{
"label": "Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"long" : 2.178389,
"dimension"...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"index"
"dimension" : {: {
...

{
"label": “Population of Catalonia, January 1st, 2011",
"value": [7539618] ,
"dimension" : {
"id" : ["metric", ...

"value": [3732196, 3807422 , 7539618]

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex" : {
"categ...

"value": [3732196, 3807422 , 7539618]
"id" : ["metric", "time", "geo", "sex"] ,
"size" : [1, 1, 1, 3]
"sex3" : {
"attr...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [
3058628, 3115336, 3201029, 3309850, 3366329, 3468235, 3543706, 3578176, 3661028, 3713765, 3724515, 3732196,...

"value": [3732196, 3807422 , 7539618]

"sex" : {
"category" : {
"index" : {"M" : 0 , "F" : 1 , "T" : 2} ,
"lab...

"value": [3732196, 3807422 , 7539618]

"sex" : {
"category" : {
"index" : {"M" : 0 , "F" : 1 , "T" : 2} ,
"lab...

"value": [3732196, 3807422 , 7539618]

"sex" : {
"category" : {
"uri" : "http://jsonstat.dataprovider.org/dimensio...

The JSON-stat
Community
Welcomes You!

The JSON-stat
Community
Welcomes You!
Needs You!

Standards for statistical data dissemination

a wish list
Xa...

Dan Taylor
borman818 / Daniel Borman
Christian Cable
Ian Mu...

Photos of products and webpages have been taken from the respective
companies’ websites. 1993 Mosaic webpage, © Board of T...

Top clipped slide