The document discusses the challenges and benefits of complex open data publishing, highlighting the need for a layered approach to accommodate various users with differing capabilities and goals. It emphasizes the importance of visualizations for engagement and the necessity of bridging gaps between raw data access and user-friendly representations. Key techniques include resource-driven design and providing clear links between user interfaces and APIs to enhance usability and understanding.

1. Data All the Way Down
Jeni Tennison
@JeniT
http://www.jenitennison.com/blog/

2. Data All the Way Down
• challenges of complex open data
• layered approach to data publishing
• essential steps
• benefits

3. Complex Datasets
• too much for a single spreadsheet
• need to navigate
• browse through data
• look at slices of larger dataset
• view summary statistics
• need to explain
• definitions of terms, provisos & disclaimers

4. User Challenge
• complex data sets have range of users
• different hardware / platforms
• different tasks / goals
• different ability / understanding
• no one interface satisfies everyone
• data owners cannot satisfy everyone
• create ecosystem around open data

5. visualisation / data gap end user vs reuser

6. Visualisations
• approachable for real people
• necessary for stakeholder buy-in
• beauty is in what's left out
• advertisement or taster of rich datasets
• often not possible in official data
• leaves questions unanswered
• what if we looked at the data in a different way?

7. Raw Data
• importable into own data store
• often only interested in particular slice
• data set may be massive / changing
• run whatever analysis you want
• requires at least some programming skills
• analysis might not be appropriate for the data
• documentation probably lacking

8. bridging the gap layered data access
Photo by Nikita Kravchuk http://www.flickr.com/photos/mi55er/3845619153/

9. Layered Architecture
• user interface
• navigation and global understanding
• API
• curated, targeted, programmable access
• query
• free-form programmable access
• raw data

10. legislation.gov.uk lists as Atom feeds

11. legislation.gov.uk content as XML

12. legislation.gov.uk layer other views

13. organograms navigable visualisation

14. organograms JSON data

15. organograms RDF / XML / HTML

16. organograms SPARQL query

17. organograms raw data

18. Key Techniques
• resource-driven design (good URIs)
• every page built based on API calls
• explicit links to API access
• for bonus points, link to your transformation code
• consistent terminology
• clear mapping from UI to API
• caching & access control at each level

19. Benefits
• fork at any point
• don't like the visualisation / API? create your own!
• everyone is human
• reusers gain understanding from user interface
• visualisation benefits the stack
• API oriented towards achieving a goal
• visual validation of data improves quality

20. Questions?