Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data, Open Data, Big Costs - tim willoughby


Published on

Presentation to the 8th Annual Public Sector ICT Conference. Irish Software Association. Tim Willoughby - Open Data, Bug Data Big Costs

Published in: Technology
  • Be the first to comment

Big Data, Open Data, Big Costs - tim willoughby

  1. 1. Tim WilloughbyBig Data Open Data Big Costs? ISA 24th October
  2. 2. Gartner Top 10 for 2012 - • Released today by Forbes
  3. 3. What has Cloud ever done for us? Apart from Scale, Speed, Agility, Low Cost, Enterprise Adoption, Enterprise Mapping, Open Data, Standards, Google, API’s, etc Open Street Maps, Map Servers,GIS - More than just Location, Spatial Analysis and wider adoption now possible 3
  4. 4. How Much Data do you create?
  5. 5. How much data to you create?
  6. 6. Retailers..• Assign each shopper a unique code - keeps tabs on everything they buy. – If you use a credit card or a coupon, – or fill out a survey, – or mail in a refund, – or call the customer help line, – or open an e-mail we’ve sent you – or visit our Web site, – we’ll record it and link it to your ID,”• “We want to know everything we can.”• “Cool is one click away from Creepy” (Fjord)
  7. 7. Your Demographic They Have or can get• You o your age, o whether you are married, or got divorced o and have kids, Your ethnicity,• Where you live , the year you bought (or lost) your house, whether you’ve moved recently, • The number of cars you own. • Distance to nearest Outlet,• Worth • Your estimated salary, if you’ve ever declared bankruptcy, Charity Giving • What credit cards you carry in your wallet• Education, where you went to college,• Career - job history,• Online -what Web sites you visit., the magazines you read, reading habits, • The topics you talk about online,• Likes- certain brands of coffee, cereal etc, political leanings,
  8. 8. Big Data goes with… More DevicesMore Access More Apps More Data…
  9. 9. 4 Global Themes Social DATA Mobile Cloud
  10. 10. Radical Change… Social has overtaken mobile …. Mobile Browsing has overtaken desktop…
  11. 11. Change…With or Without the Owners / Shareholders
  12. 12. So far ICT has not fundamentally changed government • 1990s: lCT expected to make government more transparent, efficient and user oriented • 2005+: disillusion as bureaucracy still in existence • Can Mobile / Data /Cloud Help?Jane E. Fountain – Gov 1.0 – Just Replicating the Silos on the Internet
  13. 13. Goverment dont always Understand What the people Want?
  14. 14. What governments often Deliver
  15. 15. Open Government?“If people don’t know what you’re doing, theydon’t know what you’re doing wrong”“Jim Hacker”Open Government (I980)
  16. 16. The World has changed…
  17. 17. Every day - you come across multiple…• - Devices• - applications• - access points• - Networks• - Locations• - Servers• - Data types• - Data caches• - Service providers• - Operating systems• - Storage locations• - Encryption / security protocols..
  18. 18. My Smart Device is…. Very smart Source : Peter Cochrane
  19. 19. It knows moreabout me than my partner… Source : Peter Cochrane
  20. 20. Security is changing• Security has to be appropriate• Security has to be measured• Can have things so secure that they are unusable.
  21. 21. Citizen Expectation Changed? Social Media
  22. 22. The More data we have… the less we (have time to) Understand it
  23. 23. Government as a Platform?
  24. 24. Mobile Development Spend is rapidly overtaking PC Development Spend
  25. 25. BIG DATA? Massive network of services: water, sewage, drains … Need to know asset location for planning and maintenance Many databases, varying accuracy and provenance Context  Ongoing street openings p.a.  Safety!
  26. 26. Typical Individual effortmany hours, one map
  27. 27. OpenStreetMap, 2012200,000 contributors, one map
  28. 28. Our Lives are Different…• We communicate on — Facebook, Chat, Twitter, MMS• We Research Information — Blogs, eNews, Wiki, YouTube• We Buy — eBay, Amazon, Dell, Deal Done, etc…• We Travel - Tripadvisor, Flickr, RyanAir, Hotels,• We Meet / Retate - Linkedln, FaceBook, Friends• We Play – xBox, Playstation, Online games, ¡TV• We expect the Government to fit into these paradigms too…
  29. 29. What is Big Data? • Size beyond the ability of commonly used software tools • Data growing challenges • volume increasing • velocity (speed of i/o) • variety (data type/source) • Examples: computer tracking of shipments, sales, suppliers, cust omers, email, web traffic, social networkFrom McKinsey Report
  30. 30. Why Now?. What is the Catalyst?• People – Web sites with 300+ million unique visitors/month • Facebook • Google • YouTube• Access – 1.2 Billion active mobile broadband subscriptions• Commodity (community) Hardware – possibility for cost effective processing
  31. 31. Facts!• “Data is a vital raw material of the information economy, much as coal and iron ore were in the Industrial Revolution.”• “Mining and analyzing these big new data sets can open the door to a new wave of innovation, accelerating productivity and economic growth.”• We can exploit Internet-scale data sets to uncover new businesses and predict consumer behavior and location data market shifts. sales business data social network data
  32. 32. Think…• NY Traffic – They have mountains of data – More Daily – Don’t Understand it, can’t use it – What is the True Cost of Data Mining – Systems, Software, Computers, Analysts, STORAGE – Are we collecting, collating, etc the Right Data – People being removed daily, replaced by machines – Need to Stand Back….
  33. 33. How to deal with Lots of data• 7TB a Day for Twitter…• 10,000 CD’s• 5m Floppy• 225GB during this talk…• How do you Store that…• HD Write Speed- 80mb/s…• 24.3 Hrs to write 7TB….• Government Thinking is not in this space.. Yet..
  34. 34. Irish Government• Decision – “All Government Invoices Published” – Election Manifesto – Election Promises – Programme for Government• Action – None… – Fear, Bad Reactions, Data Quality,
  35. 35. Stakeholders• Private citizens• Activists• Coders• Entrepreneurs• NGO’s• Academics• Civil Servants• Gov Institutions
  36. 36. Which Apps?• Government – Do not try to second guess what are useful applications!• No one can predict what data to make open based on potential Apps.• There are a huge number of apps available for smartphones. Impossible to guess a few years back.• Put Data out there. Data as a platform for others to build on, as infrastructure.• Apps you will like and hate, but it’s not up to you.
  37. 37. Applications?
  38. 38. Where is Open Data delivering?• Participation / empowerment• Transparency, democratic control• New and better Government products and services• Impact measurement• Efficient and Effective Government• Data Journalism (Guardian Services)• New knowledge / real innovation
  39. 39. The Internet of Things Cars, Planes, Cameras,Roads, Pipes, Public Cameras, Traffic Lights, Car Parks, Pharma and Farms – over 1 Trillion
  40. 40. Smart Local Government?
  41. 41. Mass increase in new net devices • Move to digital controls Wireless integration • Network standards – TCP/IP communications – Wireless communications60
  42. 42. Or WizardsGIS is nowMainstream
  43. 43. Water Meters• Read the Meter• View or Pay the Bill• Compare the Usage to the Local or National Averages• Look for areas of Savings
  44. 44. Roadworks• Link to System – Online Road Works Control (OLRWC)• License for Road Opening• Cost of Reinstatement (Local Authority Usage)• License for any Street usage.
  45. 45. Planning Applications• View Applications• Get notified of Changes• Make Comments• Make a Submission• Notify Local Authority of Issues?
  46. 46. Sports• What is available now• How do I get there• What if I have to Cancel• Pay for it now
  47. 47. Parking• Select County• Select Zone• Select Price willing to pay?• Where is the Cheapest
  48. 48. open source / open data / cloud the fix your street model
  49. 49. The Reality ofFixYourStreet
  50. 50. But all of these are static….. But… would you cross the road based on 5 min old information
  51. 51. Why Open Data• Because the Data already belongs to everyone!• Because there is Law, Inspire Directive - PSI• It helps change the incentives, from Fail Big, Fail Slow to Fail Small, Fail Fast, and Fail Forward (Innovation – More eyes)• Open Data will allow issues/problems to be seen, making them less likely to be repeated (learning from other people’s failures)• Open Data use can allow Incremental Innovation: - stand on the shoulders of giants. Don’t reinvent the wheel — use or improve someone else’s• Open Data quality can be increased over time — use and reuse by other people and combining with other data identifies (and helps solve) quality issues far more quickly and cheaply than internal checking
  52. 52. How do we measure Success?• Usage and Apps• Critical Mass of Companies• Search Results (Bing and Google Interested)• Business Committed to OD• Laws (Prison for breaking them)• If Open Data doesn’t cause difficulty for the Publisher to seek excellence then it isn’t working
  53. 53. Where next - Augmented RealityBring together Big Data, Visualisation, GIS, GPS - Bringing Cloud, Social, Data and Mobile together…
  54. 54. What is Augmented Reality?
  55. 55. Water Meter
  56. 56. Water Meter
  57. 57. Understand what are the Driving Forces…• Government Focus – Understanding, Efficiency, Accountability• Technology Innovation Focus – Data as a Platform, Semantic Web• Reward Focus – Profit, Recognition• Digitising Government Focus – Computerisation / Technology Drive• Problem Solving Focus – New Skills needed to work on new Challenge• Social / Public Sector / Enterprise – More Focussed Services
  58. 58. Conclusions• Change is constant• Cloud is the Future• Social is a Reality• Mobile is already taking over• Data has to keep up…
  59. 59. References• We live in “Flat Land” – There is danger in making representations more seductive than the truth• Envisioning Information - Tufte