Transcript of "Big Data, Open Data, Big Costs - tim willoughby"
Tim WilloughbyBig Data Open Data Big Costs? ISA 24th October
Gartner Top 10 for 2012 - • Released today by Forbes
What has Cloud ever done for us? Apart from Scale, Speed, Agility, Low Cost, Enterprise Adoption, Enterprise Mapping, Open Data, Standards, Google, API’s, etc Open Street Maps, Map Servers,GIS - More than just Location, Spatial Analysis and wider adoption now possible 3
Retailers..• Assign each shopper a unique code - keeps tabs on everything they buy. – If you use a credit card or a coupon, – or fill out a survey, – or mail in a refund, – or call the customer help line, – or open an e-mail we’ve sent you – or visit our Web site, – we’ll record it and link it to your ID,”• “We want to know everything we can.”• “Cool is one click away from Creepy” (Fjord)
Your Demographic They Have or can get• You o your age, o whether you are married, or got divorced o and have kids, Your ethnicity,• Where you live , the year you bought (or lost) your house, whether you’ve moved recently, • The number of cars you own. • Distance to nearest Outlet,• Worth • Your estimated salary, if you’ve ever declared bankruptcy, Charity Giving • What credit cards you carry in your wallet• Education, where you went to college,• Career - job history,• Online -what Web sites you visit., the magazines you read, reading habits, • The topics you talk about online,• Likes- certain brands of coffee, cereal etc, political leanings,
Big Data goes with… More DevicesMore Access More Apps More Data…
Radical Change… Social has overtaken mobile …. Mobile Browsing has overtaken desktop…
Change…With or Without the Owners / Shareholders
So far ICT has not fundamentally changed government • 1990s: lCT expected to make government more transparent, efficient and user oriented • 2005+: disillusion as bureaucracy still in existence • Can Mobile / Data /Cloud Help?Jane E. Fountain – Gov 1.0 – Just Replicating the Silos on the Internet
Goverment dont always Understand What the people Want?
Mobile Development Spend is rapidly overtaking PC Development Spend
BIG DATA? Massive network of services: water, sewage, drains … Need to know asset location for planning and maintenance Many databases, varying accuracy and provenance Context Ongoing street openings p.a. Safety!
OpenStreetMap, 2012200,000 contributors, one map
Our Lives are Different…• We communicate on — Facebook, Chat, Twitter, MMS• We Research Information — Blogs, eNews, Wiki, YouTube• We Buy — eBay, Amazon, Dell, Deal Done, etc…• We Travel - Tripadvisor, Flickr, RyanAir, Hotels,• We Meet / Retate - Linkedln, FaceBook, Friends• We Play – xBox, Playstation, Online games, ¡TV• We expect the Government to fit into these paradigms too…
What is Big Data? • Size beyond the ability of commonly used software tools • Data growing challenges • volume increasing • velocity (speed of i/o) • variety (data type/source) • Examples: computer tracking of shipments, sales, suppliers, cust omers, email, web traffic, social networkFrom McKinsey Report
Why Now?. What is the Catalyst?• People – Web sites with 300+ million unique visitors/month • Facebook • Google • YouTube• Access – 1.2 Billion active mobile broadband subscriptions• Commodity (community) Hardware – possibility for cost effective processing
Facts!• “Data is a vital raw material of the information economy, much as coal and iron ore were in the Industrial Revolution.”• “Mining and analyzing these big new data sets can open the door to a new wave of innovation, accelerating productivity and economic growth.”• We can exploit Internet-scale data sets to uncover new businesses and predict consumer behavior and location data market shifts. sales business data social network data
Think…• NY Traffic – They have mountains of data – More Daily – Don’t Understand it, can’t use it – What is the True Cost of Data Mining – Systems, Software, Computers, Analysts, STORAGE – Are we collecting, collating, etc the Right Data – People being removed daily, replaced by machines – Need to Stand Back….
How to deal with Lots of data• 7TB a Day for Twitter…• 10,000 CD’s• 5m Floppy• 225GB during this talk…• How do you Store that…• HD Write Speed- 80mb/s…• 24.3 Hrs to write 7TB….• Government Thinking is not in this space.. Yet..
Irish Government• Decision – “All Government Invoices Published” – Election Manifesto – Election Promises – Programme for Government• Action – None… – Fear, Bad Reactions, Data Quality,
Which Apps?• Government – Do not try to second guess what are useful applications!• No one can predict what data to make open based on potential Apps.• There are a huge number of apps available for smartphones. Impossible to guess a few years back.• Put Data out there. Data as a platform for others to build on, as infrastructure.• Apps you will like and hate, but it’s not up to you.
Where is Open Data delivering?• Participation / empowerment• Transparency, democratic control• New and better Government products and services• Impact measurement• Efficient and Effective Government• Data Journalism (Guardian Services)• New knowledge / real innovation
The Internet of Things Cars, Planes, Cameras,Roads, Pipes, Public Cameras, Traffic Lights, Car Parks, Pharma and Farms – over 1 Trillion
But all of these are static….. But… would you cross the road based on 5 min old information
Why Open Data• Because the Data already belongs to everyone!• Because there is Law, Inspire Directive - PSI• It helps change the incentives, from Fail Big, Fail Slow to Fail Small, Fail Fast, and Fail Forward (Innovation – More eyes)• Open Data will allow issues/problems to be seen, making them less likely to be repeated (learning from other people’s failures)• Open Data use can allow Incremental Innovation: - stand on the shoulders of giants. Don’t reinvent the wheel — use or improve someone else’s• Open Data quality can be increased over time — use and reuse by other people and combining with other data identifies (and helps solve) quality issues far more quickly and cheaply than internal checking
How do we measure Success?• Usage and Apps• Critical Mass of Companies• Search Results (Bing and Google Interested)• Business Committed to OD• Laws (Prison for breaking them)• If Open Data doesn’t cause difficulty for the Publisher to seek excellence then it isn’t working
Where next - Augmented RealityBring together Big Data, Visualisation, GIS, GPS - Bringing Cloud, Social, Data and Mobile together…
Understand what are the Driving Forces…• Government Focus – Understanding, Efficiency, Accountability• Technology Innovation Focus – Data as a Platform, Semantic Web• Reward Focus – Profit, Recognition• Digitising Government Focus – Computerisation / Technology Drive• Problem Solving Focus – New Skills needed to work on new Challenge• Social / Public Sector / Enterprise – More Focussed Services
Conclusions• Change is constant• Cloud is the Future• Social is a Reality• Mobile is already taking over• Data has to keep up…
References• We live in “Flat Land” – There is danger in making representations more seductive than the truth• Envisioning Information - Tufte
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.