Big data presentation from Gilbane Boston 2012 event

1. Applying Semantics to Unstructured
Data (Big and Getting Bigger)
Wednesday, November 30, 2012
4:00 – 5:00
Bryan Bell
Vice President, Enterprise Solutions, Expert
System
Lynda Moulton,
Analyst & Consultant, LWM Technology Services
Peter O'Kelly
Principal Analyst, O'Kelly Associates

2. Overall Session Agenda
• Introduction and context-setting
• "Big Data" 101 for Business
• Semantics and the Big Data Opportunity
2

3. Big Data 101 Agenda
• Big data in context
• Recap
• Risks
• Recommendations
3

4. Big Data in Context
• What is “big data”?
– Unhelpfully, both “big data” and “NoSQL,” generally
considered a key part of the big data wave, are defined
more in terms of what they aren’t than what they are
– A typical big data definition (Wikipedia):
• “[…] data sets that grow so large that they become awkward to
work with using on-hand database management tools”
– Often associated with Gartner’s volume, variety (and
complexity), and velocity model
• Also value and veracity considerations
4

5. Big Data in Context
• Why is big data a big deal now?
– Commoditized hardware, software, and networking
• Capability and price/performance curves that continue to
defy all economic “laws”
• Cloud services with radical new capability/cost equations
– Maturation and uptake of related open source
software, especially Hadoop
• Powerful and often no- or low-cost
5

6. Big Data in Context
• Why is big data a big deal now (continued)?
– Market enthusiasm for “NoSQL” systems
– Useful and often “open source”/public domain data
sources and services
– Mainstreaming of semantic tools and techniques
6

7. A Prime Minicomputer, c1982
7

8. Fast-Forward to 2012
8

9. Fast-Forward to 2012
9

10. Fast-Forward to 2012
10

11. Fast-Forward to 2012
11

12. Fast-Forward to 2012
12

13. Google BigQuery
13

14. Hadoop
• Hadoop is often considered central to big data
– Originating with Google’s MapReduce architecture,
Apache Hadoop is an open source architecture for
distributed processing on networks of commodity
hardware
– From Wikipedia:
• “’Map’ step: The master node takes the input, divides it into
smaller sub-problems, and distributes them to worker nodes
• ‘Reduce’ step: The master node then collects the answers to
all the sub-problems and combines them in some way to
form the output – the answer to the problem it was
originally trying to solve”
14

15. Hadoop
• Commercial application domains include (from
Wikipedia)
– Log and/or clickstream analysis of various kinds
– Marketing analytics
– Machine learning and/or sophisticated data mining
– Image processing
– Processing of XML messages
– Web crawling and/or text processing
– General archiving, including of relational/tabular data,
e.g. for compliance
15

16. Hadoop
• Hadoop is popular and rapidly evolving
– Most leading information management vendors
have embraced Hadoop
– There is now a Hadoop ecosystem
16

17. Meanwhile, Back in the Googleplex
• Dremel, BigQuery, Spanner, and other really
big data projects
17

18. Meanwhile, Back in the Googleplex
18

19. Google Now
19

20. A NoSQL Taxonomy
• From the NoSQL Wikipedia article:
20

21. A View of the NoSQL Landscape
21

22. Another NoSQL Landscape View

23. NoSQL Perspectives
• The “NoSQL” meme confusingly conflates
– Document database requirements
• Best served by XML DBMS (XDBMS)
– Physical database model decisions on which only DBAs and
systems architects should focus
• And which are more complementary than competitive with DBMS
– Object databases, which have floundered for decades
• But with which some application developers are nonetheless
enamored, for minimized “impedance mismatch,” despite significant
information management compromises
– Semantic (e.g., RDF) models
• Also more complementary than competitive with RDBMS/XDBMS
• Also consider: the “traditional” DBMS players can leverage
the same underlying technology power curves
23

24. Data as a Service
• The (single source of) truth is out there?...
– High-quality data sources are being commoditized
– Value is shifting to the ability to discern and leverage conceptual
connections, not just to manage big databases
• Some resources and developments to explore
– Social networking graphs and activities
– Data.com (Salesforce.com)
– Data.gov
– Google Knowledge Graph
– Linked Data
– Microsoft Windows Azure Data Marketplace
– Wikidata.org
– Wolfram Alpha
24

25. Mainstreaming Semantics
• Tools and techniques applied in search of
more meaning, e.g.,
– Vocabulary management
– Disambiguation and auto-categorization
– Text mining and analysis
– Context and relationship analysis
• It’s still ideal to help people capture and apply
data and metadata in context
– Semantic tools/techniques are complementary
25

26. Mainstreaming Semantics
• The Semantic Web is still more vision than reality
– But Google, Microsoft, and Yahoo, and Yandex, for
example, are improving Web searches by capturing
and applying more metadata and relationships via
schema.org schemas in Web pages
– And Google’s Knowledge Graph is about “things, not
strings,” with, as of mid-2012, “500 million objects, as
well as more than 3.5 billion facts about and
relationships between these different objects”
26

27. Recap
• Commoditization and cloud
– Very significant new opportunities
• Hadoop and related frameworks
– Complementary to RDBMS and XDBMS
• NoSQL
– Likely headed for meme-bust…
• Data services
– Game-changing potential
• Semantic tools and techniques
– Rapidly gaining momentum
27

28. Risks
• The potential for an ever-expanding set of information silos
– Focus on minimized redundancy and optimized integration
• GIGO (garbage in, garbage out) at super-scale
– New opportunities for unprecedented self-inflicted damage, for
organizations that don’t model or query effectively
• Cognitive overreach
– The potential for information workers to create and act on
nonsensical queries based on poorly-designed and/or
misunderstood information models
• Skills gaps can create competitive disadvantages
– Modeling, query formulation, and data analysis
– Critical thinking and information literacy
28

29. Recommendations
• Aim high: big data is in many respects just
getting started…
– A lot of technology recycling but also
significant and disruptive innovation
• Work to build consensus among stake-
holders on the opportunities and risks
• Focus on human skills – e.g., critical
thinking and information literacy
– For now, an instance of the most creative and
powerful type of semantic big data processor
we know of is between your ears
29