Understanding Big Data for policy professionals

B
A
(alexj@businessabstraction.com)
It is not only Hadoop…

B
A
• Big Data are the new types of data that let go of the limitations
we had to impose decades ago due to the state of hardware
and software back then
• The main challenge is therefor unlearning said limitations,
and learning to incorporate Big Data capabilities and agility
into [policy] work
• Traditional reporting and BI works with “known knowns”. Big
data allows working with “known unknowns”, “unknown
knowns” and “unknown unknowns”.
• There are several distinctive types of technologies that fall
under the “Big Data“ moniker, which has their unique
capabilities: Hadoop, NOSQL, Semantic, Graph
2© Copyright Business Abstraction Pty Ltd 2014-2015

B
A
• Consists of tables tightly packed with data, specific type per
row
• Tables identified and created in advance
• Tables populated from human input
• Tables used by filtering, grouping by rows, as well as
performing a limited number of joins, for reports, OLAP etc
• Text data are supposed to be read by people

B
A
• Data coming from all over the Internet
• Data from Internet of Things.
• Human circumstances
• XML structures
• Data come from someplace, designed by someone else
• Machine learning
• Clustering
• Graph algorithms

B
A
• Traditional for IT
• Fully defined data
• Traditional Database
• Master Data Model
• Data Warehouse
• New generation of ideas
and technologies
• Presumes only part of
information is known
• Internet
• Information across
multiple enterprises
• Information extracted
from texts

B
A
New generations of tools, often coming from Internet companies,
designed for “New Data”
• Hadoop File System
• NoSQL: Cassandra, MarkLogic, Couchbase, DynamoDB
• Column-store RDB
• Semantic DBs
• Graph DBs
• Map/Reduce of different flavours
• Xquery
• Sparql
• Gremlin

B
A
• Write anything associated with a primary key (akin to a file
path)
• Distributed over commodity servers
• Highly concurrent write and read
• Everything is cheap – hardware, “design” etc
• However, small records have to be stored in Sequence files or
Map Files
• Anything at scale – can store files in Petabytes
• Designed for Map/Reduce batch work, data lakes
• Anything interactive requires massive hardware

B
A
The term “NoSQL” means “not relational”, and as such covers a
lot of different models. Some of them are suitable for complexity
of generic data storage. They are called “semi-structured” as
although individual data items are structures, the structures are
not necessarily defined in advanced
NoSQL platforms combine Hadoop’s “store anything” capability
with indexing and
• store and index XML or JSON documents (“trees”).
• A deep row store can be seen as a document database where
depth of trees limited to 2..
• Tables with named fields per row

B
A
• “Interactive Hadoop”
• Low-granularity Hadoop
• Data Lake
• Operations DBs with complex data
• Data consolidation
• Dynamic Data Warehouse
• Operational Data Warehouse
• Data presumed “forests” of “trees” – connected data are
handled not as good
• A touch more expensive than Hadoop

B
A
Provide traditional RDB interface in the new world
• Different internal structure
• Less suitable for OLTP
• Suitable for sparse data – empty fields don’t take space or
penalise for read
• Much faster for analytics, especially if only selected fields are
used
• Analytics when schema is known
• Cannot do schema-on-read

B
A
Support Resource Description Framework (RDF), originally
created for Semantic Web metadata. It stores information in
Subject-Predicate-Object “triples”, the most flexible
representation possible. Use Sparql for queries.
• Graph patterns
• Metadata for Hadoop/NoSQL. Lack of internal schema
requires external metadata
• Do not scale as much
• Hype-contaminated: people who understand enterprise and
understand Semantic Tech are rare

B
A
Graph Databases see data as one huge graph. They are
optimised for navigating the edges of the graph. Use Gremlin.
• Implementing Graph Analytics
• Bespoke graph logic
• Backend for general apps (if BASE jumping is too boring)
• Not as scalable as NoSQL
• Lack declarative data type, patterns & rules definitions of
Semantic DBs
• Depend on ability to build and maintain a graph

B
A
Platform for massively parallel computations, enables effective
sharing of workload between commodity servers.
• MapReduce
• YARN
• Apache Spark
• Batch jobs over massive data
• On-demand queries where some lag is acceptable
• Implementations have powerful Analytics/Machine Learning
libraries
• Latency

B
A
• Ensure datasets are identifiable
• Capture metadata
• Ensure your data are not lost
• Profile data across field names, structures etc
• Locate data as needed
• As you learn more about data, build up your metadata
• Hadoop
• NoSQL

B
A
• A server in $1,000-$10,000 range
• 0.5TB – 25TB per server
• A lot of them if needed
• Doubling the number of servers reduces the time to execute
the task by the factor of 2.

B
A
Perhaps more complex than learning
• There are a lot of data you do not know about which is
available and can be used
• For many types of objects, it is natural to have uncommon
attributes
• Data storage is cheap. It doesn’t cost much to store everything
remotely related
• No massive pre-work.
• Ask everything

B
A
• Traditional reporting, BI
• Predictive analytics
• Data consolidation, Semantic Integration, Object-based
Intelligence
• Clustering

B
A
• Straightforward operation, no design upfront
• Can take immensely complex metadata, like UML & BPMN
models
• Apply OWL for classification
• SWRL builds complex linkages
• Refer to Classes defined by lower-level Ontologies rather than
data

B
A
• Words to be converted to tags (URLs)
• Some words have multiple meanings
• Ontology provides possible tags for nouns
• Software tries to resolve expected predicates
• The tag that can find necessary relations (predicates) wins
• Use ontology to restrict search
• Much more flexible than “foreign key”

B
A
• Information about data
• Traditional metadata was stored in form of data schema
• With schema-less storage, metadata should be stored
separately
• Incremental discovery process requires Open World
Assumption – you don’t know what other data are there.
• Reasoning to handle complexity
• Relationships as first-class citizens and the basis for
classification

B
A
• Work in progress – not there yet
• Different data paradigms mandate different views
• SQL view of Big Data (Apache PIG etc)
• Excel import
• Analytic visualisation frontends
• 30+ JavaScript libraries
• Presume development
• Mahoot & other libraries
• Writing code in Scala, Java, Python, Groovy

B
A
• Better picture of the current state
• What if prediction
• Researching impact
• Increasing the number of categories, by several orders of
magnitude if necessary
• Common, meaningful view of individual, organisation etc
• Prevention of undesirable effects on insights, complex events
and prediction

B
A

B
A
• Description Logic, while using First-Order Predicate Logic
terminology
• Reduced for practical purposes
• Is not necessary to be productive
• Can be applied to anything
• Class can be derived depending on values
• A State is a Class
• New triples can be derived from existing

Understanding Big Data for policy professionals

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Understanding Big Data for policy professionals

Similar to Understanding Big Data for policy professionals (20)

Recently uploaded

Recently uploaded (20)

Understanding Big Data for policy professionals

Editor's Notes