Harry Potter and Enormous Data (Pavlo Baron)

Harry potter and enormous data

Pavlo Baron http://www.pbit.org [email_address] @pavlobaron

Why? When? How? Wherewith?

Finance/insurance dilemma the better your offer, the better you need it to get

Trading/brokerage dilemma the faster/cheaper you are, the faster/cheaper you need to get

Web dilemma the more people visit you, the more people you need to reach

Telco/public utility dilemma the more traffic you have, the better you need it to get

Medicine dilemma the more you check, the more you need to check

Law and order dilemma the more you protect, the more you need to perceive

Intelligence dilemma the more you know, the more you need to know

The price of success is data growth. The more success you have/want, the more data you have/need

Why ? When? How? Wherewith?

Your data is (on) the web

Your data is for/from mobile devices

Your data comes from sensors/RFID

Your data comes over GPS

Your data is video/audio

You data is (totally) unstructured

You get flooded by tera-/petabytes of data

You simply get bombed with data

Your data flows on streams at a very high rate from different locations

You have to read The Matrix

You need to distribute your data over the whole world

Your existence depends on (the quality of) your data

It’s not sufficient anymore just to throw it at your Oracle DB and to hope it works

You have no chance without science

To control enormous data means to learn/know algorithms/ADTs computing systems networking operating systems database systems distributed systems

You need to scale very far. And to scale with enormous data means to...

Chop in smaller pieces

Chop in bite-size, manageable pieces

Separate reading from writing

Update and mark, don’t delete physically

Minimize hard relations

Separate archive from accessible data

Trash everything that has only to be analyzed in real-time

Parallelize and distribute

Avoid single bottle necks

Decentralize with “ equal” nodes

Minimize the distance between the data and its processors

Design with Byzantine faults in mind

Build upon consensus, agreement, voting, quorum

Don’t trust time and timestamps

Strive for O(1) for data lookups #

Utilize commodity hardware

Consider hardware fallibility

Relax new hardware startup procedure

Bring data to its users

Build upon asynchronous message passing

Consider network unreliability

Consider asynchronous message passing unreliability

Design with eventual actuality/consistency in mind

Implement redundancy and replication

Consider latency an adjustment screw

Consider availability an adjustment screw

Be prepared for disaster

Utilize the fog/clouds/CDNs

Utilize GPUs

Design for theoretically unlimited amount of data

Design for frequent structure changes

Design for the all-in-one mix

There are storage approaches much better suited for enormous data amounts than your MySQL

There are preparation/processing mechanisms much better suited for enormous data than your shell scripts

There are technologies much better suited for spreading enormous data all over the world than your Apache cluster

There are mechanisms much better suited for statistical analysis of enormous data than your DWH

There are real-time processing approaches much better suited for enormous data than your SQL queries

There are technologies much better suited for the visualization of enormous data than your Excel

Just pick right tools for the job

I want to optimize and customize my offers, improve business forecasts. How can I do that?

Protocol every single user activity and MapReduce the logs. Even in real-time

You can use a distributed MapReduce framework to analyze your textual/binary log files

Hadoop, Disco, Skynet, Mars etc.

Or you can log straight into a distributed data store and MapReduce it there

Riak, Cassandra, Greenplum etc.

Use graph DBs to store and track your customers’ relations of any kind and any depth

Neo4j, GraphBase, OrientDB etc.

I want to monitor my market, competitors’ prices and automatically adjust my offers, offer and trade in first. How can I do that?

Build upon CEP/high speed trading platform to process events, fire alerts and execute business logic as events come, to make predictions

Esper, LMAX, StreamBase etc.

Keep it all in memory

Redis, Memcached etc.

I want to act and react on events and anomalies just in time, perceive and predict risks. How can I do that?

Build upon CEP to process events and fire alerts as they come, consider their causality

Esper, StreamBase etc.

I want to analyze huge video and audio files and streams real fast. How can I do that?

Parallelize calculations on many GPUs

OpenCL, CUDA, GPGPU etc.

I want to automatically optimize my nets and infrastructure according to traffic. How can I do that?

You can build upon CEP to ad hoc adjust traffic/net from SNMP traps, sensors

Esper, StreamBase etc.

You can use a distributed MapReduce framework to analyze your log files

Hadoop, Disco, Skynet, Mars etc.

I want my users all over the world to access my content and offers as fast as possible. How can I do that?

Go with a CDN to bring your content and offers as close to the consumer as possible

Akamai, Amazon CloudFront etc.

Use cloud based stores to make your data globally accessible. Split and spatially distribute your data there

Amazon S3, Rackspace Cloud Files etc.

I want to easily visualize my huge data amounts in dashboards, reports in almost no time. How can I do that?

Go with a general purpose platform for statistical computing and/or data visualization

R, Mondrian etc.

Thank you

Most images originate from istockphoto.com except few ones taken from Wikipedia and product pages or generated through public online generators

http://paper.li	24
http://coderwall.com	23
http://a0.twimg.com	7

Harry Potter and Enormous Data (Pavlo Baron)

by Pavlo Baron

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 54

Statistics