Big data and APIs for PHP developers - SXSW 2011

Big Data and APIsfor PHP Developers SXSW Interactive 2011 Austin, Texas #BigDataAPIs

Text REDCROSS to 90999 to make a $10 donation andsupport the American Red Cross disaster relief efforts to helpthose affected by the earthquake in Japan and tsunamithroughout the Pacific.

Topics & GoalTopics: o Introductions o Definition of Big Data o Working with Big Data o APIs o Visualization o MapReduce Goal: To provide an interesting discussion revolving around allaspects of big data & to spark your imagination on the subject.

Who we are

Bradley Holt (@bradleyholt)Curator of this WorkshopCo-Founder and Technical Director,Found LineAuthor

Julie Steele (@jsteeleeditor)Pythonista (but we like her anyway)Acquisitions Editor, OReilly MediaGraphic Designer, freelanceVisualization curator

Laura Thomson (@lxt)Webtools Engineering Manager,Mozillacrash-stats.mozilla.comPHP, Python, Scaling, Systems

Eli White (@eliw)Worked for: Digg, Tripadvisor, HubblePHP guy at heartCurrently unemployed (Hiring?)Author:

Dennis Yang (@sinned)Director of Product & Marketing, InfochimpsPreviously: mySimon, CNET, & cofounder of Techdirt

David Zuelke (@dzuelke)Lead Developer: AgaviManaging Partner, Bitextender GmbHAddicted to rockets, sharks w/friggin laser beams, helicopters,HTTP, REST, CouchDB andMapReduce. And PHP.

Who are you?Lets Learn about the Audience!

Tell Us About You• Who is currently working on a Big Data problem?• Have you integrated with an existing API?• Do you publish your own API?• How many of you are PHP developers?• Tell us what you hope to learn here today.

What is Big Data?

Different Types of Big DataLarge chunks of data o Massive XML files, Images, Video, log filesMassive amounts of small data points o Typical Web Data, Survey votes, TweetsRequests/Traffic o Serving lots of data, regardless of set sizeProcessing vs storing o Different concerns if only processing versus storing

Working with Big Data

CAP TheoremConsistency All clients will see a consistent view of the data.Availability Clients will have access to read and write data.Partition Tolerance The system wont fail if individual nodes cant communicate.You cant have all three—pick two!http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf

Common Tools Hadoop and HBase Cassandra Riak MongoDB CouchDB Amazon Web Services

Scaling Data vs. Scaling RequestsHighly related? (or are they?)At Digg, both were related, as more requests meant more data,and more data made it harder to handle more requests.At TripAdvisor, it was handling pass through big data, we neverstored it, but had to process it quickly.Mozillas Socorro has a small number of webapp users (100?),but catches 3 million crashes a day via POST, median size150k, and must store, process, and analyze these.

OLTP vs. OLAPOLTP: low latency, high volumeOLAP: long running, CPU intensive, low volume (?)Trying to do both with the same system happens more oftenthan you might think (and will make you cry)One bad MapReduce can bring a system to its knees

Online vs offline solutionsKeep only recent data in online systemReplicate / snapshot data to secondary systemRun expensive jobs on secondaryCache output of expensive jobs in primaryMake snapshot/window data available to end users for ad hocprocessing

Case Study: Socorro

Whats Socorro?

Scaling upA different type of scaling:Typical webapp: scale to millions of users without degradationof response timeSocorro: less than a hundred users, terabytes of data.Basic law of scale still applies:The bigger you get, the more spectacularly you fail

Some numbersAt peak we receive 3000 crashes per minute3 million per dayMedian crash size 100k -> 150k30TB stored in HBase and growing every day

What can we do?Does betaN have more (null signature) crashes than otherbetas?Analyze differences between Flash versions x and y crashesDetect duplicate crashesDetect explosive crashesEmail victims of a malware-related crashFind frankeninstalls

War storiesHBase Low latency writes Fast to retrieve one record or a range, anything else MRStability problems and solutions Big clusters: network is key Need spare capacity, secondary systems Need great instrumentation Redundant layers (disk, RDBMS, cache) Next: Universal APIBig data is hard to move

APIs for Public Datasets

Sources for public data• Data.gov• DataSF: http://datasf.org/• Public Data Sets on AWS: http://aws.amazon.com/publicdatasets/• UNData: http://data.un.org/• OECD: http://stats.oecd.org/

InfochimpsOver 10,000 data sets listed at InfochimpsOver 1,800 data sets available through our APIAPIs allow easy access to terabyte scale data

Data is the lifeblood of your productAny great application consists of:• Awesome code• The right dataAnd, as weve seen with sites like CNET, NPR, Netflix & Twitter,it has become a best practice to build applications against APIsto access that data.

Infochimps Screennames AutocompleteAPIGET http://www.infochimps.com/datasets/twitter‐screen‐name‐autocomplete?prefix=infochi&apikey=api_test-W1cipwpcdu9Cbd9pmm8D4Cjc469{"completions":["infochimps","InfoChile","infochiapas","infochilecompra","Infochick","infochimp","infoChief1","infochip2","infochild","infochiocciola",

How we make the ScreennameAutocomplete API• Flying Monkey Scraper o continually crisscrosses the user graph, to discover new users o 1B objects to insert a day, 8 nodes• Hadoop o To do the processing, 15+ nodes o Pig, Wukong o Precalculate 100M usernames -> prefixes -> a few hundred million rows o Sorted by Trstrank• Apeyeye o load balanced cluster of 10 nodes, across 2 data centers

Infochimps Yahoo Stock APIGET http://api.infochimps.com/economics/finance/stocks/y_historical/price_range{"results":[{"open":19.1,"adj_close":9.55,"close":19.09,"high":19.33,"symbol":"AAPL","date":20010813,"exchange":"NASDAQ","volume":5285600,"low":18.76},

How we make the Stock APIChanges every day...You can get Yahoo stock data, every day too, in CSV form.• Hackbox: o little piece of armored sausage that takes the data and munges it up to be useful• Troop: o Publishes it into the datastore o Writes the API docs o Puts the API endpoint into the code o Stages for deploy

Build these APIs yourself.. if you want.You can build these APIs yourself.Check out http://infochimps.com/labs to see our opensourced codebase.Or, you can let us do it, and you can focus on writing awesomecode for your application, and let us do the monkeying with thedata.

And actually....These two APIs illustrate great reasons why you *wouldnt*want to build them yourself:• The data involved is too large to practically handle• The data involved updates frequently, so its a hassle

Examples of data available through theInfochimps API• Trstrank o How trustworthy is a Twitter user?• Conversations o Given two users, get a summary of their interactions• Twitter Name Autocomplete o Given a string, find twitter names that complete it• Geo Name Autocomplete o Given a string, find place names that complete it• Qwerly o maps from Twitter handle to other social networks• Daily Historical Stock Quotes• IP to Census o Given an IP address, map to Geo, then map to US Census info

And many more APIs...Many more APIs available:• Word Frequencies• Word Lists• Freebase• DBPedia• AggData• Material Safety Data Sheets (MSDS)• Internet "Weather" from CedexisAnd... many more to come.Relocate your subroutine to the cloud.http://infochimps.com/apis

Developing a Big Data API

Rules of thumbGenerate APIs that only give incremental access, dont shovemore data than needed at a user.For performance reasons, dont allow the user to request toomuch data in one request, and throttle requests.Consider building API against secondary system.Asynchronous APIs: request/queue data, pick it up later.Caches for big data dont necessarily help.

Visualization

Exploring vs. Explaining L. fineartamerica.com/featured/exploring-archimedes-david-robinson.html R. sgp.undp.org/web/projects/10771/environmental_awareness_and_familiarity_with_animals.html

Structure has a purpose. fiveless.deviantart.com/art/Periodic-Table-of-the-Elements-147350318

squidspot.com/Periodic_Table_of_Typefaces.html

michaelvandaniker.com/labs/browserVisualization/

Firefox rulz

It hurts. Make it stop.

Chart junk makes baby Jesus cry.L. www.flickr.com/photos/santo_cuervo/3693877386/R. www.gereports.com/a-good-look-at-the-cost-of-chronic-diseases/

http://jec.senate.gov/republicans/public/index.cfm?p=CommitteeNews&ContentRecord_id=bb302d88-3d0d-4424-8e33-3c5d2578c2b0

www.flickr.com/photos/robertpalmer/3743826461/

It deosnt mttaer waht oredr the ltteers in a wrodare, the olny iprmoetnt tihng is taht the frist andlsat ltteres are at the rghit pclae. The rset can be atatol mses and you can sitll raed it wouthit aporbelm. Tihs is bcuseae we do not raed erveylteter by it slef but the wrod as a wlohe.IT DEOSNT MTTAER WAHT OREDR THE LTTEERS INA WROD ARE, THE OLNY IPRMOETNT TIHNG IS TAHTTHE FRIST AND LSAT LTTERES ARE AT THE RGHITPCLAE. THE RSET CAN BE A TATOL MSES AND YOUCAN SITLL RAED IT WOUTHIT A PORBELM. TIHS ISBCUSEAE WE DO NOT RAED ERVEY LTETER BY ITSLEF BUT THE WROD AS A WLOHE.

The Stroop Effect RED YELLOW BLUE GREEN RED YELLOW BLUE GREEN

Color: function vs. decorationMartin Wattenberg and Fernanda Viégas, Chapter 11, Beautiful Visualization (OReilly Media)

Andrew Odewahn, Chapter 8, Beautiful Visualization (OReilly Media)

Be kind to the color blind courses.washington.edu/info424/Labs/ChoroplethMap.html

Cool tools to try• ManyEyes www-958.ibm.com/software/data/cognos/manyeyes/• Wordle www.wordle.net/• GraphViz www.graphviz.org/• Protovis vis.stanford.edu/protovis/• Tableau www.tableausoftware.com/

Your turn!datavis.tumblr.com/post/2746708037/the-sequel-map

MapReduceOr: How to even store and process that much data in the first place?

how much is "that much"?

Some Numbers

Some NumbersFacebook, new data per day:

Some NumbersFacebook, new data per day:• 03/2008: 200 GB

Some NumbersFacebook, new data per day:• 03/2008: 200 GB• 04/2009: 2 TB

Some NumbersFacebook, new data per day:• 03/2008: 200 GB• 04/2009: 2 TB• 10/2009: 4 TB

Some NumbersFacebook, new data per day:• 03/2008: 200 GB• 04/2009: 2 TB• 10/2009: 4 TB• 03/2010: 12 TB

Some NumbersFacebook, new data per day: Googles processing jobs:• 03/2008: 200 GB• 04/2009: 2 TB• 10/2009: 4 TB• 03/2010: 12 TB

Some NumbersFacebook, new data per day: Googles processing jobs:• 03/2008: 200 GB 400 PB per month (in 2007!)• 04/2009: 2 TB• 10/2009: 4 TB• 03/2010: 12 TB

Some NumbersFacebook, new data per day: Googles processing jobs:• 03/2008: 200 GB 400 PB per month (in 2007!)• 04/2009: 2 TB Average job size is 180 GB• 10/2009: 4 TB• 03/2010: 12 TB

what if you have this much data?

what if its just 1% of what Facebook has to deal with?

no problemo, you say?

reading 180 GB off a hard disk will take ~45 minutes

and then you havent even processed it yet!

todays computers process data way faster than it can be read

solution: parallelize your I/O

but now you need to coordinate

thats hard

what if a node dies?

what if a node dies?does the whole job have to re-start?

what if a node dies?does the whole job have to re-start? can another node take over?

what if a node dies?does the whole job have to re-start? can another node take over? how do you coordinate this?

Enter: Our HeroMapReduce to the Rescue!

olden days

distribute workload across a grid

ship data between nodes

store it centrally on a SAN

I/O bottleneck

MapReduce: Simplified Data Processing on Large Clusters http://labs.google.com/papers/mapreduce.html

distribute the data up front across nodes

then ship computing nodes to where the data is

share-nothing architecture

scaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalable

Basic Principle: The Mapper

Basic Principle: The MapperMapper reads records and emits key and value pairs.

Basic Principle: The MapperMapper reads records and emits key and value pairs.Take an Apache web server log file as an example:

Basic Principle: The MapperMapper reads records and emits key and value pairs.Take an Apache web server log file as an example:• Each line is a record.

Basic Principle: The MapperMapper reads records and emits key and value pairs.Take an Apache web server log file as an example:• Each line is a record.• Mapper extracts request URI and number of bytes sent.

Basic Principle: The MapperMapper reads records and emits key and value pairs.Take an Apache web server log file as an example:• Each line is a record.• Mapper extracts request URI and number of bytes sent.• Mapper emits the URI as the key and the bytes as the value.

Basic Principle: The MapperMapper reads records and emits key and value pairs.Take an Apache web server log file as an example:• Each line is a record.• Mapper extracts request URI and number of bytes sent.• Mapper emits the URI as the key and the bytes as the value.Parallelize by having log files per hour, splitting up the files intoeven smaller chunks (by line) and so forth.

Basic Principle: The Reducer

Basic Principle: The ReducerAll values (from all nodes) for the same key are sent to thesame reducer.

Basic Principle: The ReducerAll values (from all nodes) for the same key are sent to thesame reducer.Keys get sorted, and in case of a simple count or sum, you cando a first reduce run on each mapper node once its finished tocut down on I/O (thats the combiner).

Basic Principle: The ReducerAll values (from all nodes) for the same key are sent to thesame reducer.Keys get sorted, and in case of a simple count or sum, you cando a first reduce run on each mapper node once its finished tocut down on I/O (thats the combiner).Apache web server log example to the rescue again:

Basic Principle: The ReducerAll values (from all nodes) for the same key are sent to thesame reducer.Keys get sorted, and in case of a simple count or sum, you cando a first reduce run on each mapper node once its finished tocut down on I/O (thats the combiner).Apache web server log example to the rescue again:• Reducer is invoked for a URI like "/foobar" and a list of all number of bytes.

Basic Principle: The ReducerAll values (from all nodes) for the same key are sent to thesame reducer.Keys get sorted, and in case of a simple count or sum, you cando a first reduce run on each mapper node once its finished tocut down on I/O (thats the combiner).Apache web server log example to the rescue again:• Reducer is invoked for a URI like "/foobar" and a list of all number of bytes.• Sum up the bytes, and we have the total traffic per URI!

Hello, HadoopMapReduce for the Masses

Hadoop is a MapReduce framework

comes with a distributed FS, task tracker and so forth

so we can focus on writing MapReduce jobs

works quite well, too

Big Hadoop Installations

Big Hadoop InstallationsFacebook:

Big Hadoop InstallationsFacebook:• Mostly used with Hive

Big Hadoop InstallationsFacebook:• Mostly used with Hive• 8400 cores, 13 PB total storage capacity

Big Hadoop InstallationsFacebook:• Mostly used with Hive• 8400 cores, 13 PB total storage capacity o 8 cores, 32 GB RAM, 12 TB disk per node

Big Hadoop InstallationsFacebook:• Mostly used with Hive• 8400 cores, 13 PB total storage capacity o 8 cores, 32 GB RAM, 12 TB disk per node o 1 GbE per node

Big Hadoop InstallationsFacebook:• Mostly used with Hive• 8400 cores, 13 PB total storage capacity o 8 cores, 32 GB RAM, 12 TB disk per node o 1 GbE per node• 4 GbE between racks

Big Hadoop InstallationsFacebook: Yahoo:• Mostly used with Hive• 8400 cores, 13 PB total storage capacity o 8 cores, 32 GB RAM, 12 TB disk per node o 1 GbE per node• 4 GbE between racks

Big Hadoop InstallationsFacebook: Yahoo:• Mostly used with Hive • 40% of jobs use Pig• 8400 cores, 13 PB total storage capacity o 8 cores, 32 GB RAM, 12 TB disk per node o 1 GbE per node• 4 GbE between racks

Big Hadoop InstallationsFacebook: Yahoo:• Mostly used with Hive • 40% of jobs use Pig• 8400 cores, 13 PB total • > 100,000 CPU cores in storage capacity > 25,000 servers o 8 cores, 32 GB RAM, 12 TB disk per node o 1 GbE per node• 4 GbE between racks

Big Hadoop InstallationsFacebook: Yahoo:• Mostly used with Hive • 40% of jobs use Pig• 8400 cores, 13 PB total • > 100,000 CPU cores in storage capacity > 25,000 servers o 8 cores, 32 GB RAM, 12 • Largest cluster: 4000 TB disk per node nodes o 1 GbE per node• 4 GbE between racks

Big Hadoop InstallationsFacebook: Yahoo:• Mostly used with Hive • 40% of jobs use Pig• 8400 cores, 13 PB total • > 100,000 CPU cores in storage capacity > 25,000 servers o 8 cores, 32 GB RAM, 12 • Largest cluster: 4000 TB disk per node nodes o 1 GbE per node o 2 x 4 CPU cores and 16 GB RAM per node• 4 GbE between racks

Hadoop at Facebook

Hadoop at FacebookDaily usage:

Hadoop at FacebookDaily usage:• 25 TB logged by Scribe

Hadoop at FacebookDaily usage:• 25 TB logged by Scribe• 135 TB compressed data scanned

Hadoop at FacebookDaily usage:• 25 TB logged by Scribe• 135 TB compressed data scanned• 7500+ Hive jobs

Hadoop at FacebookDaily usage:• 25 TB logged by Scribe• 135 TB compressed data scanned• 7500+ Hive jobs• ~80k compute hours

Hadoop at FacebookDaily usage: Data per day growth:• 25 TB logged by Scribe• 135 TB compressed data scanned• 7500+ Hive jobs• ~80k compute hours

Hadoop at FacebookDaily usage: Data per day growth:• 25 TB logged by Scribe • I/08: 200 GB• 135 TB compressed data scanned• 7500+ Hive jobs• ~80k compute hours

Hadoop at FacebookDaily usage: Data per day growth:• 25 TB logged by Scribe • I/08: 200 GB• 135 TB compressed data • II/09: 2 TB compressed scanned• 7500+ Hive jobs• ~80k compute hours

Hadoop at FacebookDaily usage: Data per day growth:• 25 TB logged by Scribe • I/08: 200 GB• 135 TB compressed data • II/09: 2 TB compressed scanned • III/09: 4 TB compressed• 7500+ Hive jobs• ~80k compute hours

Hadoop at FacebookDaily usage: Data per day growth:• 25 TB logged by Scribe • I/08: 200 GB• 135 TB compressed data • II/09: 2 TB compressed scanned • III/09: 4 TB compressed• 7500+ Hive jobs • I/10: 12 TB compressed• ~80k compute hours

HDFSHadoop Distributed File System

HDFS Overview

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)• Designed for streaming rather than random reads

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)• Designed for streaming rather than random reads• Write-once, read-many (although there is a way to append)

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)• Designed for streaming rather than random reads• Write-once, read-many (although there is a way to append)• Stores data redundantly (three replicas by default), is aware of your network topology

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)• Designed for streaming rather than random reads• Write-once, read-many (although there is a way to append)• Stores data redundantly (three replicas by default), is aware of your network topology• Namenode has metadata and knows where blocks reside

HDFS Overview• Designed for very large data sets, transparent compression, block-based storage (64 MB block size by default)• Designed for streaming rather than random reads• Write-once, read-many (although there is a way to append)• Stores data redundantly (three replicas by default), is aware of your network topology• Namenode has metadata and knows where blocks reside• Datanodes hold the data

Task ProcessingHow Hadoop Gets the Job Done

Job Processing

Job Processing• Input Formats split up your data into individual records

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts• Combiner can perform local pre-reduce on each mapper

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts• Combiner can perform local pre-reduce on each mapper• Reducers perform reduction for each key

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts• Combiner can perform local pre-reduce on each mapper• Reducers perform reduction for each key• Mapper, Combiner and Reducer can be an external process

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts• Combiner can perform local pre-reduce on each mapper• Reducers perform reduction for each key• Mapper, Combiner and Reducer can be an external process o Called Hadoop Streaming, uses STDIN & STDOUT

Job Processing• Input Formats split up your data into individual records• Mappers do their work, then a partitioner partitions & sorts• Combiner can perform local pre-reduce on each mapper• Reducers perform reduction for each key• Mapper, Combiner and Reducer can be an external process o Called Hadoop Streaming, uses STDIN & STDOUT  Shameless plug: http://github.com/dzuelke/hadoophp

</BigDataAPIs>

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com• Julie Steele, @jsteeleeditor, jsteele@oreilly.com

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com• Julie Steele, @jsteeleeditor, jsteele@oreilly.com• Laura Thomson, @lxt, laura@mozilla.com

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com• Julie Steele, @jsteeleeditor, jsteele@oreilly.com• Laura Thomson, @lxt, laura@mozilla.com• Eli White, @eliw, eli@eliw.com

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com• Julie Steele, @jsteeleeditor, jsteele@oreilly.com• Laura Thomson, @lxt, laura@mozilla.com• Eli White, @eliw, eli@eliw.com• Dennis Yang, @sinned, dennis@infochimps.com

</BigDataAPIs>• Bradley Holt, @bradleyholt, bradley.holt@foundline.com• Julie Steele, @jsteeleeditor, jsteele@oreilly.com• Laura Thomson, @lxt, laura@mozilla.com• Eli White, @eliw, eli@eliw.com• Dennis Yang, @sinned, dennis@infochimps.com• David Zuelke, @dzuelke, david.zuelke@bitextender.com

http://lanyrd.com	22
http://stg.lanyrd.org	4
http://paper.li	1

Big data and APIs for PHP developers - SXSW 2011

by Eli White on Mar 14, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 27

Statistics

Big data and APIs for PHP developers - SXSW 2011 — Presentation Transcript