Big data for the rest of us

Not Just HadoopNoSQL in the Enterprise

Talking aboutWhat is BIG DataBIG Data & youReal world examplesThe future of Big Data

@spf13 AKASteve Francia16+ years buildingthe internet Father, husband, skateboarderChief Evangelist @responsible for drivers,integrations, web & writing

What isBIGdata ?

2000Google IncToday announced it has releasedthe largest search engine on theInternet.Google’s new index, comprisingmore than 1 billion URLs

2008Our indexing system for processinglinks indicates thatwe now count 1 trillion unique URLs(and the number of individual webpages out there is growing byseveral billion pages per day).

An unprecedentedamount of data isbeing created and isaccessible

Data Growth 1,0001000 750 500 500 250 250 120 55 4 10 24 1 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 Millions of URLs

Truly Exponential GrowthIs hard for people to graspA BBC reporter recently: "Your current PCis more powerful than the computer theyhad on board the ﬁrst ﬂight to the moon".

Moore’s LawApplies to more than just CPUsBoiled down it is that things double atregular intervalsIt’s exponential growth.. and applies tobig data

How BIG is it?

How BIG is it?2008

How BIG is it? 20072008 2005 2006 2003 2004 2001 2002

We’ve had BIG Data needs for a long timeIn 1998 Google won the search racethrough custom software & infrastructure

We’ve had BIG Data needs for a long timeIn 2002 Amazon again wrote custom &proprietary software to handle their BIGData needs

We’ve had BIG Data needs for a long timeIn 2006 Facebook started with off theshelf software, but quickly turned todeveloping their own custom builtsolutions

Ability to handle big data isone of the largest factors indetermining winners vslosers.

For over a decade BIG Data =custom software

Why all thistalk about BIG Data now?

In the past fewyears open sourcesoftware emergedenabling ‘us’ tohandle BIG Data

The Big Data Story

Is actuallytwo stories

Doers & Tellers talking about different things http://www.slideshare.net/siliconangle/trendconnect-big-data-report-september

Tellers

Doers talk a lot more about actual solutions

They know it’s a two sided story Storage Processing

Take awaysMongoDB and HadoopMongoDB for storage &operationsHadoop for processing &analytics

How MongoDB enables big data•Flexible schema• Horizontal scale built in & free•Operates at near speed of memory• Optimized for modern apps

MongoDB @ Orbitz Rob Lancaster October 23 | 2012

Use Cases • Hotel Data Collection • Hotel Rate Feed: • Supply hotel rates to Google for their Hotel Finder • Uses MongoDB: - Maintain state of data sent to Google - Identify changes in rates as they occur • Makes use of complex querying, secondary indexing • EasyLoader: • Feature allowing suppliers to easily load inventory to Orbitz • Uses MongoDB to persist all changes for auditing purposes 29

Hotel Data Collection • Goals: • Better understand performance of our hotel path • Optimize our hotel rate cache • Methods: • Capture every action performed by our hotel search engine. • Persist this data for long periods. • Challenges: • Need high performance capture. • Scalable, inexpensive storage. 30

Requirements Collection Storage & Processing • High write throughput • High data volume • 500 servers • ~500 GB/day • > 100 million documents/day • 7 TB/month compressed • Flexibility • Scalable • Complex extendable documents • Inexpensive • No forced schema • Proximity with other data • Scalability • Simplicity 31

The Solution • Utilize MongoDB as a collector: • ~ 500 clients • Utilize unsafe writes for high throughput • Heterogeneous documents • New collection for each hour • HDFS for storage & processing: • Data moved via M/R job: - One job per collection - One mapper per MongoDB instance • Additional processing and analysis by other jobs 32

Challenges & Conclusions •Challenges? None really. •Achieved a robust and simple solution •MongoDB has been entirely worry free • Very high write throughput • Reads (well, full collection dumps across the wire) are slower 33

The Futureof BIG data

What is BIG? BIG today isnormal tomorrow

Data Growth 9,00090006750 4,4004500 2,1502250 1,000 500 55 120 250 1 4 10 24 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Millions of URLs

How BIG is it?

How BIG is it?2012

How BIG is it? 20112012 2009 2010 2007 2008 2005 2006

2012Generating over250 Millions oftweets per day

MongoDB enables us to scalewith the redeﬁnition of BIG.Tools like Hadoop areenabling us to process thenew BIG.

MongoDB iscommitted to working with best data tools including Hadoop, Storm,Disco, Spark & more

http://spf13.com http://github.com/s @spf13Question download at mongodb.orgWe’re hiring!! Contact us at jobs@10gen.com

http://spf13.com	547
http://feeds.feedburner.com	11
http://localhost	6
http://www.diffbot.com&_=1351805064729 HTTP	1
http://core.traackr.com	1

Big data for the rest of us

by Steve Francia on Oct 29, 2012

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 566

Statistics

Big data for the rest of us Presentation Transcript