Tour of Big Data
•Download as PPTX, PDF•
0 likes•713 views
Presentation at Southern California Code Camp July 2013 in San Diego. This talk presents you with basic concepts in world of big data and data science, with focus on relational databases, noSQL, MapReduce, machine learning, and data visualization, along with demos of MapReduce in action and Pig on Hadoop. The purpose of this presentation is to get you familiar with terminologies and concepts in data science, and whet your appetite for further exploration into the world of big data. This presentation is adapted from an online course by Coursera with similar title and scope
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Tour of Big Data
Similar to Tour of Big Data (20)
Recently uploaded
Recently uploaded (20)
Tour of Big Data
- 1. Tour of Big Data Raymond Yu Socal Code Camp 2013
- 2. About myself •Sr. Database Architect @ BridgePoint Education •Blog www.yutechnet.com •LinkedIn www.linkedin.com/in/raymondyu1 •@yutechnet
- 3. About this talk… 7/28/2013yutechnet.com •Inspired by “Introduction to Data Science” on Coursera (Bill Howe, UW) •Guided tour of topics in data science – MapReduce, Pig – noSQL – Machine Learning – Information Visualization •Goal
- 4. Big Data •Volume – Size of data •Velocity – The latency of data processing relative to the growing demand of interactivity •Variety – The diversity of sources, formats, quality, and structures Big Data is any data that is expensive to manage and hard to extract value from. -Michael Franklin
- 5. Where does big data come from? •“Data exhaust” from customers •New censor technologies •Individually contributed data in massive scale •Cheap to keep data
- 6. Data Science •Data Preparation (at scale) •Analytics •Communication The ability to take data, understand it, process it, extract value from it, visualize it, and communicate it - Hal Varian, Google's Chief Economist
- 7. Context… src. Introduction to Data Science course
- 8. Relational Databases •SQL as Declarative Language •Indexes – Extract small result from big dataset – Built easily and automatically used when appropriate •Data consistency •“Old-style” scalability
- 9. MapReduce •Google paper 2004 •Hadoop 2008 •High level programming model for large- scale parallel data processing •Divide-and-conquer •Mapper + Reducer
- 10. “Hello World” of MapReduce Count word frequency in millions of documents
- 11. MapReduce Programming Model src. Course slide
- 12. Show me the MapReduce… •www.jsmapreduce.com
- 14. Pig • An engine to execute programs on top of Hadoop • Language layer Pig Latin • An Apache open source project (http://pig.apache.org) •Yahoo! 2009
- 15. Why use Pig?
- 16. In MapReduce…
- 17. In Pig Latin
- 19. Context… src. Introduction to Data Science course
- 20. noSQL definitions •A term to designate databases which differ from classic relational databases – Transactional model – Data model •Not much to do with SQL •“not only SQL”
- 21. Concepts • CAP Theorem – Consistency – Availability – Partition Tolerance • Eventual consistency Src: blog.beany.co.kr
- 23. Let’s walk through a few •Column definitions •RDBMS •Memcache •Dynamo •CouchDB •BigTable (Hbase)
- 24. noSQL Common Features • The ability to replicate and partition data over many servers (scale) • Horizontally scale simple operation throughput over many servers • A simple API - no query language (no SQL) • Weaker concurrency model than ACID transactions (no transaction) • The ability to dynamically add new attributes to data records (no schema)
- 25. Machine Learning • Systems that automatically learn programs from data • Prediction – Given examples of inputs and outputs – Learn the relationship between them – Apply the relationship to larger set • Different from statistics model – Large data set over simple model trumpets small data set over sophisticated model
- 30. Closing example Src. http://commons.wikimedia.org/wiki/File:ElectoralCollege2012.svg Nate Silver fivethirtyeight.com Obama’s Data- Driven Campaign • Massive voter db • Hadoop as ETL • Vertica db for slice- and-dice
- 31. Questions?
Editor's Notes
- Whenever you see “yutechnet”, it is me Next ask audience about:Developer? DBA? DBE?Worked on any databases beyond relational database?Use Hadoop and other noSQL on a daily basis?
- Dummy down version of the courseHard to pick topics to shareMajor areas of data science, focus on big data and noSQLGoal: familiar with the big picture and terminologies of data science and speak intelligently about this field, springboard into specific areas you are further interested in
- Franklin’s key idea: “Big” is relative, it depends on what you try to do
- Analytics: statistics model, machine learning, slice-dice
- Call out a few great features about relational databases to set the context of how we got here, and we don’t get lost in the context of big data and noSQL, with bad name/impression as old guardDeclarative – specify what you want, no need to worry about logical or physical operation and optimization
- Map, shuffle, and reduce.
- Touch base on HDFS layer about fault tolerance, job tracker, task tracker, etc.
- Comments in lieu of demo:schema-on-read with LOADRelational JOIN operationOptimization – relational algebraLazy evaluation – no work is done until STORE
- Pig performance: initially not good as MR, but caught up quickly, now almost the same as MRHive not covered, but 2011 data showed that >90% MR jobs are executed via HiveClear win for a declarative languageDon’t feel bad if you know SQL
- About EC:Databases: “Everyone MUST see the same thing, either old or new, no matter how long it takes.”NoSQL: “For large applications, we can’t afford to wait that long, and maybe it doesn’t matter anyway”
- Memcache: load everything into memory, and scale across hundreds of machines, consistent hashingBigTable – Google 2006, complementary to MapReduce, added index (zoom-in), fast key-based lookup
- Statistics emphasizes accuracy of model, while ML cares less about the nature of modelThink of the example of building a super-accurate gun