• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Gail Zhou on "Big Data Technology, Strategy, and Applications"
 

Gail Zhou on "Big Data Technology, Strategy, and Applications"

on

  • 415 views

Dr. Gail Zhou presented this topic at DevNexus on Feb 25, 2014. Big Data history, opportunities, and applications. Big Data key concepts, reference architecture with open source technology stacks. ...

Dr. Gail Zhou presented this topic at DevNexus on Feb 25, 2014. Big Data history, opportunities, and applications. Big Data key concepts, reference architecture with open source technology stacks. Hadoop architecture explained (HDFS, Map Reduce, and YARN). Big Data start-up challenges and strategies to overcome them. Technology update: Hadoop and Cassandra based technology offerings.

Statistics

Views

Total Views
415
Views on SlideShare
414
Embed Views
1

Actions

Likes
1
Downloads
9
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Gail Zhou on "Big Data Technology, Strategy, and Applications" Gail Zhou on "Big Data Technology, Strategy, and Applications" Presentation Transcript

    • DevNexus 2014, Data + Integration Big Data Technology, Strategy, and Applications Dr. Gail Zhou Gail Z Associates, LLC February 25, 2014 LinkedIn: http://www.linkedin.com/in/gailZhou Email: gail.r.zhou@gmail.com Gail Z Associates, LLC
    • Outline •What is Big Data and why is it such a big deal? Where can we use Big Data? • Big Data Key Concepts and Technologies using Hadoop as an example •Big Data Challenges and Start up Strategy: What are the challenges? How do you get started on Big Data? Appendix: Other Big Data Technologies, Integration of Big Data with Existing Applications (an example) 2 Gail Z Associates, LLC
    • What is Big Data and why is it such a big deal? 3 Gail Z Associates, LLC
    • A Brief History of Big Data Sources: Wikipedia, Forbes.com, and other articles • 1941: “Information Explosion” term coined. • 1963: Physicist and science historian Derek Price concluded the number of new journals grown exponentially. • 1990: Computer Scientist Peter J. Denning, “Saving All the Bits”, what machines can we build to monitor, process, and understand the data, its meanings, and patterns? – Intelligence out of the data? • 1998: Steve Bryson et all, “Visually exploring gigabyte data sets in real time”, ACM, Section “Big Data for Scientific Visualization”. 4 Gail Z Associates, LLC
    • A Brief History of Big Data Cont’d Sources: Wikipedia, Forbes.com, and other articles • 2001, Doug Laney, Meta Group, “3D Data Management, Controlling Data Volume, Velocity, and Variety” (More now: Veracity, Variability, and Value) 5 Gail Z Associates, LLC
    • A Brief History of Big Data Cont’d Sources: Wikipedia, Forbes.com, and other articles • 2001 - 2003: Google outgrown as a result of new revenue model, 5 cents per click. Google is now a giant big data leader. • 1994 – Present: Yahoo!, Hadoop Shop (10K Nodes), Genome, Big Data Analytics. • 1994 – Present: Amazon, AWS Cloud. • 2003 – Present: Facebook, Twitter, LinkedIn, etc. • 2013 and beyond : Many others. 6 Gail Z Associates, LLC
    • 7 Gail Z Associates, LLC
    • Population Growth Chart: Does it have something to do with Big Data? Machines, Satellites, Cameras, Internet, computers, and mobile phones are just “enablers” of big data. Source: Global Education Project 8 Gail Z Associates, LLC
    • Source: Newbury College, UK www.spchui.net Information Explosion. It is just the real beginning. You got mail (too much). You are embarrassed to admit you don’t know a lot of cool things happening in the world. www.ucg.org Don’t despair. You are not alone. 9 Gail Z Associates, LLC
    • Big Data Opportunities 10 Gail Z Associates, LLC
    • Big Data Opportunities • Medical Research and Healthcare: Massive collected research and clinical information can be used to predict and prevent diseases, moving us from ‘sick care’ to ‘health care’. • Telecom: Traffic data and patterns can be utilized in real time to re-route. • Defense: Satellite images and other information can be meshed up to identify threats. • Utilities: Smart meter monitoring. • Public Safety: Pattern recognition and social media can help to predict crimes. • Financial Industry: Patten recognition and business rules to flag fraudulent activities. • Functional Areas: Investigational Search, Pricing Optimization, Risk Analysis, Churn Analysis, Behavior Analysis, Transactions Analysis, Revenue Assurance, Recommendation Engines, etc. 11 Gail Z Associates, LLC
    • Where Big Data Can Shine • Traditional (Examples)  Financial Transactions  Energy and Infrastructure  Transportation  Life Science and HealthCare •Big Data (Examples) Advertisements Search and Indexing Social Networks Science Research Communications • Notes – Big Data Technology is not the replacement – Big Data is complementary – In some cases, Big Data is the only way to get things done – Big Data has its own challenges 12 Gail Z Associates, LLC
    • Key Concepts in Big Data – Technology and Architectures 13 Gail Z Associates, LLC
    • 14 Gail Z Associates, LLC
    • 15 Gail Z Associates, LLC
    • Hadoop HDFS Blocks (64M, 128M, etc.) are saved in different nodes with a replication factor ( default 3) 16 Gail Z Associates, LLC
    • Hadoop Logical View http://nosqlessentials.com Professor: Fernando Rodriguez Olivera 17 Gail Z Associates, LLC
    • Hadoop Logical View (HDFS + Map Reduce) 18 Gail Z Associates, LLC
    • Hadoop V1 – Map Reduce Jobs Execution 19 Gail Z Associates, LLC
    • Hadoop 2.0 with YARN Gail Z Associates, LLC
    • YARN Interaction & Sequence Gail Z Associates, LLC
    • Big Data Challenges, Suggested Startup Strategy 22 Gail Z Associates, LLC
    • Big Data Start up Challenges  Business urgency, time to market pressures  Big Data start up needs careful planning  Big Data needs infrastructure, software stacks, people, start up plan  Lack of Big Data Resources, Lack of Sponsorships (except in some companies)  Big Data is complex and multiple skill sets (mostly new to many companies) – Infrastructure, Administration, Security, Programming, Testing, etc.  Skepticism about Big Data  Integration with Existing Technologies and Systems  Can not develop isolated big data solutions  Integration with existing systems will be a top challenge (requires both sides to do additional work)  Open Sources: Stability, Maturity, and Security Gail Z Associates, LLC
    • Suggested Big Data Start up Strategy  Full business needs and information requirements analysis. Business Drivers  Revenue generation? Cost reduction? Customer retention? Compliance?  Process Improvement? Fraud detection? Analytics? Dashboard?  Solving a tough problem? Retiring/replacing technologies and systems?  Technology Evaluation and Selection  Define requirements and objective first  Evaluation a variety of technology stacks – develop a framework first  Executive Support for Start up Resources  Prototyping, Discovery, and Planning  Rent Infrastructure in Cloud – VMWare, Amazon EC2, and others  Use Spare Hardware and Network Bandwidth  Assessment, Proposal. Project/Program Plan for next steps  Start small and keep delivering  Architecture Design, Estimation, Business Case  Obtain funding and executive sponsorships, owners, etc.  SDLC, don’t forget Hardware, Security, Testing, etc. Gail Z Associates, LLC
    • Appendix 25 Gail Z Associates, LLC
    • Hadoop & Cassandra Based Offerings Name Offerings Notes Apache Hadoop Hadoop Core Enhancement: YARN Cloudera Enhanced Hadoop Leader DataStax Enhanced Apache Cassandra Cassandra is a distributed NoSQL DB Hortonworks Hadoop Development and support. Hortonworks Data Platform (HDP) Yahoo Funded $23M + Others . Major alliances. MapR Develops and sells Hadoop-derived software. M3. M5, M7. Alliance with EMC, Amazon, and Google. Sqoop HDFS and SQL Integration Hue Hadoop GUI Tools Amazon AWS, Cloud Hadoop Cluster Microsoft Windows Azure HDInsight IBM, Dell, etc. Hardware, Software, Services Gail Z Associates, LLC
    • Hadoop Related Technologies (Examples) Name Functions Notes Apache Hue Hadoop GUI Hadoop has cmd. Apache HBase NoSQL Distributed DB, Key/value Column Family Store, runs on top of Hadoop Big Table Like Storage for Hadoop, written in Java. Apache PIG High Level programming language for Map Reduce Pig Latin, interoperability with Python, JavaScript, Ruby and Groovy Apache HIVE Data Warehouse on top of Hadoop. HiveQL Summaries, queries, and analysis. Open Sourced by Facebook. Apache Zoo Keeper Hadoop Configuration / Build Tools Distributed configuration, synchronization, etc) Apache Sqoop Move RDBMS data into Hadoop Command lines Gail Z Associates, LLC
    • Cassandra http://nosqlessentials.com Professor: Fernando Rodriguez Olivera Gail Z Associates, LLC
    • HBase http://blog.sematext.com/2012/07/16/hbase-memstore-what-you-should-know/ Gail Z Associates, LLC
    • http://blog.cloudera.com/blog/2011/06/biodiversity-indexing-migration-from-mysql-to-hadoop/ Gail Z Associates, LLC