Big data ppt


Published on

  • Hi please send this to me. I'm working on a project and I believe that it would help me with it.
    Are you sure you want to  Yes  No
    Your message goes here
  • Please send this to me ( ~^^ Thanks so much
    Are you sure you want to  Yes  No
    Your message goes here
  • please send me ( the ppt as I need to understand and explain the features of hadoop at my college.
    Are you sure you want to  Yes  No
    Your message goes here
  • sir, please send this to me
    Are you sure you want to  Yes  No
    Your message goes here
  • sir, please send this to me to
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • ,
  • Big data ppt

    1. 1. Introduction toBIG DATA Thiru
    2. 2. What is BIGDATA? variety-what-you-need-to-know-about-big-data
    3. 3.  Search EngineData Scalability  10KB / doc * 20B docs = 200TBProblems  Reindex every 30 days: 200TB/30days = 6 TB/day  Log Processing / Data Warehousing  0.5KB/events * 3B pageview events/day = 1.5TB/day  100M users * 5 events * 100 feed/event * 0.1KB/feed = 5TB/day  Multipliers: 3 copies of data, 3-10 passes of raw data  Processing Speed (Single Machine)  2-20MB/second * 100K seconds/day = 0.2-2 TB/day
    4. 4. What’s the social sentiment How do I better predict futurefor my brand or products outcomes? How do I optimize my fleet based on weather and traffic patterns?
    5. 5. Traditional E-Commerce Data Flow
    6. 6. New E-Commerce Big Data Flow
    7. 7. Introduction to Hadoop
    8. 8. Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which mayHADOOP be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.
    9. 9. HadoopHistory Jan 2006 – Doug Cutting joins Yahoo Feb 2006 – Hadoop splits out of Nutch and Yahoo starts using it. Dec 2006 –Yahoo creating 100-node Webmap with Hadoop Apr 2007 –Yahoo on 1000-node cluster Jan 2008 – Hadoop made a top-level Apache project Dec 2007 –Yahoo creating 1000-node Webmap with Hadoop Sep 2008 – Hive added to Hadoop as a contrib project
    10. 10. • Commodity hardwareBIG DATA compatibility ECONOMICS • Reduction in storage costEconomics • Open source system • The Web economy
    11. 11. Column Store Database
    12. 12. Row Store andColumn Store
    13. 13.  Can be significantly faster than row stores for some applications  Fetch only required columns for a query  Better cache effects  Better compression (similar attribute values within a column)Why Column  But can be slower for other applications  OLTP with many row inserts, ..Store?  Long war between the column store and row store camps :-)
    14. 14. So How Does It Work?
    15. 15. So How Does It Work?
    16. 16. The Hadoop Ecosystem
    17. 17. Traditional RDBMS vs. MapReduceComparisons
    18. 18. HDFS, the storage layer of Hadoop, is a distributed, scalable, Java-based file system adept at storing large volumes of unstructured data. MapReduce is a software framework that serves as the compute layer of Hadoop. MapReduce jobs are divided into two (obviously named) parts. The “Map” function divides a query into multiple parts and processes data at the node level. The “Reduce” function aggregates the results of the “Map” functionHadoop to determine the “answer” to the query.Ecosystem Hive is a Hadoop-based data warehouse developed by Facebook. It allows users to write queries in SQL, which are then converted to MapReduce. This allows SQL programmers with no MapReduce experience to use the warehouse and makes it easier to integrate with business intelligence and visualization tools such as Microstrategy, Tableau, Revolutions Analytics, etc. Pig Latin is a Hadoop-based language developed by Yahoo. It is relatively easy to learn and is adept at very deep, very long data pipelines (a limitation of SQL.)
    19. 19. HBase is a non-relational database that allows for low-latency, quick lookups in Hadoop. It adds transactional capabilities to Hadoop, allowing users to conduct updates, inserts and deletes. EBay and Facebook use HBase heavily. . Flume is a framework for populating Hadoop with data. Agents are populated throughout ones IT infrastructure – inside web servers, application servers and mobile devices, for example – to collect data and integrate it into Hadoop.HadoopEcosystem Oozie is a workflow processing system that lets users define a series of jobs written in multiple languages – such as Map Reduce, Pig and Hive -- then intelligently link them to one another. Oozie allows users to specify, for example, that a particular query is only to be initiated after specified previous jobs on which it relies for data are completed Whirr is a set of libraries that allows users to easily spin-up Hadoop clusters on top of Amazon EC2, Rackspace or any virtual infrastructure. It supports all major virtualized infrastructure vendors on the market.
    20. 20. Avro is a data serialization system that allows for encoding the schema of Hadoop files. It is adept at parsing data and performing removed procedure calls Mahout is a data mining library. It takes the most popular data mining algorithms for performing clustering, regression testing and statistical modeling and implements them using the Map Reduce modelHadoopEcosystem Sqoop is a connectivity tool for moving data from non-Hadoop data stores – such as relational databases and data warehouses – into Hadoop. It allows users to specify the target location inside of Hadoop and instruct Sqoop to move data from Oracle, Teradata or other relational databases to the target. BigTop is an effort to create a more formal process or framework for packaging and interoperability testing of Hadoops sub-projects and related components with the goal improving the Hadoop platform as a whole.
    21. 21. Microsoft & Hadoop
    22. 22. Insights to allusers byactivating newtypes of data
    23. 23. Microsoft BI
    24. 24. Stats Machine Legend Graph Pipeline / Workflow processing Learning (Pegasus) Red = Core Hadoop (RHadoop) (Mahout) Blue = Data (Oozie) Metadata processing (HCatalog) Purple = Microsoft ( ODBC / SQOOP/ REST) integration points Scripting Query Data Integration NoSQL Database and value adds (Pig) (Hive) Yellow = DataMicrosoft (HBase) Movement Distributed Processing Event PipelineHadoop Stack (MapReduce) Green = Packages (Flume) White = Coming Soon Distributed Storage (HDFS) Monitoring & Active Directory Deployment (Security) (System Center)
    25. 25. Others
    26. 26. HadoopCommercialDistributors
    27. 27. Other Big DataWorlds
    28. 28. Other Big DataWorlds
    29. 29. Big DataIntegrations,Visualizations& Analytics
    30. 30. Thank You