This document discusses big data, including the large amounts of data being collected daily, challenges with traditional DBMS solutions, the need for new approaches like Hadoop and Aster Data to handle large volumes of structured and unstructured data, techniques for analyzing big data, and case studies of companies like Mobclix and Yahoo using big data solutions.

1. BigDataShankar RadhakrishnanHCL TechnologiesJuly, 2011

2. Big Data in the NewsSavingsAmerican Health-Care: $300 Billion/YearEuropean Public Sector: €250 Billion/YearProductivity Margins: 60% increaseSources: McKinsey Global Institute

3. TopicsWhat do we collect today?DBMS LandscapeThe DisconnectThe NeedWhat is BigData?CharacteristicsApproachArchitectural RequirementsTechniquesChallengesSolutionsIssuesDeep Dive – Practical Approaches to Big DataHadoopAster DataCase StudiesBig Data @ HCLProject CarbonSMILeCoE - Opportunities, Challenges

4. What do we collect?In 2010, people stored data to fill 60,000 Library of Congress (LoC collected 235TB in Apr/2011)YouTube receives 24hours of video, every minute5 Billion mobile phones in use in 2010Tesco (British Retailer) collects 1.5 billion pieces of information to adjust prices and promotionsAmazon.com: 30% of sales is out of its recommendation enginePlanecast, Mobclix : Track & Target systems promotes contextual promotionsA Boeing Jet Engine produces 20TB/Hour for engineers to examine in real time to make improvementsSources: Forrester, The Economist,McKinsey Global Institute

5. Collect MoreBusiness OperationsTransactionsRegistersGatewaysCustomer InformationCRMProduct InformationBarcodesRFIDWebPagesWeb RepositoriesUnstructured InformationSocial MediaSignalsMobileGPS, GeoSpatial

6. DBMS SolutionsLegacyFaster RetrievalEfficient StorageDivide and AccessData ConsolidationBroader TablesAccess all as a rowFine GrainAccessSecurityRules and PoliciesProblemsData GrowthWhen storage cost is not an issueScalability IssuesPerformance IssuesNew types of requirementsDeciding what to analyze, when and how?Cost of a change in the subject-area to analyze

7. The DisconnectOld DBMS vs. New Data Types/StructuresOld DBMS vs. New volumeOld DBMS vs. New AnalysisOld DBMS vs. Data RetentionOld DBMS vs. Data Element StripingOld DBMS vs. Data InfrastructureOld DBMS vs. One DB Platform for all

8. The NeedSystem that can handle high volume dataPerform complex operationsScalableRobustHighly AvailableFault TolerantEconomicNew Approach

9. Big Data“Tools and techniques to manage different types of data, in high volume, in high velocitywith varied requirements to mine them”CharacteristicsSizeScale up and scale out: Terabyte, Petabyte …StructureStructuredUnstructured : Audio, Video, Text, GeoSpatialSchema Less StructuresStreamTorrent of real-time informationOperationMassively Parallel Processing (MPP)

10. ApproachHardwareCommodity HardwareApplianceDynamic ScalingFault TolerantHighly AvailableNo constraints on StorageCloudVirtual Environment, StorageProcessing ModelsIn-memoryIn-databaseInterfaces/AdaptersWorkload ManagementDistributed Data ProcessingSoftwareFrameworks – Hadoop, MapReduce, Vrije, BOOM, BloomOpen SourceProprietary

11. Architectural RequirementsIntegration FrameworkDevelopment FrameworkManagement FrameworkModeling FrameworkProcessing FrameworkData Management Framework

12. ChallengesVolumetric AnalysisComplexityStreaming Data/Real Time DataNetwork TopologyInfrastructurePattern-based Strategy

13. TechniquesControlled and Variate TestingMiningMachine LearningNatural Language Processing (NLP)Cohort AnalysisNetwork or Path AnalysisPredictive ModelsCrowd SourcingRegression ModelsSentiment AnalysisProcessing SignalsSpatial AnalyticsVisualizationTime-series Analysis

14. SolutionsIBM: Infosphere BigInsights, StreamsTeradata/Aster Data: nCluster, SQL-MRFrameworksHadoopMapReduceInfobright*SplunkCloudera*CassandraNoSQL, NewSQLGoogle’s Big TableApplianceTeradataNetezza (IBM)Columnar DatabasesVertica (HP)ParAccelManaged Services Available

15. IssuesLatencyFaultinessAccuracyACIDAtomicityConsistencyIsolationDurabilitySetup CostDevelopment CostCost-to-fly

16. Deep DiveHadoop

17. Top level Apache projectOpen sourceSoftware Framework - JavaInspired by Google’s white papers onMap/Reduce (MR)Google File System (GFS)Big TableOriginally developed to support Apache NutchDesignedLarge scale data processingFor batch processingFor sophisticated analysisTo deal with structured and unstructured dataDB Architect’s Hadoop : "Heck Another Darn Obscure Open-source Project"

18. Why Hadoop?Runs on commodity hardwarePortability across heterogeneous hardwareand software platformsShared-nothing architectureScale hardware when ever you wantSystem compensates for hardware scalingand issues (if any)Run large-scale, high volume data processesScales well with complex analysis jobs(Hardware) “Failure is an option”Ideal to consolidate data from both new andlegacy data sourcesHighly IntegrableValue to the business

19. Hadoop EcosystemHDFS Hadoop Distributed File SystemMap/Reduce Software framework for Clustered, Distributed data processingZooKeeper SchedulerAvro Data SerializationChukwa Data Collection System to monitor Distributed SystemsHBase Data storage for distributed large tablesHive Data warehousePig High-Level Query LanguageScribe Log CollectionUDF User Defined Functions

20. Hadoop Flow (Example)Network StorageWeb ServersScribeOracleMySQLHadoop Hive DWHMySQLOracleAppsFeeds

21. HDFSHadoop Distributed File SystemMaster/Slave ArchitectureRuns on commodity hardwareFault TolerantHandle large volumes of dataProvides High ThroughputStreaming data-accessSimple file coherency modelPortable to heterogeneous hardware and softwareRobustHandles disk failures, replication (& re-replication)Performs cluster rebalancing, data integrity checks

22. HDFS ArchitectureName nodeFile system operations

23. Maps data-nodesData nodeProcess read/write

24. Handles Data-blocks

25. ReplicationHadoop M/RTagged by a jobSplits input data-set into separate chunk’sProcessed by map tasks, in parallelSorts the output of the mapsProcessed by reduce tasks, in parallelTypically stored and processed in a file systemFramework takes care ofScheduling tasksMonitoringRe-executing failed tasksInfrastructure issuesLoad-balancing, Load-redistributionReplication, Failover

26. Mapper Functioncat * | grep | sort | uniq –c | cat > fileinput | map | shuffle | reduce | output

27. Reduce Functioncat * | grep | sort | uniq –c | cat > fileinput | map | shuffle | reduce | output

28. Hadoop Environment

29. Who uses Hadoop?

30. Deep DiveAster Data

31. Aster DataNow part of TeradataMassively ParallelSQL Layer on MR (MapReduce)In-Database AnalyticsAppliance vs. Software Stack ModelCloud OptionsnPath and Statistical OptionsData Integration

32. nCluster

33. Deep DiveCase Studies

34. Aster nCluster @ MobclixMobclix App LibrarySmart PhoneApp-Ad FabricRelevanceRecommendationEventsAdsAd ExchangeContextBIDBDBDBDBTargetingDBDBDBDBAmazon EC2AnalyticsnClusterAmazon EC2nCluster1 TB/Hour1-2 TB/HourIP of Mobclix