Downloaded 84 times
More Related Content
Hadoop Technology
- 1.
- 2. Big Data TURKCELL DAHİLİ Volume: Velocity: Oftentime sensitive , big data must be used as it is streaming in to the enterprise it order to maximize its value to the business. Batch ,Near time , Real-time ,streams Big data comes in one size : large . Enterprises are awash with data ,easy amassing terabytes and even petabytes of information. TB , Records , Transactions ,Tables , Files. Value Variety: Verification: Big data extends beyond structured data , including semi-structured and unstructured data to all varieties :text , audio , video ,click streams ,log files and more With all the big data there will be bad data and with diverse data there will be more diverse quality and security levels of users. Structured , Unstructured , Semi-structured Good , Undefined , bad , Inconsistency , Incompleteness , Ambiguity
- 3. TURKCELL DAHİLİ Big Data– Data Sources
- 4. TURKCELL DAHİLİ Big Data– Data Growth
- 5. Hadoop Characteristics • • • • • • Open Source Distributeddata replication Commodity hardware Data and analysis co-location Scalability Reliable error handling TURKCELL DAHİLİ
- 6. Hadoop Storyline TURKCELL DAHİLİ 20032006 2008 2009 2011 2012 Apache Hadoop Project started for Yahoo requirements Google published GFS & MapReduce Paper Cloudera founded First commercial Hadoop Distribution released. Enterprise support is available Hortonworks founded Ecosystem reaches 300 companies
- 7. TURKCELL DAHİLİ Hadoop forEnterprise
- 8.
- 9. RDBMS vs. Hadoop TURKCELLDAHİLİ RDBMS Hadoop Data Size Terabytes Petabytes Schema Required on Write Required on Read Speed Reads are fast Writes are fast Access Interactive and Batch Batch Updates Write and Read Many Times Write once Read many times Scaling Scale up Scale out Data Types Structured Multi and unstructured Integrity High Low Best Use Interactive OLAP Analytics Complex ACID transactions Operational Data Store Data Discovery Processing unstructured data Massive storage/processing
- 10. TURKCELL DAHİLİ Benefits ofAnalysing with Hadoop • Previously impossible or impractical to do analysis • Analysis conducted at lower cost • Greater flexibility
- 11. TURKCELL DAHİLİ BigData &Hadoopin Turkcell • Processing «Big Data» since 2009 with Cirrus • Hadoop is on production since December’12 • ~4.5B records/~3.5TB data is processed with Cirrus • Data not stored for future analysis • Cloudera Distribution for Hadoop (nonsupported) • 5 x 24 core machines with SAN storage (not reference arch)
- 12. TURKCELL DAHİLİ Common Hadoop-ableProblems • • • • • Modeling True Risk Customer Churn Analysis Recommendation Engine Ad Targeting Pos Transaction Analysis • Analyze Network Data to Predict failure • Threat Analysis • Search Quality • Data ‘sandbox’
- 13.
- 14. Modeling True Risk TURKCELLDAHİLİ • Source, parse and aggregate disparate data sources to build comprehensive data picture • E.g. Credit card records, call recordings, chat sessions,emails, banking activity • Structure and analyze • Sentiment analysis, graph creation, pattern recognition • Typical industry • Financial Services(Banks, Insurance)
- 15. TURKCELL DAHİLİ Customer ChurnAnalysis
- 16. TURKCELL DAHİLİ Customer ChurnAnalysis • Rapidly test and build behavioral model of customer from disparate sources • Structure and analyse with Hadoop • Traversing • Graph creation • Pattern recognition • Typical Industry • Telecommunications, Financial Services
- 17.
- 18. TURKCELL DAHİLİ Recommendation Engine •Batch processing framework • Allow execution in parallel over large datasets • Collaborative filtering • Collecting ‘taste’ information from many users • Utilizing information to predict what similar users like • Typical industry • Ecommerce, Manufacturing, Retail
- 19.
- 20. Ad Targeting TURKCELL DAHİLİ •Data analysis can be conducted in parallel, reducing processing times from days to hours • With hadoop, as data volumes grow the only expansion cost is hardware • Add more nodes without degradation in performance • Typical Industry • Advertising
- 21. TURKCELL DAHİLİ Point ofSale Transaction Analysis
- 22. TURKCELL DAHİLİ Point ofSale Transaction Analysis • Batch processing framework • Allow execution in parallel over large datasets • Pattern Recognition • Optimizing over multiple data sources • Utilizing information to predict demand • Typical Industry • Retail
- 23. Analyzing Network Datato PredictTURKCELL DAHİLİ Failure
- 24. Analyzing Network Datato PredictTURKCELL DAHİLİ Failure • Take the computation to the data • Extending the range of indexing techniques from simple scans to more complex data mining • Better understand how network reacts to fluctuations • How previously thought discrete anomalies may, in fact, be interconnected • Identify leading indicators of components • Typical Industry • Utilities, Telecommunications, Datacenters
- 25.
- 26. Threat Analysis • Parallelprocessing over huge datasets • Pattern recognition to identify anomalies i.e. Threats • Typical Industry • Security, Financial Services, click fraud.. TURKCELL DAHİLİ
- 27.
- 28. Search Quality • Analysingsearch attempts in conjunction with structured data • Pattern recognition • Browsing pattern of users performing searches in different categories • Typical Industry • Web • Ecommerce TURKCELL DAHİLİ
- 29.
- 30. Data ‘Sandbox’ TURKCELL DAHİLİ •With Hadoop an organization can dump all this data into HDFS cluster • Then use Hadoop to start trying out different analysis on data • See patterns or relationships that allow the organization to derive additional value from data • Typical Industry • Common across all industries
- 31.
- 32. Apache Hadoop Core TURKCELLDAHİLİ • Hadoop is a distributed storage and processing technology for large scale applications • HDFS: Self healing, distributed file system for multi-structured data; breaks files into blocks & stores redundantly across cluster. • Map Reduce: Framework for running large data processing jobs in parallel across many nodes & combining results.
- 33.
- 34. TURKCELL DAHİLİ Hadoop DistributedFile System • The Hadoop Distributed File System (HDFS) stores files across all of the nodes in a Hadoop cluster. • It handles breaking the files into large blocks and distributing them across different machines. • It also makes multiple copies of each block so that if any one machine fails, no data is lost or unavailable.
- 35. HDFS- Features • • • • • TURKCELL DAHİLİ Highlyfault-tolerant High throughput Suitable for applications with large data sets Streaming access to file system data Can be built out of commodity hardware
- 36. TURKCELL DAHİLİ Hadoop DistributedFile System • The brain of HDFS is the NameNode. • • • • Maintains the master list of files in HDFS Handles mapping of filenames to blocks Knows where each block is stored Ensure each block is replicated the appropriate number of times. • DataNodes are machines that store HDFS data. • Each DataNode is colocated with a TaskTracker to allow moving of the computation to data.
- 37. HDFS-Design TURKCELL DAHİLİ • VeryLarge files • Streaming data access • Time to read the whole file is more important than the reading the first record • Commodity hardware • Optimized for high throughput • Not fit for • Low latency data access • Lots of small files • Multiple writers, arbitrary file modifications
- 38.
- 39. MapReduce TURKCELL DAHİLİ • MapReduceis the framework for running jobs in Hadoop. It provides a simple and powerful paradigm for parallelizing data processing. • The JobTracker is the central coordinator of jobs in MapReduce. It controls which jobs are being run, which resources they are assigned, etc. • On each node in the cluster there is a TaskTracker that is responsible for running the map or reduce tasks assigned to it by the JobTracker.
- 40.
- 41.
- 42.
- 43. YARN TURKCELL DAHİLİ • TheYARN resource manager, which coordinates the allocation of compute resources on the cluster. • The YARN node managers, which launch and monitor the compute containers on machines in the cluster. • The MapReduce application master, which coordinates the tasks running the MapReduce job. The application master and the MapReduce tasks run in containers that are scheduled by the resource manager and managed by the node managers.
- 44. Pig TURKCELL DAHİLİ • Pigprovides an engine for executing data flows in parallel on Hadoop. • PigLatin is a simple-to-understand data flow language used in the analysis of large data sets. • Pig scripts are automatically converted into MapReduce jobs by the Pig interpreter • Pig has an optimizer that rearranges some operations in Pig Latin scripts to give better performance, combines MapReduce jobs together
- 45. Hive TURKCELL DAHİLİ • Isa datawarehouse system layer built on Hadoop • Allows you to define a structure for your unstructured Big Data • Simplifies analysis and queries with an SQL like scripting language called HiveQL • Produces MapReduce jobs in background • Extensible (UDFs,UDAFs,UDTFs) • Support uses such as: • Adhoc queries • Summarization • Data Analysis
- 46. Hive is not TURKCELLDAHİLİ • … a relational database • … designed for online transaction processing • … suited for realtime queries and row-level updates
- 47.
- 48. Ambari TURKCELL DAHİLİ • Ambarifor Hadoop Clusters • Provision • Manage • Monitor
- 49. Ambari TURKCELL DAHİLİ • Providesstep-by-step wizard for installing Hadoop services across any number of hosts • Handles configuration of Hadoop services for the cluster.
- 50. Sqoop and Flume TURKCELLDAHİLİ • Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. • Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large quantities of streaming data(ex:logs) into HDFS. It has a simple and flexible architecture based on streaming data flows.
- 51. Schemas - HCatalog TURKCELLDAHİLİ • A table and storage management service for data created using Apache Hadoop • Providing a shared schema and data type mechanism. • Providing a table abstraction so that users need not be concerned with where or how their data is stored. • Providing interoperability across data processing tools such as Pig, Map Reduce, and Hive. • Example • stocks_daily= load ‘nyse_daily' using HCatLoader(); • cleansed = filter stocks_daily by symbol is not null;
- 52. Mahout TURKCELL DAHİLİ • TheApache Mahout™ machine learning library's goal is to build scalable machine learning libraries. • Core algorithms for clustering, classification and batch based collaborative filtering are implemented on top of Apache Hadoop using the map/reduce paradigm. • The core libraries are highly optimized to allow for good performance also for non-distributed algorithms
- 53. TURKCELL DAHİLİ Hadoop Corein Detail
- 54. Map Phase TURKCELL DAHİLİ •In the map phase, MapReduce gives the user an opportunity to operate on every record in the data set individually. This phase is commonly used to project out unwanted fields, transform fields, or apply filters. • Certain types of joins and grouping can also be done in the map (e.g., joins where the data is already sorted or hash-based aggregation).
- 55.
- 56. Combiner Phase TURKCELL DAHİLİ •Minimize the data transferred between map and reduce tasks. • The combiner gives applications a chance to apply their reducer logic early on.
- 57. Shuffle Phase TURKCELL DAHİLİ •Data arriving on the reducer has been partitioned and sorted by the map, combine, and shuffle phases. • By default, the data is sorted by the partition key. For example, if a user has a data set partitioned on user ID, in the reducer it will be sorted by user ID as well. Thus, MapReduce uses sorting to group like keys together. • It is possible to specify additional sort keys beyond the partition key
- 58.
- 59. Reduce Phase TURKCELL DAHİLİ •The input to the reduce phase is each key from the shuffle plus all of the records associated with that key. • Because all records with the same value for the key are now collected together, it is possible to do joins and aggregation operations such as counting. • The MapReduce user explicitly controls parallelism in the reduce.
- 60.
- 61. Output Phase TURKCELL DAHİLİ •The reducer (or map in a map-only job) writes its output via an OutputFormat. • OutputFormat is responsible for providing a RecordWriter, which takes the key-value pairs produced by the task and stores them. • This includes serializing, possibly compressing, and writing them to HDFS, HBase, etc
- 62. TURKCELL DAHİLİ Map ReduceLogical Flow
- 63. TURKCELL DAHİLİ Map ReduceLogical Flow
- 64.
- 65. Speculative Execution TURKCELL DAHİLİ •If a Mapper runs slower than the others, a new instance of the Mapper will be started on another machine operating on the same data. • The result of the first Mapper to finish will be used. • Hadoop will kill of the Mapper which is still running.
- 66. Distributed Cache TURKCELL DAHİLİ •Sometimes all or many of the tasks in a MapReduce job will need to access a single file or a set of files. • When thousands of map or reduce tasks attempt to open the same HDFS file simultaneously, this puts a large strain on the NameNode and the DataNodes storing that file. • To avoid this situation, MapReduce provides the distributed cache. • The distributed cache allows users to specify—as part of their MapReduce job—any HDFS files they want every task to have access to. • These files are then copied onto the local disk of the task nodes as part of the task initiation. Map or reduce tasks can then read these as local files.
- 67. TURKCELL DAHİLİ Setting upEnvironment • Hortonworks Sandbox: http://hortonworks.com/products/sandboxinstructions/ • VMware: http://www.vmware.com/products/player/overvi ew.html • Setup Guide:http://hortonworks.com/wpcontent/uploads/2013/03/InstallingHortonworks SandboxonWindowsUsingVMwarePlayerv2.pdf
- 68.
- 69.
- 70. MapReduce Demo • EclipsePlugin: • HDFS Operations • Running WordCount, TopK • Generating jars for HDP Sandbox • Sandbox: • HDFS Operations • Loading and Running jar files • Oozie and Ambari TURKCELL DAHİLİ
- 71. Hive Demo • • • • • Create Tablewith HCatalog Load Data in to Hive Query Data OUTPUT to Table/HDFS/Local JOIN TURKCELL DAHİLİ
- 72.
- 73.