Bigdata Nedir? Hadoop Nedir? MapReduce Nedir? Big Data.

1. Big Data Zekeriya Beşiroğlu http://zekeriyabesiroglu.com http://bilginc.com http://twitter.com/zbesiroglu

2. Zekeriya Besiroglu • Bilginc IT Academy - Expert Consultant • + 16 IT • +14 ORACLE DB/DWH • +7 WEBLOGIC • +3 BIG DATA • TROUG • Speaker

3. Bilginc IT Academy

4. DATA TRENS - Facebook has around 60 PB warehouse and it’s constantly growing - Twitter messages are 140 bytes each generating 8TB data per day. -Data is more than doubling every year.  -Almost 80% of data will be unstructured data. -Amazon: 35% of product sales come from product recommendations

5. New Type of DATA? • Sentiment : Understand how your customers feel about your products / company • Sensor/Machine:Discover patters in data streaming automatically from sensors and machines. • Unstructured: text,video,pictures. • Server Logs:Search logs ﬁnd pattern • Geographic:Analyze location-based data • Clickstream:Capture and analyze website visitors data

6. Big Data https://www.youtube.com/watch?v=1GU4Imbo6R8

7. Capacity vs Cost Year Capacity(GB) Cost per GB(USD) 1990 0.10 $4000 1997 2 $150 2002 80 $3.75 2007 750 $0.35 2012 3.000 $0.05 2015 10.000 $0.02

8. What is Big Data • Big Data is When the Volume,Velocity,Variety of data gets to the point where it is too difﬁcult/ expensive for traditional systems to work with.

9. 3Vs of Big Data

10. Traditional Large scale Computing System Problems • Computation has been processor bound • Relatively small amount of data • Complex processing • Need bigger computers • More memory,More/fast processor

11. Better Solution • Distributed Systems- Multiple machine run for single job Problem Of Distributed Systems Data Stored central location Data Copied processor runtime

12. Todays • Total Data size PetaBytes • Daily Terabytes We Need New Solution HADOOP

13. HADOOP • Distribute the Data when it is stored SPARK Data is Distributed in Memory

14. RDBMS vs HADOOP

15. Hadoop • Hadoop consist of two component • HDFS • Map Reduce • Hadoop ecosystem • Pig,Hive,Hbase,Flume,Oozie,Sqoop,etc

16. Traditional ETL Source Layer Structured Data DWH Data Mart ETL/ELT ETL/ELT Hadoop ETL Source Layer Structured Data UnStructed Data DWH Data MartHADOOP

17. HDFS • Hadoop Distributed File System:Storing data • Data Split into blocks. 64 Mb… • Each Block replicated e.g 3 times. replicas store different nodes. • Based on Google File system • ext3,ext4,xfs • No random writes allowed. Prefer large streaming reads

18. HDFS

19. HDFS • hadoop fs -ls (user home directory) • hadoop fs -ls / (root directory) • hadoop fs -cat /user/zekeriya/deneme.txt • hadoop fs -mkdir • hadoop fs -rm -r veri

20. MapReduce • Process Data in the Hadoop Cluster • Two Stage MAP and REDUCE

21. MAPREDUCE map(String input_key, String input_value) foreach word w in input_value: emit(w, 1) reduce(String output_key, Iterator<int> intermediate_vals) set count = 0 foreach v in intermediate_vals: count += v emit(output_key, count) (1000,’Galatasaray sampiyon olur’) (2000,’beşiktas sampiyon olur’) (2200,’Galatasaray Türkiyedir’)

22. MAPREDUCE Output Mapper (‘Galatasaray’, 1), (‘sampiyon’, 1), (‘olur’, 1), (‘beşiktas’, 1), (‘sampiyon, 1), (‘olur’, 1), (‘Galatasaray’, 1), (‘Türkiyedir’, 1) Intermediate Data Reducer’a gönderilen (‘Galatasaray’,[1,1]) (‘sampiyon’,[1,1]) (‘olur’,[1]) (‘beşiktas’,[1]) (‘Türkiyedir’,[1]) Reducer’ın son cıktısı (‘Galatasaray’,2) (‘sampiyon’,2) (‘olur’,1) (‘beşiktas’,1) (‘Türkiyedir’,1)

23. Hadoop Ecosystem • HIVE • LIKE SQL • User query data in hadoop cluster without knowing Java and Map reduce. • PIG • Uses a dataﬂow scripting language • IMPALA • Open source project created by cloudier • Very similar to HiveQL.Produces much faster.

24. Hadoop Ecosystem • FLUME • Import data into HDFS as it is generated • Log files from a Web Server • Sqoop • Import data from tables in a OLTP into HDFS • Populate database tables from files in HDFS • Oozi • Developers create a workflow of MapReduce Jobs

25. Hadoop Ecosystem • HBASE • HADOOP DATABASE • NOSQL DATASTORE • HUGE DATA STORE,GB,TB,PB • Query Language get/put/scan • Read/write Throughput Millions of query ps ,rdbms is 1000s queries/second

26. Big Data • Finance ,Fraud detection,Customer risk analysis • Retail, Product recommendation,buy and discount • Advertising,More effective web ads • Defense • Telco • Healthcare

27. Analyzing Twitter Data • https://github.com/cloudera/cdh-twitter- example

28. Career Path • Develop with Hadoop • Hadoop Administration • Hadoop for Data Scientists & Analysts

29. Zekeriya Beşiroğlu http://zekeriyabesiroglu.com http://twitter.com/zbesiroglu http://bilginc.com http://troug.org mail to:zekeriyab@bilginc.com zekeriyabesiroglu@gmail.com