Understanding Big Data summarizes big data and popular big data technologies. It discusses how big data is generated from various sources and is too large to be processed by traditional databases. Popular technologies like Hadoop, HDFS, MapReduce, Hive, Pig, HBase, and Mahout are able to collect, store, process, and analyze big data. Companies are using big data to gain insights from customer data, optimize operations, prevent fraud, and make recommendations.

1. UNDERSTANDING BIG DATAS
THINK BIG
MEHMET BURAK AKGÜN

2. First of All What is Big Data?
• BIG-DATA is collected as
non-structural from
different sources Such as
Social media sharing,
network logs, blogs,
photos, videos, log files,
etc..
• BIG-DATA is Can not be
analyzed before Enormous
size and / or diversity
information over the data

3. 1) Why Big Data ? How did we come this point?
2) What is the Big Data’s Components?
3) How will the integration with existing systems?
4) Is there any BIG Data Platform / Applications?

4. • Almost all data scientists believed that too important advantages of
companies using with the ability to analyze information collected from all
sources .
• For example, retailers; according to scientists analyze they can through data
can improve their operations by 60 percent profit margin. Similar rates are
also valid for the public sector.
• According to research, The US healthcare using to data scientists can save
300 million dollars a year.
• According to estimates in 2020, from household appliances to cars and
phones and about 50 billion devices will produce data and will be silently
communicate with each other.
• For companies which wish to take decisions in the forward-looking ,
predictions should be critically correct from these «Big Data».

5. ● Energy companies, using smart grids and meters, consisting of the use
of data relating to individual subscribers, store, handle the event.
● Banks has become 7/24 branch as according to the information they
store regarding collecting customers , recognizing the user, the
Internet branch knows the day nor to enter and accordingly the main
page menu that makes the most efficient, customers who reminders,
offers customizable interfaces, rich, fast and convenient branch .
● Hospitals saving datas on their databases for effective to provide
medical services
● This information will be stored as "Big Data"

6. DATA = PRODUCT

7. Google without using conventional methods, created their technologies of
requirenments,improving itself was a success. Google has billions of web pages
on the Google File System keeps uses Big Table in the database, using
MapReduce for processing Big Data.
See http://www.google.com/about/datacenters
Google and Amazon publishes academic articles are related to their work. Some
developers who inspired by the articles such as Doug Cutting created similar
technology as secret. These are usually the most beautiful examples of the
Apache Lucene, Left, Hadoop, HBase such projects. Each of these projects can
successfully use the Big Datas .
The second generation of the companies such as Facebook, Twitter, Linkedin,
are going a step forward by publishing them store it as open-source projects
developed for Big data. Cassandra, Hive, Pig, Voldemort, Storm, indextank
projects are examples.

8. Peki Büyük Veri nasıl depolanacak?
Nasıl işlenecek?
Nasıl analiz edilecek?

9. Volume
of Tweets
create daily.
12+ terabytes
Variety
of different
types of data.
100’s
Veracity
decision makers trust
their information.
Only 1 in 3
4 «components» for being Big
trade events
per second.
5+ million
Velocity

11. • It is expecting to reach 3 billion users.In August
2015,
12TB / day «log data» is producing.
The end of 2014 expected reach is 500+ million
users.
160 million users are online.
•
•
• 100 million active users. 12+ TB of data
tweets / day! ..
Social Networks and Social Work

12. Google process 24
Petabytes data every day
4.6 Billion mobile
phone exist
2 Billion Internet users annual traffic
in 2014 equals 667 Exabytes
Social Networks and Social Work

13. Users are not just people! ..

14. Each engine 10TB / 30 Mins data produces.

15. “Data generated by machines and sensors
will exceed that generated by social media by
at least a factor of 10.” *
Leon Katsnelson
Program Director, Big Data & Cloud
Computing IBM

16. By the way, We don't have enough space to store all this data!

17. We actually appear as a game company.In fact we are
data analysis company..
Ken Rudin, Zynga VP of Analytics
• Offers completely free game facilities.
• Gaining revenue by selling virtual goods.
• The monthly average has 232m active users.
• 95% of players never visited shop!
• With Using Big Data analysis they disturbed to the game world.

18. Four Entry Points of Big Data
Unlock Big
Data
Simplify Your
Warehouse
Preprocess
Raw Data
Analyse
Streaming
Data
IBM Big Data Platform
Systems
Management
Application
Development
Visualization
& Discovery
Accelerators
Information Integration & Governance
Hadoop
System
Stream
Computing
Data Warehouse
BI / Reporting Exploration /
Visualization
Functional
App
Industry
App
Predictive
Analytics
Content
Analytics
Analytic Applications

19. Applications for Big Data Analytics
Homeland Security
FinanceSmarter Healthcare Multi-channel sales
Telecom
Manufacturing
Traffic Control
Trading Analytics Fraud and Risk
Log Analysis
Search Quality
Retail: Churn, NBO

20. "Companies which give importance to Social media is gaining"
In Turkey Organized by the Teradata "Big Data: Great Opportunity" themed event brought together senior
executives of the company in Istanbul. President of Teradata EMEA Hermann Wimmer, ‘companies that want to
increase their profitability by preventing competitors from social media by analyzing the data obtained in a short
time, said he helped to make quick decisions.’

21. Technological solution is preferring when it provides advantages

23. Big Data and Open Source Nested
• Open Source Community contribution made over the years
- Apache Hadoop ve Jaql, Apache Derby, Apache Geronimo, Apache
Jakarta
- Eclipse: was founded by IBM.
- Lucene ;IBM Lucene Extension Library (ILEL)
- DRDA, XQuery, SQL, XML4J, XERCES, HTTP,
Java, Linux...
• Open source IBM Softwares
– WebSphere: Apache
– Rational: Eclipse and Apache
– InfoSphere: Eclipse and Apache
• IBM’s BigInsights (Hadoop) is %100 open source software.

24. Hadoop
• Open Source
• Distributed
Computing
• Very Simply
MapReduce
• Connected
computers

25. • Ebay – 532 node 532x8 Core – 4256 Core
• Facebook – 1100 node 8800 core 12 PB
Storage
• LinkedIn – 1200 + 580 + 120 node
• Quantcast – 3000 node 3.5 PB 1PB+ daily
data
Example Hadoop Installations

26. Hadoop
• Components:
– DataStorage-HDFS-Distributed DiskFileSystem
– DataProcessing-MapReduce

27. HDFS
• Google File System
• Distributed Disk File System
• File Clustering(Usually Files sizes are over then GBs
• Fault tolerant
• Replication
• HDFS is knowledge about the location of physical.

28. HDFS
• Accessible by with Hadoop Shell , Java API or Web UI
• Four application node:
– NameNode – manages the metadata of the file
– SystemJob Tracker – MapReduce
– Task Tracker – MapReduce
– Data Node – InformationshidingwithNameNode

29. BIG DATA is not just HADOOP
Manage & store huge volume
of any data
Hadoop File System
MapReduce
Manage streaming data Stream Computing
Analyze unstructured data Text Analytics Engine
Data WarehousingStructure and control data
Integrate and govern all
data sources
Integration, Data Quality, Security, Lifecycle
Management, MDM
Understand and navigate
federated big data sources
Federated Discovery and Navigation

30. MapReduce
• Big Data Processing -‐Google
• Distributed Calculation Model
• Key - the value binary data processing
• Easy programming framework
– for use:you should improve map() ve reduce()
functions

31. Map: First Step
• Make ready the next processing elements
while matching with a key
– Data Cleaning
– Simply Calculating
– Split Strings

32. Reduce: Last Step
• Takes the list of values for Same key with
Iterators
– Filtering
– Combining
– Samping
So reduced..
The result is written to HDFS or HBase

33. Architecture:MapReduce
HDFS CLUSTER
K1,V1
K1,V1
…..
K1,V1
K1,V1
…
K1,V1
K1,V1
….
K1,V1
K1,V1
…
M K2,V2
K2,V2
M K2,V2
K2,V2
M K2,V2
K2,V2
K2,V2
M …
M
M
M
M
K2,V2
K2,V2
K2,V2
K2,V2
K2,V2
K2,V2
K2,V2
K2,V2
K2,V2
…
GROUPING
RANKING
K2, Iterable<V2>
K2, Iterable<V2>
K2, Iterable <V2>
R
K3,V3
K3,V3
…

34. Apache Pig
• Yahoo!
• Designed to easy analysis of large data sets
• Easy than Map Reduce function in Java
– Pig Lan coding
– Can be improved
• Similar with each languages
• 10 lines of Pig code may be equivalent to hundreds of lines Java

35. Run To Pig !
• Grunt – Shell
• Java interface
• Eclipse and IntelliJ IDEA plugins
%tweets = load ‘/today/tweets’ as (user,
mention, tweet)
%twitters = group tweets by mention

36. Apache Hive
• Data Warehouse Project for Hadoop
• SummarizingData
• Instant queries
• SQL-like language –HiveQL
– Allows to define custom mapper and reducer
Special Note * : Hive compiler translates the
MapReduce operations to SQL queries

37. HBase
Every time we do not need to relational databases
We need Scalability
Table size can be very large so We want very fast
access
Distributed Key-‐Sorted Persistent Map

38. HBase
Google – Big Table clon
Works on HDFS
*fault tolerance
*scalability
*MapReduces input-output
Hbase = HDFS + Random read/ write

39. HBase
Where to Use:
Social Media
Recommended Systems
Search Engines
Intelligence and Monitoring Services
Financial Systems - fraud

40. Apache Mahout
• Let us go beyond the simple analysis
• Classification - Email Spam - Call Center
• Clustering - finding new news
• Recommended Systems

41. Apache Mahout
Ready Algorithms::
• Classification:
• Logisac Regression
• Bayesian
• Random Forests
– Application
• Call forwarding
• Face recognition

42. Apache Mahout
Ready Algorithms:
Recommendation:
• Distributed Item based
• Matrix Factorizaaon
• Non distributed item/user based
– Application:
• eCommerce - Amazon
• Movie recommendation - netflix
• Music recommendation - LastFM
• Venue Recommendation - foursquare

43. In summary:Some Big
Data Applications
 Log Analytics (IT for IT)
 Smart Grid / Smarter Utilities
 RFID Tracking & Analytics
 Fraud / Risk Management & Modeling
 360° View of the Customer
 Warehouse Extension
 Email / Call Center Transcript
Analysis
 Call Detail Record Analysis
 IBM Watson

44. The IBM Big Data Platform

45. IBM Big Data Platform
IBM Big Data & Netezza Product Group
InfoSphere BigInsights Hadoop-based,
low latency, diverse and high-volume data
analysis
Hadoop
IBM Netezza High
Capacity Appliance
Archived questionable
structural data
IBM Netezza 1000
BI+Ad Hoc
Structured Data Analysis
IBM Smart
Analytics System
Structural analysis
of operational data
IBM Informix
Timeseries
Time-structured analytics
IBM
InfoSphere
Warehouse
High volume, structural
veri analizi
Stream Computing
InfoSphere Streams Fluid
analysis for low latency data
MPP Data Warehouse
Information
IntegrationInfoSphere Information
Server High volume data
integration and transformation

46. Big Data Exploration: Value & Diagram
File
Systems
Relational
Data
Content
Management
Email
CRM
Supply
Chain
ERP
RSS Feeds
Cloud
Custom
Sources
DataExplorerApplication/
Users
Find, Visualize & Understand all
big data to improve business
knowledge
• Greater efficiencies in business
processes
• New insights from combining and
analyzing data types in new ways
• Develop new business models with
resulting increased market presence
and revenue

47. Data Analysis in Different Diversity
Making the analysis on the data in mixed feature.
Dinamic Data Analysis
High volume flow data, ad-hoc analysis
Explore and experiment
Data on Ad-hoc analysis, discovery and
inspection data
Manage and Plans
Data Rules,Data integrity checking and
application
Very High Volume Data Analysis
The data in the PB scale appropriate price /
performance criteria for analysis
What does IBM platform?