This document discusses a study on using a separation-based approach for analyzing big data from various sources. It proposes a system with four main components: a main switch to separate relevant data types, a key generator to encrypt data, a subnet switch master to assign work to map workers, and map workers that process assigned data. This bottom-up approach aims to more easily analyze large amounts of data by starting with smaller subsets. The document also covers challenges of big data, existing grid computing systems, Hadoop tools, and concludes the proposed framework would improve upon previous methods for big data analysis.

2. A STUDY ON BIG DATA AND ITS
TRAFFIC MANAGEMENT –
SEPARATION BASED ON
VARIETIES OF DATA
N. Naveen Kumar
itnaveenkumarn@gmail.com
S. Vijayabaskaran
vijay.shiva68@gmail.com
B.Tech Information Technology,
Sri Sai Ram Engineering College,
Chennai.

3. BIG DATA
• A massive volume of data sets which are
difficult to process by traditional techniques
within elapsed time.
• Data’s present here are expressed in exa-
bytes.

4. BIG DATA - PROPERTIES
• Variety
• Volume
• Velocity
• Varacity
• Value

5. Where Big Data is?
• Log storage in IT industries
• Sensor Data
• Risk analysis
• Social media

6. Challenges and issues in Big Data
• Privacy & Security
• Data access & sharing of information
• Storage & processing issues
• Analytical challenges
• Skill requirement
• Technical challenges

7. TOOLS AND TECHNIQUES
• Hadoop:
– Open source available software.
– Handled by map reduce.
– Managed by replication of data.
– Reads data from multiple disks.
• Grid Computing:
– Distributed computing.
– To save, distribute & analyze.

8. Managing Big Data
• HDFS
– Name node.
– Data node.
• Map Reduce
– Map task.

9. Existing System
Grid Computing:
SFTI(Search For Extra Terrestrial Intelligence) is created in order to
save, distribute and analyze.
Grid Center:
Number of servers that are inter-connected by high speed network.
Elements of Grid Center:
– Computing Elements is used to manage resource of
grid and manages jobs.
– Storage Elements is used in storage and data transfer
services.
– Worker Node are servers that offers the processing
power.

10. Contd…
• User Interface is also present.
• Virtual organization is the set of peoples defines rules
for accessing the grid.
• It has work load management.
• User access User Interface through secure shell (SSH).
• The user then sends the job written in Job Description
Language (JDL) and PYC (Python Code) to WMS
(Workload Management System).
• WMS checks availability of Computing Elements and
assigns Worker Nodes to process it.
• Worker Node completes job and send to WMS and
state of job to Computing elements.

11. Hadoop with HPC and Grid Computing
• In Hadoop, the Map reduce component makes
use of data locality property.
• Grid basically uses Message Passing Interface
(MPI)
• Map reduce itself give information about
Failed tasks and reschedules it.

12. Statistics on Big Data

13. Proposed System
• Separation of relevant data.
• Analysis by bottom-up approach.
• 4 major components:
Main- Switch
Key- generator
Sub-net Switch (master)
Map-workers

14. SCHEMATIC REPRESENTATION

15. Functions
i) Main switch: Separates the relevant data
Varieties of data’s are
1. Real-time
2. Social media
3. Business Organization
4. Other data
Map Reduce concept has 2 main functions:
a. Indexing
b. Key generation

16. ii) Key Generator:
• This generates key by using Kerberos algorithm.
• Encrypts the input data with the key generated.
iii) Subnet Switch:
- This subnet switch works like a master.
- This assigns works to map workers.
- A large works is sub divided and is given to map
workers.
- It also checks whether all workers did their job or not.
- In case any crash in map worker the jobs will be
reassigned to it.

17. iv) Map workers:
• Works with the data key which was assigned by the subnet
switch.
• After completion of work it sends the result of job to subnets
switch and gets other jobs that is to be processed.
• Our concept mainly focuses on bottom up approach because
when going for bottom the data amount will be less so we can
analyze it easily. Further going, the data amount increases.
• Top down approach fails in big data because analyze on large
data is not possible.

18. BIG DATA GOOD PRACTICES
• Creating dimensions of all the data being stored.
• All data’s should have a durable keys.
• Structured and unstructured data are analyzed
together.
• Importance of big data to consumers is growing.
• Data quality needs to be better.
• The scalability has certain limits of storing the
data.
• Investment in quality and metadata reduce
processing time.

19. CONCLUSION
• This developed framework brings out a better
analysis of data over the previous methods.
• This method uses the map reduce concept
that reduces the analysing effort of data.
• By using bottom up approach the data
analysis is made easy and it also provide
better analysis result.
• This system uses map worker concept that
ensures that the data is analyzed in a best way
to use it.

20. ANY QUERIES?