Your SlideShare is downloading. ×
0
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Dataintensive
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Dataintensive

83

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
83
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. SULFATH.M NO:51 CPS5
  • 2. WHAT IS DATA INTENSIVE COMPUTING? INTRODUCTION NEED OF THIS COMPUTING DATA-INTENSIVE vs COMPUTE-INTENSIVE DATA PARALLELLISM CHARACTERISTICS  PROCESSING APPROACH
  • 3. COMMON CHARACTERISTICS LIFE CYCLE DATA-INTENSIVE SYSTEM CHALLENGE SYSTEM ARCHITECTURE MAPREDUCE HADOOP HPCC ADVANTAGES
  • 4. DATA INTENSIVE COMPUTING DATA INTENSIVE COMPUTING is a class of parallel computing applications  It use a data parallel approach to - Big Data. Computing applications-compute-intensive  whereas computing applications which require large volumes of data and devote most of their processing time to I/O and manipulation of data are deemed dataintensive.
  • 5. INTRODUCTIO NEED OF THIS COMPUTING The rapid growth of the Internet N and World Wide Web has led to vast amounts of information available online business and government organizations create large amounts of both structured and unstructured information which needs to be processed, analyzed, and linked. An IDC white paper sponsored by EMC estimated the amount of information currently
  • 6. Another study of information explosion it was estimated that 95% of all current information exists in unstructured form. The storing, managing, accessing, and processing of this vast amount of data represents a fundamental need and an immense challenge in order to satisfy needs to search, analyze, mine, and visualize this data as information.
  • 7. Why So Much Data? We Can Get It Automation + Internet We Can Keep It Seagate 1 TB Barracuda We Can Use It Scientific breakthroughs Business process efficiencies Realistic special effects Better health care Could We Do More? Apply more computing power to this data
  • 8. DATA-INTENSIVE vs COMPUTE-INTENSIVE Parallel processing approaches can be generally classified as either compute-intensive, or data-intensive. Compute-intensive is used to describe application programs that are compute bound. Such applications devote most of their execution time to computational requirements as opposed to I/O, and typically require small volumes of data Parallel processing of compute-intensive applications typically involves parallelizing individual algorithms within
  • 9. which can then be executed in parallel on an appropriate . computing platform to achieve overall higher performance than serial processing.  In compute-intensive applications, multiple operations are performed simultaneously, with each operation addressing a particular part of the problem. This is often referred to as task parallelism.
  • 10. Data-intensive is used to describe applications that are I/O bound or with a need to process large volumes of data. Such applications devote most of their processing time to I/O and movement and manipulation of data. Parallel processing of data-intensive applications typically involves partitioning or subdividing the data into multiple segments which can be processed independently using the same executable application program in parallel on an appropriate
  • 11. DATA-PARALLELISM Computer system architectures which can support data parallel applications are a potential solution to the terabyte &petabyte scale data processing  Data-parallelism can be defined as a computation applied independently to each data item of a set of data which allows the degree of parallelism to be scaled with the volume of data. The most important reason for developing
  • 12. The key issues with developing applications using data-parallelism are are the choice of the algorithm the strategy for data decomposition load balancing on processing nodes message passing communications b/w nodes the overall accuracy of the results.
  • 13. CHARACTERISTICS OF DATAINTENSIVE COMPUTING SYSTEMS The National Science Foundation believes that data-intensive computing requires a “fundamentally different set of principles” NSF is seeking to “increase understanding of the capabilities and limitations of data-intensive computing. The key areas of focus are: Approaches to parallel programming to address the parallel
  • 14. models, languages, and algorithms which allow a natural expression of parallel processing of data. Design of data-intensive computing platforms to provide high levels of reliability, efficiency, availability, and scalability. Identifying applications that can exploit this computing paradigm and determining how it should evolve to support emerging data-intensive applications
  • 15. Pacific Northwest National Labs has defined data-intensive computing as “capturing, managing, analyzing, and understanding data at volumes and rates that push the frontiers of current technologies”. They believe that to address the rapidly growing data volumes and complexity requires “epochal advances in software, hardware, and algorithm development” which can scale readily with size of the data and provide effective and timely
  • 16. PROCESSING APPROACH Current data-intensive computing platforms typically use a parallel computing approach combining multiple processors and disks in large commodity computing clusters connected using high-speed communications switches and networks  which allows the data to be partitioned among the available computing resources and processed independently. A cluster can be defined as a type of parallel and distributed system, which consists of a collection of inter-connected stand-alone
  • 17. This approach to parallel processing is often referred to as a “shared nothing” approach since each node consisting of processor, local memory, and disk resources shares nothing with other nodes in the cluster. In parallel computing this approach is considered suitable for data-intensive computing and problems which are “embarrassingly parallel” where it is relatively easy to separate the problem into a number of parallel tasks These types of data processing problems are adaptable to various forms of distributed
  • 18. COMMON CHARACTERISTICS  There are several important common characteristics of data-intensive computing systems that distinguish them from other forms of computing: 1. minimize the movement of data 2. The programming model utilized ,machine-independent, high-level operations on data 3. systems are designed to be fault resilient .includes redundant copies of all data files on disk, storage of intermediate processing results on
  • 19. LIFE CYCLE OF BIG DATA Life-cycle for big data , which consists of the following steps: 1. Data Generation 2. Data Processing and Organization 3. Data Analytics, Mining and Knowledge Discovery 4. Actions, Feedback and Refinement
  • 20. Data-Intensive System Challenge For Computation That Accesses 1 TB in 5 minutes  Data distributed over 100+ disks •Assuming uniform data partitioning Compute using 100+ processors Connected by gigabit Ethernet (or equivalent) System Requirements Lots of disks Lots of processors Located in close proximity •Within reach of fast, local-area network
  • 21. SYSTEM ARCHITECTURE A variety of system architectures have been implemented for data-intensive computing and large-scale data analysis applications including parallel and distributed relational database management systems. Several solutions have emerged including the  MapReduce architecture pioneered by Google open-source implementation called Hadoop used
  • 22. MapReduce Programming Model The MapReduce architecture and programming model pioneered by Google is an example of a modern systems architecture designed for data-intensive computing. It allows programmers to use a functional programming style  map function that processes a key-value pair associated with the input data to
  • 23. Characteristics Computation broken into many, short-lived tasks •Mapping, reducing Use disk storage to hold intermediate results Strengths Great flexibility in placement, scheduling, and load balancing Handle failures by recomputation Can access large data sets Weaknesses Higher overhead Lower raw performance
  • 24. Hadoop is an open source software project sponsored by The Apache Software Foundation which implements the MapReduce architecture. Supported and used by Yahoo Prototype on single machine, map onto cluster Pig was developed at Yahoo! to provide a specific language notation for data analysis applications and to improve programmer productivity and
  • 25. HPCC It stands forHigh-Performance Computing Cluster developed and implemented by LexisNexis Risk Solutions. The LexisNexis approach also utilizes commodity clusters of hardware running the Linux operating system. LexisNexis also implemented a new highlevel language for data-intensive computing called ECL.  The ECL programming language is the
  • 26. Advantages for DISC Systems Focus on Data Terabytes, not tera-FLOPS Problem-Centric Programming Platform-independent expression of data parallelism Interactive Access From simple queries to massive computations Robust Fault Tolerance Component failures are handled as routine events

×