Downloaded 10 times
Uploaded byDeddy Setyadi
Big data
This document discusses big data, where it comes from, and how it is processed and analyzed. It notes that everything we do online now leaves a digital trace as data. This "big data" includes huge volumes of structured, semi-structured, and unstructured data from various sources like social media, sensors, and the internet of things. Traditional computing cannot handle such large datasets, so technologies like MapReduce, Hadoop, HDFS, and NoSQL databases were developed to distribute the work across clusters of machines and process the data in parallel.
More Related Content
Big data
- 1.
- 2. ... - 2003 2days in 2011 10 minutes in 2013 5 billion GB Live stats 2016
- 3. Where those datacomes from? Activity Listening music, reading a book, searching, shopping, etc. Our conversations in social media are now digitally recorded.Conversation We upload and share 100s of thousands of them on social media sites every second. Photo and Video We are increasingly surrounded by sensors that collect and share data. Sensor We now have smart TVs that are able to collect and process data.The Internet of Things
- 4. The basic ideabehind the phrase 'Big Data' is that everything we do is increasingly leaving a digital trace (or data), which we (and others) can use and analyse
- 5. Big data : meansreally a big data, it is a collection of large datasets that cannot be processed using traditional computing techniques.
- 6. Big Data includeshuge volume, high velocity, and extensible variety of data. Structured Item 2 Semi Structured Unstructured ● Database ● Census records ● Economic data ● Phone numbers ● JSON ● XML ● Word ● PDF ● Text ● Media Logs
- 7. Benefits of BigData https://www.youtube.com/watch?v=HqsBensINkE
- 8. Big Data Technologies OperationalBig Data This include systems like MongoDB that provide operational capabilities for real- time, interactive workloads where data is primarily captured and stored. NoSQL Big Data systems are designed to allow massive computations to be run inexpensively and efficiently. This makes operational big data workloads much easier to manage, cheaper, and faster to implement. Analytical Big Data This includes systems like Massively Parallel Processing (MPP) database systems and MapReduce that provide analytical capabilities for retrospective and complex analysis. A system based on MapReduce can be scaled up from single servers to thousands of high and low end machines.
- 9.
- 10. Traditional Approach In thisapproach, an enterprise will have a computer to store and process big data. Here data will be stored in an RDBMS, process the required data and present it to the users for analysis purpose. tutorialspoint.com
- 11. Google’s Solution Google solved thisproblem using an algorithm called MapReduce. This algorithm divides the task into small parts and assigns those parts to many computers connected over the network, and collects the results to form the final result dataset. tutorialspoint.com
- 12. Hadoop Hadoop runs applicationsusing the MapReduce algorithm, where the data is processed in parallel on different CPU nodes. In short, Hadoop framework is capable enough to develop applications, capable of running on clusters of computers and they could perform complete statistical analysis for a huge amounts of data. tutorialspoint.com
- 13.
- 14.
- 15. MapReduce Data Map Converts data intoanother set of data. Elements are broken down into tuples (key/value pairs). Reduce Shuffle stage and the Reduce stage that produces a new set of output, which will be stored in the HDFS. 1 2 3
- 16.
- 17.
- 18. HDFS ● Fault detectionand recovery : HDFS should have mechanisms for quick and automatic fault detection and recovery. ● Huge datasets : HDFS should have hundreds of nodes per cluster to manage the applications having huge data sets. ● Hardware at data : A requested task can be done efficiently. tutorialspoint.com
- 19.
- 20.
- 21.
- 22.
Editor's Notes
- #3 Perkembangan teknologi, alat, dan media komunikasi yang semakin pesat, berbanding lurus dengan jumlah data yang dihasilkan oleh umat manusia. Dari awal terbentuknya bumi sampai 2003, ketika bilik-bilik warnet masih sepi, dan internet masih benda asing, data yang dihasilkan umat manusia itu sebanyak 5 milliar GB. Kemudian di tahun-tahun berikutnya, muncul friendster, facebook, twitter, pun perangkat baru mulai bermunculan seperti ipod, nokia yang dibekali dengan gprs sehingga umat manusia mulai menggunakan internet. Delapan tahun berlalu, blackberry mulai booming, disertai dengan whatsapp, twitter, dan dalam 2 hari mampu memproduksi 5 milyar GB meskipun untuk paketan internet saat itu masih eman-eman. Android pun mulai menjamur beberapa tahun sesudahnya, pengguna pun mulai banyak, umat manusia sudah mulai terbiasa dengan paketan internet dan akhirnya data sebanyak 5 milyar GB dapat diproduksi dalam eaktu 10 menit.
- #4 Simple activities like listening to music or reading a book are now generating data. Digital music players and eBooks collect data on our activities. Your smart phone collects data on how you use it and your web browser collects information on what you are searching for. Your credit card company collects data on where you shop and your shop collects data on what you buy. It is hard to imagine any activity that does not generate data. Our conversations are now digitally recorded. It all started with emails but nowadays most of our conversations leave a digital trail. Just think of all the conversations we have on social media sites like Facebook or Twitter. Even many of our phone conversations are now digitally recorded. Just think about all the pictures we take on our smart phones or digital cameras. We upload and share 100s of thousands of them on social media sites every second. The increasing amounts of CCTV cameras take video images and we up-load hundreds of hours of video images to YouTube and other sites every minute . We are increasingly surrounded by sensors that collect and share data. Take your smart phone, it contains a global positioning sensor to track exactly where you are every second of the day, it includes an accelometer to track the speed and direction at which you are travelling. We now have sensors in many devices and products. We now have smart TVs that are able to collect and process data, we have smart watches, smart fridges, and smart alarms. The Internet of Things, or Internet of Everything connects these devices so that e.g. the traffic sensors on the road send data to your alarm clock which will wake you up earlier than planned because the blocked road means you have to leave earlier to make your 9am meeting…
- #7 Volume refers to the vast amounts of data generated every second. We are not talking Terabytes but Zettabytes or Brontobytes. If we take all the data generated in the world between the beginning of time and 2008, the same amount of data will soon be generated every minute. New big data tools use distributed systems so that we can store and analyse data across databases that are dotted around anywhere in the world. Velocity refers to the speed at which new data is generated and the speed at which data moves around. Just think of social media messages going viral in seconds. Technology allows us now to analyse the data while it is being generated (sometimes referred to as in-memory analytics), without ever putting it into databases. Variety refers to the different types of data we can now use. In the past we only focused on structured data that neatly fitted into tables or relational databases, such as financial data. In fact, 80% of the world’s data is unstructured (text, images, video, voice, etc.) With big data technology we can now analyse and bring together data of different types such as messages, social media conversations, photos, sensor data, video or voice recordings.
- #11 Limitation : This approach works well where we have less volume of data that can be accommodated by standard database servers, or up to the limit of the processor which is processing the data. But when it comes to dealing with huge amounts of data, it is really a tedious task to process such data through a traditional database server.