This document provides an overview of big data and how it can be used to forecast and predict outcomes. It discusses how large amounts of data are now being collected from various sources like the internet, sensors, and real-world transactions. This data is stored and processed using technologies like MapReduce, Hadoop, stream processing, and complex event processing to discover patterns, build models, and make predictions. Examples of current predictions include weather forecasts, traffic patterns, and targeted marketing recommendations. The document outlines challenges in big data like processing speed, security, and privacy, but argues that with the right techniques big data can help further human goals of understanding, explaining, and anticipating what will happen in the future.
Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.
Content:
Introduction
What is Big Data?
Big Data facts
Three Characteristics of Big Data
Storing Big Data
THE STRUCTURE OF BIG DATA
WHY BIG DATA
HOW IS BIG DATA DIFFERENT?
BIG DATA SOURCES
BIG DATA ANALYTICS
TYPES OF TOOLS USED IN BIG-DATA
Application Of Big Data analytics
HOW BIG DATA IMPACTS ON IT
RISKS OF BIG DATA
BENEFITS OF BIG DATA
Future of big data
Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves.
Content:
Introduction
What is Big Data?
Big Data facts
Three Characteristics of Big Data
Storing Big Data
THE STRUCTURE OF BIG DATA
WHY BIG DATA
HOW IS BIG DATA DIFFERENT?
BIG DATA SOURCES
BIG DATA ANALYTICS
TYPES OF TOOLS USED IN BIG-DATA
Application Of Big Data analytics
HOW BIG DATA IMPACTS ON IT
RISKS OF BIG DATA
BENEFITS OF BIG DATA
Future of big data
Big Data may well be the Next Big Thing in the IT world. The first organizations to embrace it were online and startup firms. Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the beginning.
Disclaimer :
The images, company, product and service names that are used in this presentation, are for illustration purposes only. All trademarks and registered trademarks are the property of their respective owners.
Data/Image collected from various sources from Internet.
Intention was to present the big picture of Big Data & Hadoop
Big data nowadays is a new challenge to be managed, not as a barrier to grow up business. Data storages costs relatively is inexpensive, with more transactions generated from social media, machine, and sensors, data increased from pieces by pieces into pentabytes.
This slide explained what the challenges of Big Data (Volume, Velocity, and Variety) and give a solution how to managed them.
There are many tools that could help to solve the problems, but the main focus tools in this slide is Apache Hadoop.
A Seminar Presentation on Big Data for Students.
Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware.
everyone need to some storage and data.this big data is increase the data capacity and processing power.
Big Data may well be the Next Big Thing in the IT world.
• Big data burst upon the scene in the first decade of the 21st century.
• The first organizations to embrace it were online and startup firms. Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the beginning.
• Like many new information technologies, big data can bring about dramatic cost reductions, substantial improvements in the time required to perform a computing task, or new product and service offerings.
Big data is a huge volume of heterogenous data often generated at high speed.Big data cannot be handles with traditional data analytic tools. Hadoop is one of the mostly used big data analytic tool.Map Reduce, hive, hbase are also the tools for analysis in big data.
All about Big Data components and the best tools to ingest, process, store and visualize the data.
This is a keynote from the series "by Developer for Developers" powered by eSolutionsGrup.
Big Data Analysis : Deciphering the haystack Srinath Perera
A primary outcome of Bigdata is to derive useful and actionable insights from large or challenges data collections. The goal is to run the transformations from data, to information, to knowledge, and finally to insights. This includes calculating simple analytics like Mean, Max, and Median, to derive overall understanding about data by building models, and finally to derive predictions from data. Some cases we can afford to wait to collect and processes them, while in other cases we need to know the outputs right away. MapReduce has been the defacto standard for data processing, and we will start our discussion from there. However, that is only one side of the problem. There are other technologies like Apache Spark and Apache Drill graining ground, and also realtime processing technologies like Stream Processing and Complex Event Processing. Finally there are lot of work on porting decision technologies like Machine learning into big data landscape. This talk discusses big data processing in general and look at each of those different technologies comparing and contrasting them.
Big Data may well be the Next Big Thing in the IT world. The first organizations to embrace it were online and startup firms. Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the beginning.
Disclaimer :
The images, company, product and service names that are used in this presentation, are for illustration purposes only. All trademarks and registered trademarks are the property of their respective owners.
Data/Image collected from various sources from Internet.
Intention was to present the big picture of Big Data & Hadoop
Big data nowadays is a new challenge to be managed, not as a barrier to grow up business. Data storages costs relatively is inexpensive, with more transactions generated from social media, machine, and sensors, data increased from pieces by pieces into pentabytes.
This slide explained what the challenges of Big Data (Volume, Velocity, and Variety) and give a solution how to managed them.
There are many tools that could help to solve the problems, but the main focus tools in this slide is Apache Hadoop.
A Seminar Presentation on Big Data for Students.
Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware.
everyone need to some storage and data.this big data is increase the data capacity and processing power.
Big Data may well be the Next Big Thing in the IT world.
• Big data burst upon the scene in the first decade of the 21st century.
• The first organizations to embrace it were online and startup firms. Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the beginning.
• Like many new information technologies, big data can bring about dramatic cost reductions, substantial improvements in the time required to perform a computing task, or new product and service offerings.
Big data is a huge volume of heterogenous data often generated at high speed.Big data cannot be handles with traditional data analytic tools. Hadoop is one of the mostly used big data analytic tool.Map Reduce, hive, hbase are also the tools for analysis in big data.
All about Big Data components and the best tools to ingest, process, store and visualize the data.
This is a keynote from the series "by Developer for Developers" powered by eSolutionsGrup.
Big Data Analysis : Deciphering the haystack Srinath Perera
A primary outcome of Bigdata is to derive useful and actionable insights from large or challenges data collections. The goal is to run the transformations from data, to information, to knowledge, and finally to insights. This includes calculating simple analytics like Mean, Max, and Median, to derive overall understanding about data by building models, and finally to derive predictions from data. Some cases we can afford to wait to collect and processes them, while in other cases we need to know the outputs right away. MapReduce has been the defacto standard for data processing, and we will start our discussion from there. However, that is only one side of the problem. There are other technologies like Apache Spark and Apache Drill graining ground, and also realtime processing technologies like Stream Processing and Complex Event Processing. Finally there are lot of work on porting decision technologies like Machine learning into big data landscape. This talk discusses big data processing in general and look at each of those different technologies comparing and contrasting them.
ICTER 2014 Invited Talk: Large Scale Data Processing in the Real World: from ...Srinath Perera
Large scale data processing analyses and makes sense of large amounts of data. Although the field itself is not new, it is finding many usecases under the theme "Bigdata" where Google itself, IBM Watson, and Google's Driverless car are some of success stories. Spanning many fields, Large scale data processing brings together technologies like Distributed Systems, Machine Learning, Statistics, and Internet of Things together. It is a multi-billion-dollar industry including use cases like targeted advertising, fraud detection, product recommendations, and market surveys. With new technologies like Internet of Things (IoT), these use cases are expanding to scenarios like Smart Cities, Smart health, and Smart Agriculture. Some usecases like Urban Planning can be slow, which is done in batch mode, while others like stock markets need results within Milliseconds, which are done in streaming fashion. There are different technologies for each case: MapReduce for batch processing and Complex Event Processing and Stream Processing for real-time usecases. Furthermore, the type of analysis range from basic statistics like mean to complicated prediction models based on machine Learning. In this talk, we will discuss data processing landscape: concepts, usecases, technologies and open questions while drawing examples from real world scenarios.
http://icter.org/conference/invited_speeches
Big Data and Data Science: The Technologies Shaping Our LivesRukshan Batuwita
Big Data and Data Science have become increasingly imperative areas in both industry and academia to the extent that every company wants to hire a Data Scientist and every university wants to start dedicated degree programs and centres of excellence in Data Science. Big Data and Data Science have led to technologies that have already shaped different aspects of our lives such as learning, working, travelling, purchasing, social relationships, entertainments, physical activities, medical treatments, etc. This talk will attempt to cover the landscape of some of the important topics in these exponentially growing areas of Data Science and Big Data including the state-of-the-art processes, commercial and open-source platforms, data processing and analytics algorithms (specially large scale Machine Learning), application areas in academia and industry, the best industry practices, business challenges and what it takes to become a Data Scientist.
Book: Software Architecture and Decision-MakingSrinath Perera
Uncertainty is the leading cause of mistakes made by practicing software architects. The primary goal of architecture is to handle uncertainty arising from user cases as well as architectural techniques. The book discusses how to make architectural decisions and manage uncertainty. From the book, You will learn common problems while designing a system, a default solution for each, more complex alternatives, and 5Q & 7P (Five Questions and Seven Principles) that help you choose.
Book, https://amzn.to/3v1MfZX
Blog: http://tinyurl.com/swdmblog
Six min video - https://youtu.be/jtnuHvPWlYU
We have critically evaluated how AI will shape integration use cases, their feasibility, and timelines. Emerging Technology Analysis Canvas (ETAC), a framework built to analyze emerging technologies, is the methodology of our study.
We observe that AI can significantly impact integration use cases and identify 13 AI-based use case classes for integration. Points to note include:
Enabling AI in an enterprise involves collecting, cleaning up, and creating a single representation of data as well as enforcing decisions and exposing data outside, each of which leads to many integration use cases. Hence, AI indirectly creates demand for integration.
AI needs data, which in some cases lead to significant competitive advantages. The need to collect data would drive vendors to offer most AI products in the cloud through APIs.
Due to lack of expertise and data, custom AI model building will be limited to large organizations. It is hard for small and medium size organization to build and maintain custom models.
The Role of Blockchain in Future IntegrationsSrinath Perera
We have critically evaluated blockchain-based integration use cases, their feasibility, and timelines. Emerging Technology Analysis Canvas (ETAC), a framework built to analyze emerging technologies, is the methodology of our study. Based on our analysis, we observe that blockchain can significantly impact integration use cases.
In our paper, we identify 30-plus blockchain-based use cases for integration and four architecture patterns. Notably, each use case we identified can be implemented using one of the architecture patterns. Furthermore, we also discuss challenges and risks posed by blockchains that would affect these architecture patterns.
Our webinar presents a critical analysis of serverless technology and our thoughts about its future. We use Emerging Technology Analysis Canvas (ETAC), a framework built to analyze emerging technologies, as the methodology of our study. Based on our analysis, we believe that serverless can significantly impact applications and software development workflows.
We’ve also made two further observations:
Limitations, such as tail latencies and cold starts, are not deal breakers for adoption. There are significant use cases that can work with existing serverless technologies despite these limitations.
We see a significant gap in required tooling and IDE support, best practices, and architecture blueprints. With proper tooling, it is possible to train existing enterprise developers to program with serverless. If proper tools are forthcoming, we believe serverless can cross the chasm in 3-5 years.
A detailed analysis can be found here: A Survey of Serverless: Status Quo and Future Directions. Join our webinar as we discuss this study, our conclusions, and evidence in detail.
1. Blockchain potential impact is real. If successful, Blockchain technologies can transform the way we live our day to day lives.
2. We believe technology is ready for limited applications in Digital Currency, Lightweight financial systems, Ledgers (of identity, ownership, status, and authority), Provenance (e.g. supply chains and other B2B scenarios) and Disintermediation, which we believe will happen in next three years.
3. However, with other use cases, blockchain faces significant challenges such as performance, irrevocability, need for regulation and lack of census mechanisms. These are hard problems and
4. It is not clear whether blockchain can sustain the current level of effort for extended period of 5+ years. There are many startups and they run the risk of running out of money before markets are ready. Failure of startups can inhibit further funding and investments.
5. Value and need of decentralization compared to centralized and semi-centralized alternatives is not clear.
A Visual Canvas for Judging New TechnologiesSrinath Perera
In the fast-changing technology world, the technology landscape shifts faster and faster. The agents of thses changes are new emerging technologies, which sometimes even create, destroy, or transform segments. In a shifting world, prevailing advantages are fleeting. Organizations that can master change and ride technology waves owns the future.
Not all emerging technologies live up to their promise. Every year, as a part of annual planning, most organizations need to decide relevance, impact, and the probability of success of emerging technologies and pick their bets. Although it is a regular decision there is no widely accepted framework for evaluating emerging technologies.
As a solution to this problem, we present “Emerging Technology Analysis Canvas” (ETAC), a framework to assess an individual emerging technology as a solution to this problem. Inspired by the Business Model Canvas, It represents different aspects of technology visually on a single page. This approach includes a set of questions that probe the technology arranged around a logical narrative. The visual representation is concise, compact, and comprehensible in a glance.
The talk discusses how analytics can attack privacy and what we can do about it. It discusses the legal responses (e.g. GDPR) as well technical responses ( differential privacy and homomorphic encryption).
The video is in https://www.facebook.com/eduscopelive/videos/314847475765297/ from 1.18.
Blockchain is often cited as one of the most impactful technology along with AI. It has attracted many startups, venture investments, and academic research. If successful, Blockchain technologies can transform the way, we live our day to day lives.
However, blockchain faces significant challenges such as performance, irrevocability, need for regulation and lack of census mechanisms. They are hard problems, and likely it will take at least 5-10 years to find answers to those problems.
Given the risk involved as well as the significant potential returns, we recommend a cautiously optimistic approach for blockchain with the focus on concrete use cases.
Today's Technology and Emerging Technology LandscapeSrinath Perera
We have seen the rise and fall of many technologies, some disappearing without a trace while others redefining the world. Collectively they have shaped our world beyond recognition. In this talk, Srinath will start with past technologies exploring their behavior. Then he will explore current middleware landscape, its composition, and relationships between different segments. He will discuss significant developments and discuss their future. Further, he will discuss emerging technologies, forces that shape them, and the promise of each technology, and finally, speculate about their evolution. You will walk away with knowledge on the evolution of middleware, the status quo, and discussion about how, at WSO2, we think those technologies will evolve.
Some died, some get by, but some have woven themselves to today's middleware so much that we do not notice them. The point I want to make is that not all emerging technologies are fads. Some are, and some are too early, like AI. But some are lasting.
The Rise of Streaming SQL and Evolution of Streaming ApplicationsSrinath Perera
First-generation stream processors, such as Apache Storm, wanted us to write code. It was a great start. However, when building real-world apps, which are used for a long time and evolve, writing code gets us into trouble.
If we want to query a database or query data stored in storage with Hadoop, we use SQL. Why can't we query data streaming using SQL? We can. Almost all open source stream processors, including Storm, Flink, and Kafka, have switched to SQL.
In this webinar, Srinath will talk about the evolution of stream processing, streaming SQL, the status quo, and what this means to stream applications. He will also dissect the experience of building streaming applications by exploring common patterns and pitfalls.
Analytics and AI: The Good, the Bad and the UglySrinath Perera
Analytics let us question the data, which in effect questions the world around us. This let us understand, monitor, and shape the world. AI let us discover connections, predict the possible futures and automate tasks.
These twin technologies can change the world around us. On one hand, make us efficient, connected, and fulfilled. At the same time, the change of status quo can replace jobs, affect lives and build biases into our systems that can marginalize millions.
In this talk, we will discuss core ideas behind analytics and AI, their possible impact, both good and bad outcomes, and challenges.
The dawn of digital businesses is upon us, with reimagined business models that make the best use of digital technologies such as automation, analytics, integration and cloud. Digital businesses are efficient, continuously optimizing, proactive, flexible and are able to fully understand their customers. Analytics is a key technology that helps in doing so. It acts as the eyes and ears of the system and provides a holistic view on the past and present so that decision-makers can predict what will happen in the future. This webinar will explore
Why becoming a digital business is not a choice
The role of analytics in digital transformation with examples
How best to leverage state of the art analytics technology
SoC Keynote:The State of the Art in Integration TechnologySrinath Perera
This talk discusses Outline of the state of the art of Enterprise Software and how we get there, as I see it. Also second part describes Ballerina, a new programming language WSO2 has built for Enterprise Computing.
It is presented as a Keynote at 11th Symposium and Summer School On Service-Oriented Computing.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Machine learning and optimization techniques for electrical drives.pptx
Introduction to Big Data
1.
2. Outline• Forecasts and
Why Big data?
• What is Big
Data?
• Parts of the
Puzzle
• Quick Tutorial
• Conclusion
Photo by John Trainoron Flickr
http://www.flickr.com/photos/trainor/2902023575/, Licensed under CC
3. Asimov Foundation
• 20 million worlds
• A Scientist who
mathematically calculate
the fate of the Universe
• His effort to change that
Fate
• A Beautiful story
4. Consider a day in your life
• What is the best road to take?
• Would there be any bad
weather?
• What is the best way to invest
the money?
• Should I take that loan?
• Can I optimize my day?
• Is there a way to do this
faster?
• What have others done in
similar cases?
• Which product should I buy?
5. People wanted to (through ages)
• To know (what
happened?)
• To Explain (why it
happened)
• To Predict (what will
happen?)
6. Many Cultures had
Claim for Omniscience
• Oracle
• Astrology
• Book of Changes
• Tarot Cards
• Crystal balls
Claim for Omniscience
7. To know, explain and predict!!
• Grand challenge of our time
• We have been trying to do this in
many other means
• Now we trying to do this via
science
Any sufficiently advanced technology
is indistinguishable from magic.
---Arthur C. Clarke.
• We see a possibilities though “lot
of data”
8. You does not seem to be convinced!!
• Why, sometimes I've believed as many as six
impossible things before breakfast.
9. Prophecies of our time
• We can predict Weather
• We do understand language
translation pretty well
• We can predict how an air
plane behave good enough to
fly
• Our ability on forensics
• We understand remedies for
many diseases
• We can tell how astro bodies
will behave (e.g. comets)
10. This is being Done: Weather
• Remarkably accurate for few
days
– SL incident
• Data :- weather radars, satellite
data, weather balloons, planes
etc.,
• forecast models
– Simulation
– Numerical method
• Challenge is computing power
– algorithms that take more than
24 hours
– resolution is the key ……
11. Democratizing Analysis
• Forecasting was done, but in limited manner
and only by few (e.g. National Labs,
Intelligence community)
• That is changing!!
12. What is Big data?
• There is lot of data available
– E.g. Internet of things
• We have computing power
• We have technology
• Goal is same
– To know
– To Explain
– To predict
• Challenge is the full lifecycle
13. Data, the wealth of our time
"Data is a precious
thing because they last
longer than systems” -
Tim Barnes Lee
• Access to data is becoming ultimate
competitive advantage
– E.g. Google+ vs. Facebook
– Why many organizations try hard to give us free
things and keep us always logged in (e.g. Gmail,
facebook, search engine tool bars)
15. Data Avalanche/ Moore’s law of data
• We are now collecting and converting large amount
of data to digital forms
• 90% of the data in the world today was created
within the past two years.
• Amount of data we have doubles very fast
16. In real life, most data are Big
• Web does millions of activities per second, and so
much server logs are created.
• Social networks e.g. Facebook, 800 Million active
users, 40 billion photos from its user base.
• There are >4 billion phones and >25% are smart
phones. There are billions of RFID tags.
• Observational and Sensor data
– Weather Radars, Balloons
– Environmental Sensors
– Telescopes
– Complex physics simulations
17. Why Big Data is hard?
• How store? Assuming 1TB bytes it
takes 1000 computers to store a 1PB
• How to move? Assuming 10Gb
network, it takes 2 hours to copy 1TB,
or 83 days to copy a 1PB
• How to search? Assuming each record
is 1KB and one machine can process
1000 records per sec, it needs 277CPU
days to process a 1TB and 785 CPU
years to process a 1 PB
• How to process?
– How to convert algorithms to work in large
size
– How to create new algorithms
http://www.susanica.com/photo/9
18. Why it is hard (Contd.)?
• System build of many
computers
• That handles lots of data
• Running complex logic
• This pushes us to frontier of
Distributed Systems and
Databases
• More data does not mean
there is a simple model
• Some models can be complex
as the system
http://www.flickr.com/photos/mariachily/5250487136,
Licensed CC
20. Sensors
• There are sensors everywhere
– RFID (e.g. Walmart), GPS
sensors, Mobile Phone …
• Internet
– Click streams, Emails, chat,
search, tweets ,Transactions …
• Real Word
– Video surveillance, Cash flows,
Traffic, Surveillance, Stock
exchange, Smart Grid,
Production line …
– Internet of Things
http://www.flickr.com/photos/imuttoo/4257813689/ by Ian Muttoo,
http://www.flickr.com/photos/eastcapital/4554220770/,
http://www.flickr.com/photos/patdavid/4619331472/ by Pat David copyright CC
21. Collecting Data
• Data collected at sensors and sent to big data
system via events or flat files
• Event Streams: we name the events by its
content/ originator
• Get data through
– Point to Point
– Event Bus
• E.g. Data bridge
– a thrift based transport we
did that do about 400k
events/ sec
22. Storing Data
• Historically we used databases
– Scale is a challenge: replication,
sharding
• Scalable options
– NoSQL (Cassandra, Hbase) [If
data is structured]
• Column families Gaining Ground
– Distributed file systems (e.g.
HDFS) [If data is unstructured]
• New SQL
– In Memory computing, VoltDB
• Specialized data structures
– Graph Databases, Data structure
servers http://www.flickr.com/photos/keso/3631339
67/
23. Making Sense of Data
• To know (what happened?)
– Basic analytics +
visualizations (min, max,
average, histogram,
distributions … )
– Interactive drill down
• To explain (why)
– Data mining, classifications,
building models, clustering
• To forecast
– Neural networks, decision
models
24. To know (what happened?)
• Mainly Analytics
– Min, Max, average,
correlation, histograms
– Might join group data in many
ways
• Implemented with
MapReduce or Queries
• Data is often presented with
some visualizations
• Examples
– forensics
– Assessments
– Historical data/ reports/
trends
http://www.flickr.com/photos/isriya/2967310
333/
25. Search
• Process and Index the data. The killer app of
our time.
http://pixabay.com, by Steven Iodice
• Web Search
• Graph Search
• Semantic Search
• Drug Discovery
• …
26. Patterns
• Correlation
– Scatter plot, statistical
correlation
• Data Mining (Detecting
Patterns)
– Clustering and classification
– Finding Similar items
– Finding Hubs and authorities
in a Graph
– Finding frequent item sets
– Making recommendation
• Apache Mahout
http://www.flickr.com/photos/eriwst/2987739376/ and http://www.flickr.com/photos/focx/5035444779/
27. Forecasts and Models
• Trying to build a model for the
data
• Theoretically or empirically
– Analytical models (e.g. Physics)
– Neural networks
– Reinforcement learning
– Unsupervised learning (clustering,
dimensionality reduction, kernel
methods)
• Examples
– Translation
– Weather Forecast models
– Building profiles of users
– Traffic models
– Economic models
• Remember: Correlation does not
mean causality
http://misterbijou.blogspot.com/201
0_09_01_archive.html
28. Information Visualization
• Presenting information
– To end user
– To decision takers
– To scientist
• Interactive exploration
• Sending alerts
http://www.flickr.com/photos/stevefae
embra/3604686097/
31. Usecase 1: Targeted Marketing
• Collect data and build a model on
– What user like
– What he has brought
– What is his buying power
• Making recommendations
• Giving personalized deals
32. Usecase 2: Travel
• Collect traffic data +
transportation data from
sensors
• Build a model (e.g. Car
Following Models)
• Predict the traffic ..
Possibilities on congestion
• E.g. divert traffic, adjust
troll, adjust traffic lights
33. Practical Tutorial
• Data collection
– Pub/sub, event architectures
• Processing
– Store and Process: MapReduce/Hadoop
– Processing Moving Data: CEP
• Visualization
– GNU Plot/ R
34. DEBS Challenge
• Event Processing
challenge
• Real football game,
sensors in player
shoes + ball
• Events in 15k Hz
• Event format
– Sensor ID, TS, x, y, z, v,
a
• Queries
– Running Stats
– Ball Possession
– Heat Map of Activity
– Shots at Goal
35. MapReduce/ Hadoop
• First introduced by Google,
and used as the processing
model for their architecture
• Implemented by opensource
projects like Apache Hadoop
and Spark
• Users writes two functions:
map and reduce
• The framework handles the
details like distributed
processing, fault tolerance,
load balancing etc.
• Widely used, and the one of
the catalyst of Big data
void map(ctx, k, v){
tokens = v.split();
for t in tokens
ctx.emit(t,1)
}
void reduce(ctx, k, values[]){
count = 0;
for v in values
count = count + v;
ctx.emit(k,count);
}
40. How long player Spend Near the
ball?(Contd.)
void map(ctx, k, v){
event = parse(v);
cell = calcuateOverlappingCells(v.x,
v.y, v.z);
ctx.emit(cell,v.x, v.y, v.z);
}
Void reduce(ctx, k, values[]){
players = joinAndFindPlayersNearBall(values);
for p in players
ctx.emit(p,p);
}
41. Hadoop Landscape
• HIVE - Query data using SQL style queries, and Hive will
convert them to MapReduce jobs and run in Hadoop.
• Pig - We write programs using data flow style scripts, and
Pig convert them to MapReduce jobs and run in Hadoop.
• Mahout
– Collection of MapReduce jobs implementing many Data
mining and Artificial Intelligence algorithms using MapReduce
A = load 'hdi-data.csv' using PigStorage(',')
AS (id:int, country:chararray, hdi:float,
lifeex:int, mysch:int, eysch:int, gni:int);
B = FILTER A BY gni > 2000;
C = ORDER B BY gni;
dump C;
hive> SELECT country, gni from HDI WHERE gni > 2000;
42. Is Hadoop Enough?
• Limitations
– Takes time for processing
– Lack of Incremental processing
– Weak with Graph usecases
– Not very easy to create a processing pipeline (addressed
with HIVE, Pig etc.)
– Too close to programmers
– Faster implementations are possible
• Alternatives
– Apache Drill http://incubator.apache.org/drill/
– Spark http://spark-project.org/
– Graph Processors like http://giraph.apache.org/
43. Data In the Move
• Idea is to process data as they
are received in streaming
fashion
• Used when we need
– Very fast output
– Lots of events (few 100k to
millions)
– Processing without storing (e.g.
too much data)
• Two main technologies
– Stream Processing (e.g. Strom,
http://storm-project.net/ )
– Complex Event Processing (CEP)
http://wso2.com/products/complex
-event-processor/
44. Complex Event Processing (CEP)
• Sees inputs as Event streams and queried with
SQL like language
• Supports Filters, Windows, Join, Patterns and
Sequences
from p=PINChangeEvents#win.time(3600) join
t=TransactionEvents[p.custid=custid][amount>1000
0] #win.time(3600)
return t.custid, t.amount;
45. Example: Detect ball Possession
• Possession is time a
player hit the ball
until someone else
hits it or it goes out
of the ground
from Ball#window.length(1) as b join
Players#window.length(1) as p
unidirectional
on debs: getDistance(b.x,b.y,b.z,
p.x, p.y, p.z) < 1000
and b.a > 55
select ...
insert into hitStream
from old = hitStream ,
b = hitStream [old. pid != pid ],
n= hitStream[b.pid == pid]*,
( e1 = hitStream[b.pid != pid ]
or e2= ballLeavingHitStream)
select ...
insert into BallPossessionStream
http://www.flickr.com/photos/glennharper/146164820/
46. GNU Plot
set terminal png
set output "buyfreq.png"
set title "Frequency Distribution of Items
brought by Buyer";
setylabel "Number of Items Brought";
setxlabel "Buyers Sorted by Items count";
set key left top
set log y
set log x
plot "1.data" using 2 title "Frequency"
with linespoints
• Open source
• Very powerful
47. Big data lifecycle
• Realizing the big data lifecycle is hard
• Need wide understanding about many fields
• Big data teams will include members from
many fields working together
48. • Data Scientists: A new role, evolving Job
description
• His Job is to make sense of data, by using Big data
processing, advance algorithms, Statistical
methods, and data mining etc.
• Likely to be a highest paid/ cool Jobs.
• Big organizations (Google, Facebook, Banks) hiring
Math PhD by a lot for this.
Data Scientist
Statisticians
System experts
Domain Experts
DB experts
AI experts
49. Dark Side
• Privacy
– Invasion of privacy
– Data might be used for
unexpected things
• Big Brother
– Data likely to used for
control (e.g. governments)
• If technology is out there,
may be it is OK. It is very
hard to hide any thing,
which work both ways
50. Challenges
• Speed (e.g. targeted advertising, reacting to data)
• Extracting semantics and handling multiple
representations and formats
• Security Data ownership, delegation, permissions,
and Privacy
• Making data accessible to all intended parties,
from anywhere, anytime, from any device,
through any format (subjected to permissions).
• Map-Reduce good enough? What about other
parallel problems?
• Handling Uncertainty
51. Conclusions
• Lot of data, realizing data -> insight -> predications
• We do lot of predications even now.
• There is lot between data and forecasts, OK to do
them.
– Analytics
– Visualizations
– Patterns
– Data mining
• If you looking to start, learn MapReduce, CEP, and
GNU plot.
• Visualization is the key!!
• Learn some distributed systems and AI