The document discusses big data, including what it is, key factors enabling its growth like increased storage and processing power, and techniques for handling big data like distributed processing and NoSQL databases. It provides examples of tools and applications for big data and discusses challenges like ensuring patterns found in big data analysis are actually meaningful.
Learn Big data and Hadoop online at Easylearning Guru. We are offer Instructor led online training and Life Time LMS (Learning Management System). Join Our Free Live Demo Classes of Big Data Hadoop .
"Big Data" is big business, but what does it really mean? How will big data impact industries and consumers? This slide deck goes through some of the high level details of the market and how it is revolutionizing the world.
History of NoSQL and Azure Documentdb feature setSoner Altin
Short history of database systems from DBMS, RDBMS to NoSQL solutions. Introduction to SQL query support of Azure DocumentDB and integrating DocumentDB with simple Java application from Maven repository.
Combining Data Mining and Machine Learning for Effective User ProfilingCodePolitan
Slide presentasi ini dibawakan oleh Anne Regina pada Seminar & Workshop Pengenalan & Potensi Big Data & Machine Learning yang diselenggarakan oleh KUDIO pada tanggal 14 Mei 2016.
Big Data Analysis Patterns with Hadoop, Mahout and Solrboorad
Big Data Analysis Patterns: Tying real world use cases to strategies for analysis using big data technologies and tools.
Big data is ushering in a new era for analytics with large scale data and relatively simple algorithms driving results rather than relying on complex models that use sample data. When you are ready to extract benefits from your data, how do you decide what approach, what algorithm, what tool to use? The answer is simpler than you think.
This session tackles big data analysis with a practical description of strategies for several classes of application types, identified concretely with use cases. Topics include new approaches to search and recommendation using scalable technologies such as Hadoop, Mahout, Storm, Solr, & Titan.
Learn Big data and Hadoop online at Easylearning Guru. We are offer Instructor led online training and Life Time LMS (Learning Management System). Join Our Free Live Demo Classes of Big Data Hadoop .
"Big Data" is big business, but what does it really mean? How will big data impact industries and consumers? This slide deck goes through some of the high level details of the market and how it is revolutionizing the world.
History of NoSQL and Azure Documentdb feature setSoner Altin
Short history of database systems from DBMS, RDBMS to NoSQL solutions. Introduction to SQL query support of Azure DocumentDB and integrating DocumentDB with simple Java application from Maven repository.
Combining Data Mining and Machine Learning for Effective User ProfilingCodePolitan
Slide presentasi ini dibawakan oleh Anne Regina pada Seminar & Workshop Pengenalan & Potensi Big Data & Machine Learning yang diselenggarakan oleh KUDIO pada tanggal 14 Mei 2016.
Big Data Analysis Patterns with Hadoop, Mahout and Solrboorad
Big Data Analysis Patterns: Tying real world use cases to strategies for analysis using big data technologies and tools.
Big data is ushering in a new era for analytics with large scale data and relatively simple algorithms driving results rather than relying on complex models that use sample data. When you are ready to extract benefits from your data, how do you decide what approach, what algorithm, what tool to use? The answer is simpler than you think.
This session tackles big data analysis with a practical description of strategies for several classes of application types, identified concretely with use cases. Topics include new approaches to search and recommendation using scalable technologies such as Hadoop, Mahout, Storm, Solr, & Titan.
Big Data Analysis Patterns - TriHUG 6/27/2013boorad
Big Data Analysis Patterns: Tying real world use cases to strategies for analysis using big data technologies and tools.
Big data is ushering in a new era for analytics with large scale data and relatively simple algorithms driving results rather than relying on complex models that use sample data. When you are ready to extract benefits from your data, how do you decide what approach, what algorithm, what tool to use? The answer is simpler than you think.
This session tackles big data analysis with a practical description of strategies for several classes of application types, identified concretely with use cases. Topics include new approaches to search and recommendation using scalable technologies such as Hadoop, Mahout, Storm, Solr, & Titan.
A look back at how the practice of data science has evolved over the years, modern trends, and where it might be headed in the future. Starting from before anyone had the title "data scientist" on their resume, to the dawn of the cloud and big data, and the new tools and companies trying to push the state of the art forward. Finally, some wild speculation on where data science might be headed.
Presentation given to Seattle Data Science Meetup on Friday July 24th 2015.
Very basic Introduction to Big Data. Touches on what it is, characteristics, some examples of Big Data frameworks. Hadoop 2.0 example - Yarn, HDFS and Map-Reduce with Zookeeper.
Introductory Big Data presentation given during one of our Sizing Servers Lab user group meetings. The presentation is targeted towards an audience of about 20 SME employees. It also contains a short description of the work packages for our BIg Data project proposal that was submitted in March.
This presentation is prepared by one of our renowned tutor "Suraj"
If you are interested to learn more about Big Data, Hadoop, data Science then join our free Introduction class on 14 Jan at 11 AM GMT. To register your interest email us at info@uplatz.com
Big Data Analysis Patterns - TriHUG 6/27/2013boorad
Big Data Analysis Patterns: Tying real world use cases to strategies for analysis using big data technologies and tools.
Big data is ushering in a new era for analytics with large scale data and relatively simple algorithms driving results rather than relying on complex models that use sample data. When you are ready to extract benefits from your data, how do you decide what approach, what algorithm, what tool to use? The answer is simpler than you think.
This session tackles big data analysis with a practical description of strategies for several classes of application types, identified concretely with use cases. Topics include new approaches to search and recommendation using scalable technologies such as Hadoop, Mahout, Storm, Solr, & Titan.
A look back at how the practice of data science has evolved over the years, modern trends, and where it might be headed in the future. Starting from before anyone had the title "data scientist" on their resume, to the dawn of the cloud and big data, and the new tools and companies trying to push the state of the art forward. Finally, some wild speculation on where data science might be headed.
Presentation given to Seattle Data Science Meetup on Friday July 24th 2015.
Very basic Introduction to Big Data. Touches on what it is, characteristics, some examples of Big Data frameworks. Hadoop 2.0 example - Yarn, HDFS and Map-Reduce with Zookeeper.
Introductory Big Data presentation given during one of our Sizing Servers Lab user group meetings. The presentation is targeted towards an audience of about 20 SME employees. It also contains a short description of the work packages for our BIg Data project proposal that was submitted in March.
This presentation is prepared by one of our renowned tutor "Suraj"
If you are interested to learn more about Big Data, Hadoop, data Science then join our free Introduction class on 14 Jan at 11 AM GMT. To register your interest email us at info@uplatz.com
Learning “Data Analysis with R” @LearnSocial not only adds to your existing analytics knowledge but also equips you with what the industry is looking for.
Working with thousands, millions, or billions of data records in high dimensions is increasingly becoming the reality for scientific research. What are some techniques to make this kind of data volume tractable? How can parallel computing help? In this talk I'll review data management tools and infrastructures, languages, and paradigms that help in this regard. In particular, I'll discuss Hadoop, MapReduce, Python, NumPy, and Globus Online to provide a survey of ways in which researchers can manage their data and process it in parallel.
Scalable Learning Technologies for Big Data MiningGerard de Melo
These are slides of a tutorial by Gerard de Melo and Aparna Varde presented at the DASFAA 2015 conference.
As data expands into big data, enhanced or entirely novel data mining algorithms often become necessary. The real value of big data is often only exposed when we can adequately mine and learn from it. We provide an overview of new scalable techniques for knowledge discovery. Our focus is on the areas of cloud data mining and machine learning, semi-supervised processing, and deep learning. We also give practical advice for choosing among different methods and discuss open research problems and concerns.
A presentation I prepared for a workshop on inclusive spirituality, deepening your faith, and relating to people who have different traditions from yours. :)
Big data-analytics-changing-way-organizations-conducting-businessAmit Bhargava
Hi Friends ,
There is an interesting post on how to leveraging Big data analytics in an Integrated GRC Environment in an Organize to have visibility in core enterprises issues on real time basis . This presentation is from Metric stream -an international and Global GRC soloutioning providers in association with Dr. Kirk. D. Borne - Big data consultant and Adviser .Hope you like it and enjoy as well.
The AI Platform Business Revolution: Matchmaking, Empathetic Technology, and ...Steve Omohundro
Popular media is full of stories about self-driving cars, video deepfakes, and robot citizens. But this kind of popular artificial intelligence is having very little business impact. The actual impact of AI on business is in automating business processes and in creating the "AI Platform Business Revolution". Platform companies create value by facilitating exchanges between two or more groups. AI is central to these businesses for matchmaking between producers and consumers, organizing massive data flows, eliminating malicious content, providing empathetic personalization, and generating engagement through gamification. The platform structure creates moats which generate outsized sustainable profits. This is why platform businesses are now dominating the world economy. The top five companies by market cap, half of the unicorn startups, and most of the biggest IPOs and acquisitions are platforms. For example, the platform startup ByteDance is now worth $75 billion based on three simple AI technologies.
In this talk we survey the current state of AI and show how it will generate massive business value in coming years. A recent McKinsey study estimates that AI will likely create over 70 trillion dollars of value by 2030. Every business must carefully choose its AI strategy now in order to thrive over coming decades. We discuss the limitations of today's deep learning based systems and the "Software 2.0" infrastructure which has arisen to support it. We discuss the likely next steps in natural language, machine vision, machine learning, and robotic systems. We argue that the biggest impact will be created by systems which serve to engage, connect, and help individuals. There is an enormous opportunity to use this technology to create both social and business value.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
2. Introduction
◦ What is Big data?
◦ Why Big-Data?
◦ When Big-Data is really a problem?
Techniques
Tools
Applications
Literature
3.
4.
5. ‘Big-data’ is similar to ‘Small-data’, but bigger
…but having data bigger consequently requires
different approaches:
◦ techniques, tools, architectures
…with an aim to solve new problems
◦ …and old problems in a better way.
9. Key enablers for the growth of “Big Data” are:
◦ Increase of storage capacities
◦ Increase of processing power
◦ Availability of data
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20. Where processing is hosted?
◦ Distributed Servers / Cloud (e.g. Amazon EC2)
Where data is stored?
◦ Distributed Storage (e.g. Amazon S3)
What is the programming model?
◦ Distributed Processing (e.g. MapReduce)
How data is stored & indexed?
◦ High-performance schema-free databases (e.g.
MongoDB)
What operations are performed on data?
◦ Analytic / Semantic Processing (e.g. R, OWLIM)
21. Computing and storage are typically hosted
transparently on cloud infrastructures
◦ …providing scale, flexibility and high fail-safety
Distributed Servers
◦ Amazon-EC2, Google App Engine, Elastic,
Beanstalk, Heroku
Distributed Storage
◦ Amazon-S3, Hadoop Distributed File System
22. Distributed processing of Big-Data requires non-
standard programming models
◦ …beyond single machines or traditional parallel
programming models (like MPI)
◦ …the aim is to simplify complex programming tasks
The most popular programming model is
MapReduce approach
Implementations of MapReduce
◦ Hadoop (http://hadoop.apache.org/), Hive, Pig,
Cascading, Cascalog, mrjob, Caffeine, S4, MapR, Acunu,
Flume, Kafka, Azkaban, Oozie, Greenplum
23. The key idea of the MapReduce approach:
◦ A target problem needs to be parallelizable
◦ First, the problem gets split into a set of smaller problems (Map step)
◦ Next, smaller problems are solved in a parallel way
◦ Finally, a set of solutions to the smaller problems get synthesized
into a solution of the original problem (Reduce step)
24. NoSQL class of databases have in common:
◦ To support large amounts of data
◦ Have mostly non-SQL interface
◦ Operate on distributed infrastructures (e.g. Hadoop)
◦ Are based on key-value pairs (no predefined schema)
◦ …are flexible and fast
Implementations
◦ MongoDB, CouchDB, Cassandra, Redis, BigTable, Hbase,
Hypertable, Voldemort, Riak, ZooKeeper…
25.
26. …when the operations on data are complex:
◦ …e.g. simple counting is not a complex problem
◦ Modeling and reasoning with data of different kinds
can get extremely complex
Good news about big-data:
◦ Often, because of vast amount of data, modeling
techniques can get simpler (e.g. smart counting can
replace complex model-based analytics)…
◦ …as long as we deal with the scale
27. Research areas (such
as IR, KDD, ML, NLP,
Usage
SemWeb, …) are sub-
cubes within the data Quality
cube
Context
Streaming
Scalability
28. A risk with “Big-Data mining” is that an
analyst can “discover” patterns that are
meaningless
Statisticians call it Bonferroni’s principle:
◦ Roughly, if you look in more places for interesting
patterns than your amount of data will support, you
are bound to find crap
Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
29. Example:
We want to find (unrelated) people who at least twice
have stayed at the same hotel on the same day
◦ 109 people being tracked.
◦ 1000 days.
◦ Each person stays in a hotel 1% of the time (1 day out of 100)
◦ Hotels hold 100 people (so 105 hotels).
◦ If everyone behaves randomly (i.e., no terrorists) will the data
mining detect anything suspicious?
Expected number of “suspicious” pairs of people:
◦ 250,000
◦ … too many combinations to check – we need to have some
additional evidence to find “suspicious” pairs of people in
some more efficient way
Example taken from: Rajaraman, Ullman: Mining of Massive Datasets
30. Smart sampling of data
◦ …reducing the original data while not losing the
statistical properties of data
Finding similar items
◦ …efficient multidimensional indexing
Incremental updating of the models
◦ (vs. building models from scratch)
◦ …crucial for streaming data
Distributed linear algebra
◦ …dealing with large sparse matrices
31. On the top of the previous ops we perform
usual data mining/machine learning/statistics
operators:
◦ Supervised learning (classification, regression, …)
◦ Non-supervised learning (clustering, different types
of decompositions, …)
◦ …
…we are just more careful which algorithms
we choose (typically linear or sub-linear
versions)
32. An excellent overview of the algorithms
covering the above issues is the book
“Rajaraman, Ullman: Mining of Massive
Datasets”
33.
34. Good recommendations
can make a big
difference when keeping
a user on a web site
◦ …the key is how rich the
context model a system is
using to select information
for a user
◦ Bad recommendations <1%
users, good ones >5% users
click
◦ 200clicks/sec
Contextual
personalized
recommendations
generated in ~20ms
35. Domain Referring Domain Zip Code
Sub-domain Referring URL State
Page URL Outgoing URL Income
URL sub-directories Age
GeoIP Country Gender
Page Meta Tags GeoIP State Country
Page Title GeoIP City Job Title
Page Content Job Industry
Named Entities Absolute Date
Day of the Week
Has Query Day period
Referrer Query Hour of the day
User Agent
36. Trend Detection System
User Stream of
Log Files Stream
of clicks profiles profiles
(~100M
page clicks
per day)
Sales
Trends and
updated segments Segments
Segment Keywords
NYT Stock Stock Market, mortgage, banking,
Market investors, Wall Street, turmoil, New
articles York Stock Exchange
Campaign
Health diabetes, heart disease, disease, heart,
illness to sell
segments
$
Green Hybrid cars, energy, power, model,
Energy carbonated, fuel, bulbs,
Hybrid cars Hybrid cars, vehicles, model, engines,
diesel
Travel travel, wine, opening, tickets, hotel,
sites, cars, search, restaurant
Advertisers
… …
37. 50Gb of uncompressed log files
50-100M clicks
4-6M unique users
7000 unique pages with more then 100 hits
38. Alarms Server
Telecom
Network Alarms
Alarms Live feed of data Explorer
(~25 000 ~10-100/sec
devices) Server
Alarms Explorer Server implements three
real-time scenarios on the alarms stream:
1. Root-Cause-Analysis – finding which device is
responsible for occasional “flood” of alarms
2. Short-Term Fault Prediction – predict which
device will fail in next 15mins
3. Long-Term Anomaly Detection – detect
unusual trends in the network
…system is used in British Telecom
Operator Big board display
39. The aim of the project is to collect and analyze
most of the main-stream media across the world
◦ …from 35,000 publishers (180K RSS feeds) crawled in
real time (few ~10articles per second)
◦ …each article document gets extracted, cleaned,
semantically annotated, structure extracted
Challenges are in terms of complexity of
processing and querying of the extracted data
◦ …and matching textual information across the
languages (cross-lingual technologies)
http://render-project.eu/ http://www.xlike.org/
41. Extracted
graph
of triples
Plain text from text
Text
Enrichment
“Enrycher” is available as
as a web-service generating
Semantic Graph, LOD links,
Entities, Keywords, Categories,
Text Summarization
42. The aim is to use analytic techniques to
visualize documents in different ways:
◦ Topic view
◦ Social view
◦ Temporal view
50. Observe social and communication
phenomena at a planetary scale
Largest social network analyzed till 2010
Research questions:
How does communication change with user
demographics (age, sex, language, country)?
How does geography affect communication?
What is the structure of the communication
network?
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
50
51. We collected the data for June 2006
Log size:
150Gb/day (compressed)
Total: 1 month of communication data:
4.5Tb of compressed data
Activity over June 2006 (30 days)
◦ 245 million users logged in
◦ 180 million users engaged in conversations
◦ 17,5 million new accounts activated
◦ More than 30 billion conversations
◦ More than 255 billion exchanged messages
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
51
54. Count the number of users logging in from
particular location on the earth
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008
54
55. Logins from Europe
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 55
56. Hops Nodes
1 10
2 78
3 396
4 8648
5 3299252
6 28395849
7 79059497
8 52995778
9 10321008
10 1955007
11 518410
12 149945
13 44616
14 13740
15 4476
16 1542
17 536
18 167
19 71
6 degrees of separation [Milgram ’60s] 20 29
Average distance between two random users is 6.622
21 16
10
90% of nodes can be reached in < 8 hops 23 3
24 2
“Planetary-Scale Views on a Large Instant-Messaging Network” Leskovec & Horvitz WWW2008 25 3
57.
58. Big-Data is everywhere, we are just not used to
deal with it
The “Big-Data” hype is very recent
◦ …growth seems to be going up
◦ …evident lack of experts to build Big-Data apps
Can we do “Big-Data” without big investment?
◦ …yes – many open source tools, computing machinery is
cheap (to buy or to rent)
◦ …the key is knowledge on how to deal with data
◦ …data is either free (e.g. Wikipedia) or to buy (e.g.
twitter)