ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

•

1 like•644 views

Gigaom

Presentation from Aaron Kimball, WibiData #dataconf More at http://event.gigaom.com/structuredata/

Technology

ANALYZING LARGE-SCALE USER DATA

SPEAKER: Aaron Kimball
CTO
WibiData

Friday, July 27, 2012

Analyzing
Large-‐Scale
User
Data
with
Hadoop
and
HBase

Aaron
Kimball
–
CTO

WibiData,

Friday, July 27, 2012

We
can
now
collect

more
data
than
at

any
Dme
in
history.

Friday, July 27, 2012

Yesterday’s
engineering

challenge:
FiJng
the

problem
into
the

hardware.
Friday, July 27, 2012

Today’s
constrained

resource
is

understanding.

Friday, July 27, 2012

How
do
we
best
apply

data

…to
beMer
serving
our

Friday, July 27, 2012

The
best
products
are
user-‐
• IntuiDve
UI
• ConDnuously
learning
– Guided
search
– Smarter
recommenda1ons
• More
eﬀec1ve
service

Friday, July 27, 2012

What
are
we
building

Friday, July 27, 2012

Requirements

1.
Understand
the
user

populaDon
Friday, July 27, 2012

Requirements

2.
Respond
to

users
in
real

Dme

Friday, July 27, 2012

Requirements

3.
Support
graceful
data

evoluDon
Friday, July 27, 2012

Large-‐scale
data
science
is

• What
does
a
user
look
like?
– What
data
is
available
about
the
user?
– Which
features
are
important?
– Which
features
are
correlated?
• How
do
I
model
this
in
MapReduce?
• How
do
I
serve
results
in
a
Dmely

Friday, July 27, 2012

Tools
of
the
trade
• Store
all
data
about
a

user
in
one
place
• Support
real-‐Dme

get/put,
as
well
as

MapReduce

Friday, July 27, 2012

Tools
of
the
trade
• Use
complex
data

types
to
model

complex
data
• Support
extended

data
models
over

Dme

Friday, July 27, 2012

Tools
of
the
trade
• Abstract
computaDonal

model
away
from

MapReduce
• Support
computaDon

over
all
users…
or
one

user
at
a
Dme

Friday, July 27, 2012

:
for
set-‐top
boxes

Viewing/recording

history

Friday, July 27, 2012

:
for
set-‐top
boxes

Libraries
Device
and
User
Analysis

Viewing/recording

history

Personalized
oﬀers

and

recommenda=ons

Friday, July 27, 2012

:
for
set-‐top
boxes

Libraries
Device
and
User
Analysis

Viewing/recording

history

Personalized
oﬀers

and

recommenda=ons

Analysis
for

product

roadmap
Friday, July 27, 2012

:
for
set-‐top
boxes

Libraries
Device
and
User
Analysis

Viewing/recording

history

Personalized
oﬀers

and

recommenda=ons

Analysis
for

product
Tech
support

roadmap portal
Friday, July 27, 2012

:
for
set-‐top
boxes

Libraries
Device
and
User
Analysis

Viewing/recording

history

Personalized
oﬀers

and

recommenda=ons

Improve
Analysis
for

d
reports

product
Tech
support

for

roadmap portal
Friday, July 27, 2012 adver=se

The
future
• More
personalizaDon
• AdapDve
UIs
(self
arranging

dashboards)
• Targeted
content,
ads
• More
eﬀecDve
customer
service

Friday, July 27, 2012

Conclusions
• ApplicaDons
are
becoming

increasingly

user-‐centric
• Data
drives
this
capability,
but

harnessing
it
requires
a
new

distributed
architecture

Friday, July 27, 2012

www.wibidata.com
/

Aaron
Kimball
–
aaron@wibidata.com

Friday, July 27, 2012

Link to video used in exercise : http://www.youtube.com/watch?v=AytfuE-wqbA Link to document with resource list: http://www.slideshare.net/northavorange/enhancing-at-through-id-techniques-handouts Rehabilitation professionals classify needs and identify workable solutions for people with disabilities on a daily basis. Unfortunately, many of those solutions never get beyond the one person for whom they are made. The ability to develop solutions that have a more universal appeal and application would be a useful tool in the AT provider’s “tool belt.” Industrial Designers face such challenges as a matter of practice. This workshop will educate participants with regard to tools and techniques used by Industrial Designers that can help the “one-of-a-kind” solutions grow into a more universally marketable solution.

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

Gigaom

Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)Rick. Bahague

Enhancing AT through ID techniques handouts

northavorange

Link to slide cast of presentation: http://www.slideshare.net/northavorange/enhancing-at-through-id-techniques Rehabilitation professionals classify needs and identify workable solutions for people with disabilities on a daily basis. Unfortunately, many of those solutions never get beyond the one person for whom they are made. The ability to develop solutions that have a more universal appeal and application would be a useful tool in the AT provider’s “tool belt.” Industrial Designers face such challenges as a matter of practice. This workshop will educate participants with regard to tools and techniques used by Industrial Designers that can help the “one-of-a-kind” solutions grow into a more universally marketable solution.

Final Year Project Guidance

Varad Meru

My fire st petersburg 27 june 2012 (d hladky)AI4BD GmbH

Guerrilla Usability Testing for Agile/Lean

Interactive Agency

The state of drupal 8 - Drupalcamp Gent

swentel

Choosing a backend for your mobile app? Don’t roll the dice!

Codemotion

Although usually not noticed by the final users, the backend is an important piece in the context of a mobile application. In the past building a back-end meant renting some hardware or virtual machine, setting up the operative system, configuring the server, database, storage and writing the back-end logic. Nowadays great services that allow to quickly setup a back-end are spreading. I will provide an overview of the services which allow developers to integrate their mobile applications with back-end functionalities and I will illustrate the features of BaasBox, an open source back end.

Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...Vito Ostuni

Structure 2014 - The strategic value of the cloud - Joe Weinman

Gigaom

Structure 2014 - The right and wrong way to scale - Rackspace

Gigaom

Similar to ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

An Analytics Toolkit Tour

Rory Winston

THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012

Gigaom

SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:

Gigaom

BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...

Gigaom

Lecture 4: Social Web Personalization (2012)

Lora Aroyo

Best Practices - Seeqnce - 23/24-02-2012Youssef Chaker

REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012

Gigaom

Alabfi em-20120624zepheiraorg

E assessment - ZAWF at Aberdeen WorkshopDigit Class

eXo Software Factory Overview

Arnaud Héritier

Enhancing AT through ID Techniques

northavorange

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

Gigaom

Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)Rick. Bahague

Enhancing AT through ID techniques handouts

northavorange

Final Year Project Guidance

Varad Meru

My fire st petersburg 27 june 2012 (d hladky)AI4BD GmbH

Guerrilla Usability Testing for Agile/Lean

Interactive Agency

The state of drupal 8 - Drupalcamp Gent

swentel

Choosing a backend for your mobile app? Don’t roll the dice!

Codemotion

Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...Vito Ostuni

Similar to ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012 (20)

An Analytics Toolkit Tour

THE TRILLION ROW SPREADSHEET(tm) from Structure:Data 2012

SPONSORED WORKSHOP by Amplidata from Structure:Data 2012:

BIG DATA: AN AUGMENTED INTELLIGENCE FOR STRATEGIC DECISION MAKING from Struct...

Lecture 4: Social Web Personalization (2012)

Best Practices - Seeqnce - 23/24-02-2012

REALIZING REAL-TIME VALUE ON THE REAL-TIME WEB from Structure:Data 2012

Alabfi em-20120624

E assessment - ZAWF at Aberdeen Workshop

eXo Software Factory Overview

Enhancing AT through ID Techniques

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

Drupal campmanila 2012 (Responsive Web in Drupal with Omega Theme)

Enhancing AT through ID techniques handouts

Final Year Project Guidance

My fire st petersburg 27 june 2012 (d hladky)

Guerrilla Usability Testing for Agile/Lean

The state of drupal 8 - Drupalcamp Gent

Choosing a backend for your mobile app? Don’t roll the dice!

Cinemappy: a Context-aware Mobile App for Movie Recommendations boosted by DB...

Recently uploaded

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Assure Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Pushing the limits of ePRTC: 100ns holdover for 100 days

Adtran

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

UiPathCommunity

In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni. 📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath: Autopilot per Studio Web Autopilot per Studio Autopilot per Apps Clipboard AI GenAI applicata alla Document Understanding 👨‍🏫👨‍💻 Speakers: Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath Andrei Tasca, RPA Solutions Team Lead @NTT Data

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

Free Complete Python - A step towards Data Science

RinaMondal9

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Recently uploaded (20)

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

Leading Change strategies and insights for effective change management pdf 1.pdf

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Assure Contact Center Experiences for Your Customers With ThousandEyes

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

PCI PIN Basics Webinar from the Controlcase Team

Elevating Tactical DDD Patterns Through Object Calisthenics

Video Streaming: Then, Now, and in the Future

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

FIDO Alliance Osaka Seminar: Overview.pdf

Climate Impact of Software Testing at Nordic Testing Days

Pushing the limits of ePRTC: 100ns holdover for 100 days

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Securing your Kubernetes cluster_ a step-by-step guide to success !

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Free Complete Python - A step towards Data Science

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

DevOps and Testing slides at DASA Connect

ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

1. ANALYZING LARGE-SCALE USER DATA SPEAKER: Aaron Kimball CTO WibiData Friday, July 27, 2012

2. Friday, July 27, 2012

3. Analyzing Large-‐Scale User Data with Hadoop and HBase Aaron Kimball – CTO WibiData, Friday, July 27, 2012

4. We can now collect more data than at any Dme in history. Friday, July 27, 2012

5. Yesterday’s engineering challenge: FiJng the problem into the hardware. Friday, July 27, 2012

6. Today’s constrained resource is understanding. Friday, July 27, 2012

7. How do we best apply data …to beMer serving our Friday, July 27, 2012

8. The best products are user-‐ • IntuiDve UI • ConDnuously learning – Guided search – Smarter recommenda1ons • More eﬀec1ve service Friday, July 27, 2012

9. What are we building Friday, July 27, 2012

10. What are we building Friday, July 27, 2012

11. What are we building Friday, July 27, 2012

12. What are we building Friday, July 27, 2012

13. What are we building Friday, July 27, 2012

14. What are we building Friday, July 27, 2012

15. What are we building Friday, July 27, 2012

16. What are we building Friday, July 27, 2012

17. What are we building Friday, July 27, 2012

18. What are we building Friday, July 27, 2012

19. Requirements 1. Understand the user populaDon Friday, July 27, 2012

20. Requirements 2. Respond to users in real Dme Friday, July 27, 2012

21. Requirements 3. Support graceful data evoluDon Friday, July 27, 2012

22. Large-‐scale data science is • What does a user look like? – What data is available about the user? – Which features are important? – Which features are correlated? • How do I model this in MapReduce? • How do I serve results in a Dmely Friday, July 27, 2012

23. Friday, July 27, 2012

24. Tools of the trade • Store all data about a user in one place • Support real-‐Dme get/put, as well as MapReduce Friday, July 27, 2012

25. Tools of the trade • Use complex data types to model complex data • Support extended data models over Dme Friday, July 27, 2012

26. Tools of the trade • Abstract computaDonal model away from MapReduce • Support computaDon over all users… or one user at a Dme Friday, July 27, 2012

27. Friday, July 27, 2012

28. Friday, July 27, 2012

29. Friday, July 27, 2012

30. Friday, July 27, 2012

31. Friday, July 27, 2012

32. Friday, July 27, 2012

33. Friday, July 27, 2012

34. : for set-‐top boxes Viewing/recording history Friday, July 27, 2012

35. : for set-‐top boxes Viewing/recording history Friday, July 27, 2012

36. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Friday, July 27, 2012

37. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Friday, July 27, 2012

38. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Analysis for product roadmap Friday, July 27, 2012

39. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Analysis for product roadmap Friday, July 27, 2012

40. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Analysis for product Tech support roadmap portal Friday, July 27, 2012

41. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Analysis for product Tech support roadmap portal Friday, July 27, 2012

42. : for set-‐top boxes Libraries Device and User Analysis Viewing/recording history Personalized oﬀers and recommenda=ons Improve Analysis for d reports product Tech support for roadmap portal Friday, July 27, 2012 adver=se

43. The future • More personalizaDon • AdapDve UIs (self arranging dashboards) • Targeted content, ads • More eﬀecDve customer service Friday, July 27, 2012

44. Conclusions • ApplicaDons are becoming increasingly user-‐centric • Data drives this capability, but harnessing it requires a new distributed architecture Friday, July 27, 2012

45. www.wibidata.com / Aaron Kimball – aaron@wibidata.com Friday, July 27, 2012

46. Friday, July 27, 2012

ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

Recommended

Recommended

More Related Content

Similar to ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012

Similar to ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012 (20)

More from Gigaom

More from Gigaom (20)

Recently uploaded

Recently uploaded (20)

ANALYZING LARGE-SCALE USER DATA from Structure:Data 2012