A Yarn About Twine -- ISWC 2009 Keynote -- Nova Spivack

Semantic Search 3.0
A Yarn about Twine
ISWC 2009 Keynote
Nova Spivack
Founder, CEO Twine.com
nova@twine.com

In 1998 my first Internet companyEarthWebMakers of Developer.com & Dice.comhad a ginormous IPO w00t!!!
2
1.11.2008 | Product Positioning

At 29 years old, I was set for life**On paper….
3

But instead of retiring Youth and idealism won out
4

I wanted to build something REALLY big…A Web of knowledge
5

So I self-funded my own mini R&D lab, Lucid Ventures
6

Kris Thorisson (PhD) Jim Wissner (Java whiz)And a bunch of others…
7

Lucid grew to a few PhD’s And a bunch of engineers We did lots of exploring. In Java.
8

We eventually built Personal Radar
9
P2P Semantic Desktop for knowledge sharing
Cross-platform, Pure Java
Fully semantic, RDF and OWL based
Homegrown ultra-fast triplestore

We eventually spun the project out as its own company,Radar Networks
10

11

12

13

14

15

Screenshots
16

17

18

19

20

21

Oh yeah,And I also angel invested in a bunch of startupsAnd bought up more EarthWeb stock. w00t!
22

Then the Bubble burst…* *WTF???
23

Nuclear winter for IT and Web venturesBegan…
24

DON’T TRY THIS AT HOME
25

Despite the Economy, We Continued to Press ForwardWith our product plan
26

But there was no venture fundingAvailable anywhere**VC’s are scaredy cats
27

Luckily,a few days before we ran out of money…
28

SRI selected us to join the DARPA CALO project
29

Personal Radar became part of OpenIRIS* See: OpenIRIS.org *Cool project
30

2004 - 2005We worked on DARPA CALO & IRIS2006Series A venture roundFinished our work on CALO & IRISStarted to work on our own products
31

Our GoalWeb-Scale Knowledge Networking For Consumers
32

ObstacleThere was no platform that met our needs at the time. Not even our own.So from 2006 – 2007 we built one**Insane
33

To accomplish this,using RDF and OWLOn a triplestorewe really had to push the envelope
34

We built a big federated tuplestoreSupport up to 100 billion tuples Full ACL permissions on the data Capable of serving millions of consumers
35

In 2008 the platform was ready We started building apps on it
36

October 2008 Twine went Beta
37

A Slightly Ambitious Project*Semantic WebNatural language processingEntity detectionGraph theoryClusteringRecommendationsSemantic filteringCollaborative authoringAuto-taggingSummarizationVisualizationPersonalization*What the (bleep) were we thinking?
38

March 2009 We raised a $14mm Series B round A total of $24mm raised so far…**Ok, that’s what we were thinking…
39

Twine grew faster than expectedConsumers actually were using it**Hey, this #$%@ works
40

Within 6 months of launch…Up to 2.2mm unique visitors monthly 4mm pieces of content added 25K interest groups 100’s of articles about it Twine grew faster than Twitter**Our VCs were happy
41

But Twine was not finished A lot more work was needed And we were gearing up Hiring, designing, coding, spending…
42

We grew to over 30 people.Mostly engineers. But we really needed more like 70.
43

But we were optimistic and focused onDesigning the next version of Twine
44

Then the economy tankedAgain…..
45

Silicon Valley became a lot less happyAll venture funding dried up
46

So, like many other venturesWe had to cut back drasticallyon spending, hiring, and our product goals
47

Lessons LearnedTwine 1.0 tried to solve too many problemsToo many featuresToo hard to figure outOur data revealed that Twine users mainly just want to search and track interests
48

Lessons LearnedWe should not have relied on a tuplestore so centrally in the Twine 1.0 architectureWe haven’t yet seen a tuplestore that fully meets the needs of a major consumer online web site like Twine
49

The Web 2.0 party was over.Where to go from here?… A Radical Rethink
50

Continuing on the path we were onwould be too expensive to scaleMust change the architecture, or change the product,or both**We chose “or both”
51

The Future of Search
Confidential

Where Search is Headed
Semantic
Personalized
Real-time
Social
Sharing
Tracking
KM
Reasoning
53

But the Semantic Web Hasn’tHappened Yet…
At least not for consumers
And most Web developers…
54

Why hasn’t the Semantic Web taken off yet?
55

How can we demonstrate the value of the Semantic Web to consumers?And make it easily useful to Web developers?In the near-term?
56

Let’s Get Real
Consumer are not going to add semantic metadata
Webmasters are too lazy, or don’t know how
Humans are bad at generating good metadata anyway
Automated metadata generation is the only practical solution
57

Index the Semantics of the Web
Create the Semantic Web
From the non-Semantic Web
Automatically
Provide it back as a Web service
58

Provide faceted semantic search and navigation against the indexAt Web-scale
59

Kind of a Semantic Google
IBM.com
Web Site
Joe
Person
IBM
Company
Palo Alto
City
Lives in
Publisher of
Fan of
Lives in
Subscriber to
Employee of
Sue
Person
Jane
Person
Dave.com
RSS Feed
Coldplay
Band
Fan of
Friend of
Member of
Design
Team
Group
Depiction of
Married to
Source of
123.JPG
Photo
Member
of
Bob
Person
Dave.com
Weblog
Depiction of
Member of
Dave
Person
Stanford
Alumnae
Group
Member of
Author of
Member of
Person
Application

But not on anything close to Google’s budget
61

What were we smoking?**We were not Freebasing
62

Well it turns out, it’s not impossible…
63

T2
64

The T2 Approach…
65
Focus on search Keep it simple Broad appeal Scale cheaply
Connect with ecosystem

T2 ArchitectureHigh-performanceOpen-ended scalingMore semantic (RDF, OWL)Not built directly on a triplestoreNot built directly on a databaseHeavy use of SOLR (from Lucene)
66

T2 Stack
67
Twine Properties
Partners/3rd Parties
WebApp Framework
APIs
Application Services
Resource Service
Web Corpus
(e.g., BOSS, etc)
Semantic Metadata Service
Tuple Service
SOLR Farm
RDBMS

T1 Approach
Human
Tagging
AI
Semantic
Web
Make the Data Smarter
Machine
Tagging
Linguistics
Extraction
Statistics
Make the software smarter

T2 Approach
Semantic
Web
Linguistics
Extraction
Statistics

A Tactical Shift
Freebase
DBpedia
Twine
T1
EVRI
Wolfram
Alpha
Wikipedia
Open Calais
Endeca
Expert System
Flickr
Bing
Siri
FAST
Powerset
Yahoo
Delicious
Autonomy
Inxight
Hakia
Google

To a More Mainstream App
Freebase
Twine
T2
DBpedia
Twine
T1
EVRI
Wolfram
Alpha
Wikipedia
Open Calais
Endeca
Expert System
Flickr
Bing
Siri
FAST
Powerset
Yahoo
Delicious
Autonomy
Inxight
Hakia
Google

Web-Scale Semantic ExtractionAnd search
72

T2 Semantic Index
Videos
Music
People
Reviews
Games
News
Products
Recipes
Services
How-To’s
Classifieds
Events
Hotels
Bars
Resumes
Coupons
Reports
Photos
Help
73

Many Uses for this Index
Search
Interest tracking
Recommendations
Web application development
74

1. Do a Keyword Search
2. Filter by Type
3. Filter by
Attribute
Recipe
Cuisine
Web
Page
Classified Ad
Difficulty
Web
Page
Product
Ingredient
Prep Time
News
Web Page
Author
Web
Page
Hotel
Type of Dish
Web
Page
Song
Dietary Option
Website
Game
Web Page
Prep Time
Help Article
Web Page
Calories
Web Page
Etc…
Person
Etc…
75

Use Case: Cooking
76

Scenario: Family Dinner
Deciding what to cook…

Scenario: Family Dinner

Current Scenario: Family Dinner
[INSERT BING RESULTS + EXTERNAL LOGOS]

Current Scenario: Family Dinner

T2 – Tools for Developers
API to T2 Index and services
Widgets and components for 3rd party sites
Web-based ontology community – like Sourceforge
Browser plugin for easy site mapping to ontologies
85

T2 Ontology Site
Demo
86

T2 Site Mapping Tool
Confidential

T2 Business Model
T2 destination
Advertising on the site
T2 site search & ad network
Provide sites with better site search
Targeted ads
Revshare
T2 API
Partners pay us, or revshare with us
Monetize however they want to
96

T2 Near-Term Vertical Focus Areas
Lifestyle
Entertainment
Shopping
Food (Complete)
Health
Travel
People
Gaming (In Dev)
Sports
Music
TV
Film
Consumer Electronics
Health & Beauty
Classified Ads
Media Products
Help & Support

T2 Long-Term Goals
Index the structured data on the Web
Across all major vertical domains
With help from an army of outside developers and partners
Grow an ecosystem around the index
98

Timeline T1 is in Maintenance Mode T2 is in Alpha T2 Goes Beta by Q1
99

What Next?Find Follow Share
100

For all types of content Web pages Data records Real-time content
101

Find stuff (semantic search)Then follow it (interest tracking) Then share it (socialization)
102

But this time…
103

We’re starting with just “Find”
104

Conclusions Web-Scale Semantic Search is the Next-Step for Search and for Twine But after 9 years of this, it’s clear It won’t happen overnight
105

Stay Tuned…
106

Contact Information
Nova Spivack
Twitter: @novaspivack
nova@twine.com

http://www.novaspivack.com	504
http://www.slideshare.net	38
http://bagtheweb.com	31
http://www.diigo.com	1
http://translate.googleusercontent.com	1

A Yarn About Twine -- ISWC 2009 Keynote -- Nova Spivack

by Nova Spivack on Dec 21, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 575

Statistics

A Yarn About Twine -- ISWC 2009 Keynote -- Nova Spivack — Presentation Transcript