DIADEM data extraction methodology

DIADEM data extraction methodology
domain-centric intelligent automated
DIADEM
Domains to Databases
Georg Gottlob and Tim Furche (Vienna University of Technology and Oxford University)
July 2013 @ STI Summit
joint work with Giovanni Grasso, Omer Gunes, Xiaonan Guo, Andrey Kravchenko, Thomas Lukasiewicz,
Giorgio Orsi, Andreas Pieris, Christian Schallhart, Andrew Sellers, Gerardo Simari, Cheng Wang

About us …
DIADEM lab at Oxford University
2
2010 2011 2012 2013 2014 2015
DIADEM

DIADEM ›❯ The State of Search
Search engines don’t cut it any more …
6
20121995 2000 2004 2008
Jahr
Webpages
Search Results
Overall Content

DIADEM ›❯ The State of Search
Search engines don’t cut it any more …
6
20121995 2000 2004 2008
Jahr
Webpages
Search Results
Overall Content
What humans can process

DIADEM ›❯ The State of the Game
7
Advanced searchAbout 48,700,000 results (0.19 seconds)
Oxford Flats - Find Flats to Suit all Budgets | FindaProperty.com
Updated Daily. Register for Alerts.
Houses For Sale In Oxfordshire - Houses To Rent In Oxfordshire
www.findaproperty.com/flats
Flat In Oxford | TaylorWimpey.co.uk
New Flats & Houses in Oxford. Starting from £157,995.
www.taylorwimpey.co.uk/Oxford
Flat In Oxford | Primelocation.com
Search over 650,000 Luxury UK Flats from the Comfort of your Armchair!
www.primelocation.com/flats
Property to rent in Oxford, Oxfordshire
Results 1 - 20 of 582 – Review houses, flats and homes to rent in Oxford or try the ...
• Parking Space to rent in Oxford – £120 pcm – unfurnished – 0.32 miles
• Garage to rent in Oxford – £150 pcm – unfurnished – 2 additional photos
• House Share to rent in Oxford – £315 pcm – Per Person furnished – 3 additional ...
www.findaproperty.com/searchresults.aspx?edid=00...1... - Cached - Similar
Flats, flatshare rentals, Oxford - find a flatshare online
Find a e.g. BMW, 2 bed flat, sofa; in e.g. Portslade ... 1388 ads in Oxford, Flatshare, Rooms
to Rent Subscribe to email alerts ... East OxfordDate wanted: 20 Sep ...
Wanted - Flatshare in Oxford offered - Short Term
www.gumtree.com/flatshare/oxford - Cached
Flats / Houses to Rent, Oxford : Rent a house online
677 ads in Oxford, Flats & Houses for Rent Subscribe to email alerts ...
www.gumtree.com/flats-and-houses-for-rent/oxford - Cached
Ads
Homes in Oxford
A Barratt Home in Oxford
It May Be Cheaper than Renting
www.barratthomes.co.uk/Oxford
Flat/House Rentals Oxford
Browse our list of flats & houses
to rent in Oxford. Available now.
www.letting4oxford.co.uk
Houses & Flats in Oxford
Flats for sale in Oxford
by leading local estate agent.
www.johndwood.co.uk/Oxford
Oxford Luxury Short Lets
Serviced accommodation
Centrally located with parking
www.oxfordapartment.co.uk
Flats in Oxford
Oxford flats for all budgets with
award winning service. View Tod
www.propertywide.co.uk/Oxford
Oxford Accommodation
Great deals On Unsold Accommo
Across Oxford. Up To 50% Off!
laterooms.com is rated
flat in oxford Search
Ads
News Shopping Gmail moreObject Search Today @ Google

7
Ads
Homes in Oxford
A Barratt Home in Oxford
It May Be Cheaper than Renting
www.barratthomes.co.uk/Oxford
Flat/House Rentals Oxford
Browse our list of flats & houses
to rent in Oxford. Available now.
www.letting4oxford.co.uk
Houses & Flats in Oxford
Flats for sale in Oxford
by leading local estate agent.
www.johndwood.co.uk/Oxford
Oxford Luxury Short Lets
Serviced accommodation
Centrally located with parking
www.oxfordapartment.co.uk
Flats in Oxford
Oxford flats for all budgets with
award winning service. View Tod
www.propertywide.co.uk/Oxford
Oxford Accommodation
Great deals On Unsold Accommo
Across Oxford. Up To 50% Off!
laterooms.com is rated
Ads
News Shopping Gmail moreObject Search Today @ Google
doesn’t understand entity type
favors “big” aggregators & news sites
with poor quality results

8New Flats & Houses in Oxford. Starting from £157,995.
Show more results from gumtree.com
Flats For Sale In Oxford, Oxfordshire | Primelocation
Results 1 - 10 of 290 – A; Asking price of £960000; flat; 4 bedrooms. The Lion Brewery, St.

Section 1:
9
Ads
Homes in
A Barratt Ho
It May Be Ch
www.barratth
Flat/Hous
Browse our l
to rent in Ox
www.letting4
Houses &
Flats for sale
by leading lo
www.johndw
Oxford Lu
Serviced acc
Centrally loc
www.oxford
Flats in O
Oxford flats
award winnin
www.propert
Ads
News Shopping Gmail more
Object Search Today @ Google

10
[PDF]
[PDF]
OXFORD IS MY WORLD | Energy Home Energy Use
Oxford is my world Your – Guide to saving the planet! ... who wants to improve the energy
efficiency of their house or save energy at home there is ... Our 'Very Easy' steps show you
how much energy you can save … without spending a penny! ...
www.oxfordismyworld.org/home_energy.html - Cached - Similar
Escalator - Wikipedia, the free encyclopedia
Escalator step widths and energy usage ..... This device actually consisted of flat, moving
stairs, not unlike the escalators of .... the increased efficiency of each operator due to the
elimination of stair climbing. ..... ²" The Oxford English Dictionary. ...
en.wikipedia.org/wiki/Escalator - Cached - Similar
THE EFFECTIVENESS OF FEEDBACK ON ENERGY CONSUMPTION
File Format: PDF/Adobe Acrobat - Quick View
by S Darby - 2006 - Cited by 148 - Related articles
The focus is on how people change their behaviour, not on the .... recognition that energy
efficiency alone is inadequate to achieve the aims of a ...... House. Environmental Change
Institute, University of Oxford, UK. Brandon G & Lewis A ...
www.eci.ox.ac.uk/research/energy/.../smart-metering-report.pdf - Similar
The Oxford Solar House - TVE
The Oxford Solar House is the first low energy house in the United Kingdom ... reduced by
using all available energy saving technologies but without impairing ... service duct, stairs to
the first floor and a hallway to the entry porch. ...
www.tve.org/ho/series1/reports_7-12/reports.../theoxfordsolarhouse.pdf
Gordon & Erika Wilson - Pre-fabricated energy-saving homes from ...
Saving energy and the environment ... We went and knocked on the door of the neighbouring
house there and then and asked if ... Not least so by the energy efficiency. ... To the right is
a hallway leading to the stairs, and beyond to the study. .... +++ Planning permission granted
for new build in Oxford +++ VIEW NEW videos ...
www.hanse-haus.co.uk/our_projects/.../gordon_erika_wilson.html - Cached
Ads
Oxford Flats
Find Flats to Suit all Budgets.
See your ad here »
flat in oxford, energy efficient, no stairs Search
News Shopping Gmail more Sign in

10
[PDF]
[PDF]
OXFORD IS MY WORLD | Energy Home Energy Use
Oxford is my world Your – Guide to saving the planet! ... who wants to improve the energy
efficiency of their house or save energy at home there is ... Our 'Very Easy' steps show you
how much energy you can save … without spending a penny! ...
www.oxfordismyworld.org/home_energy.html - Cached - Similar
Escalator - Wikipedia, the free encyclopedia
Escalator step widths and energy usage ..... This device actually consisted of flat, moving
stairs, not unlike the escalators of .... the increased efficiency of each operator due to the
elimination of stair climbing. ..... ²" The Oxford English Dictionary. ...
en.wikipedia.org/wiki/Escalator - Cached - Similar
THE EFFECTIVENESS OF FEEDBACK ON ENERGY CONSUMPTION
by S Darby - 2006 - Cited by 148 - Related articles
The focus is on how people change their behaviour, not on the .... recognition that energy
efficiency alone is inadequate to achieve the aims of a ...... House. Environmental Change
Institute, University of Oxford, UK. Brandon G & Lewis A ...
www.eci.ox.ac.uk/research/energy/.../smart-metering-report.pdf - Similar
The Oxford Solar House - TVE
The Oxford Solar House is the first low energy house in the United Kingdom ... reduced by
using all available energy saving technologies but without impairing ... service duct, stairs to
the first floor and a hallway to the entry porch. ...
www.tve.org/ho/series1/reports_7-12/reports.../theoxfordsolarhouse.pdf
Gordon & Erika Wilson - Pre-fabricated energy-saving homes from ...
Saving energy and the environment ... We went and knocked on the door of the neighbouring
house there and then and asked if ... Not least so by the energy efficiency. ... To the right is
a hallway leading to the stairs, and beyond to the study. .... +++ Planning permission granted
for new build in Oxford +++ VIEW NEW videos ...
www.hanse-haus.co.uk/our_projects/.../gordon_erika_wilson.html - Cached
Ads
Oxford Flats
Find Flats to Suit all Budgets.
See your ad here »
flat in oxford, energy efficient, no stairs Search
News Shopping Gmail more Sign in
gets worse the more I know
doesn’t understand primary object
lacks “attributes”

11
Microsoft Bing:
“Model Every Object on the Planet”

11
Microsoft Bing:
Google:
“Knowledge Graph:
things, not strings”

11
Microsoft Bing:
Google:
“Knowledge Graph:
things, not strings”
common sense, static facts
wikipedia-like
requires high degree of redundancy
same information on many sites
not for dynamic, product data

Web Data Extraction
ref-code postcode bedrooms bathrooms available price
33453 OX2 6AR 3 2 15/10/2013 £1280 pcm
33433 OX4 7DG 2 1 18/04/2013 £995 pcm
12

: Supervised Data Extraction
Navigation
Steps
Mozilla Web
Browser
Extraction
Configuration
13

Need for Automatic Extraction Technology
14
Example: Real Estate UK > 15000 sites
many not covered by aggregators
list of all agencies easy to get (source discovery)
but: manual or semi-automatic wrapping too expensive
wrapper construction
testing
tracking changes
No existing tool or methodology can do it fully automatically

15
All search engine providers need it! Many work on it.
vertical search
object search
semantic search
no one really has done this successfully at scale yet
Raghu Ramakrishnan, Yahoo!, March 2009
current technologies are not good enough yet to provide what
search engines really need. […] any successful approach would
probably need a combination of knowledge and learning
Alon Halevy, Google, Feb. 2009

DIADEM ›❯ What?
16
This study shows: signiﬁcant long-tail effect for many attributes
>1000 sites to get above 80% coverage required
Examples of these attributes:
phone numbers and home pages of companies
restaurants, car sellers, hotels, banks, …
ISBN of books
reviews of hotels and restaurants
An analysis of structured data on the web, Dalvi et al. (Yahoo) VLDB 2012
for many kinds of information one may have to extract from
thousands of sites in order to build a comprehensive database, even
when we restrict to a given domain with known popular top sites

DIADEM ›❯ What?
Domain-Centric Data Extraction
17
1 <?xml version ="1.0" encoding="UTF-8"?
2 <results>
3 <tyre>
4 <brand>Star Performer</brand>
5 <profile>HP</profile>
6 <price>42.60</price>
7 </tyre>
8 <tyre>
9 <brand>High Performer</brand>
10 <profile>HS-3</profile>
12 </tyre>
13 ...
14 </results>
Blackbox that
turns any of the thousands of websites of a given domain
into structured data

DIADEM ›❯ What?
Domain-Centric Data Extraction
17
1 <?xml version ="1.0" encoding="UTF-8"?
2 <results>
3 <tyre>
4 <brand>Star Performer</brand>
5 <profile>HP</profile>
7 </tyre>
8 <tyre>
9 <brand>High Performer</brand>
10 <profile>HS-3</profile>
12 </tyre>
13 ...
14 </results>
Blackbox that
turns any of the thousands of websites of a given domain
into structured data
DIADEM

Web Data Extraction
Scenario ➀: Electronics retailer
electronics retailer: online market intelligence
comprehensive overview of the market
daily information on price, shipping costs, trends, product mix
by product, geographical region, or competitor
thousands of products
hundreds of competitors
nowadays: specialized companies
mostly manual, sampling
large cost
18

Web Data Extraction › Scenarios
Scenario ➂: Hotel Agency
online travel agency
best price guarantee
prices of competing agencies
average market price
19
taken and report history

Scenario ➃: Hedge Fund
house price index
published in regular intervals by national statistics agency
affects share values of various industries
hedge fund:
online market intelligence to predict the house price index
20

tenders from all over the world
existing aggregators
expensive, often incomplete
yet need to be published (online) by law in most countries
Scenario ➄: Construction
21

… and the Semantic Web
22

22
ref-code postcode bedrooms bathrooms availa
33453 OX2 6AR 3 2 15/10/2
33433 OX4 7DG 2 1 18/04/2

22
ref-code postcode bedrooms bathrooms available price
33453 OX2 6AR 3 2 15/10/2013 £1280 pcm
33433 OX4 7DG 2 1 18/04/2013 £995 pcm

23
Domain database
Whole Domain
Single schema
Rich attributes
Goal:

24
Product
provider Single agency
Few attributes

24
Product
provider Single agency
Few attributes
>15000 in the UK alone

25
Product
provider
Semantic API
(RDF)
Structured API
(XML/JSON)
HTML
interface
1
template
reverse engineering the DB

26
Product
provider
Semantic API
(RDF)
Structured API
(XML/JSON)
HTML
interface
1
template

27
HTML
nterface
1
2
Form ﬁlling

28
HTML
nterface
1
2
Form ﬁlling

30
3
Object
identiﬁcation
Energy Performance Chart
Maps
Tables
FlatText

31
Product
provider
Semantic API
(RDF)
Structured API
(XML/JSON)
HTML
interface
1
template
2
Form ﬁlling
3
Object
identiﬁcation
Maps
Tables
FlatText
Domain database
Cleaning &
integration
4

31
Product
provider
Semantic API
(RDF)
Structured API
(XML/JSON)
HTML
interface
1
template
2
Form ﬁlling
3
Object
identiﬁcation
Maps
Tables
FlatText
Domain database
Cleaning &
integration
4
Other
Provider Other
Provider
Other
Provider
Other
Provider
Otherproviders

32
DIADEM data extraction methodology
domain-centric intelligent automated

DIADEM ›❯ How
DIADEM: Methods and Examples
ROSeAnn: World-best entity extraction from text (VLDB’13+14)
over 350 entity types disambiguated through knowledge/ontology
33

DIADEM ›❯ How
BERyL: Unique block classiﬁcation (ICWE’12)
rich feature model; methodology for easy addition of new features
34
ascending_visual_siblings(X) :- numeric(X, ValueX)
direct_visual_sibling(X,Y,left), direct_visual_sibling(X,Z,right),
numeric(Y, ValueY), numeric(Z, ValueZ), ValueY < ValueX < ValueZ.
Screenshot

DIADEM ›❯ How
OPAL: World-best form understanding (WWW’12,VLDBJ‘13a)
rich feature model with ontology-based classification
35
TEMPLATE basic_concept<C,A> { concept<C>(N) ( N@A{d,e,p} }
2
TEMPLATE concept_by_segment<C,A> {
4 concept<C>(N) ( N@A{e,p} }
6 TEMPLATE concept_minmax<C,CM,A> {
concept<CM>(N1)(child(N1,G),child(N2,G),adjacent(N1,N2),
8 N1@A{e,d},(concept<C>(N2) _ N2@A{e,d})
concept<CM>(N1)(child(N1,G),child(N2,G),follows(N2,N1),
10 concept<C>(N1),N2@range_connector{e,d},¬(A1 A, N2@A1{d})
Range widget ⟸ two fields + connected by “to” or other range connector
+ some clues in the annotations or classifications

DIADEM ›❯ How
OXPath: World-best extraction language (VLDB’11,VLDBJ‘13b)
minimal resource use for cloud extraction; easy to use language
36
Bitemporal Complex Event Processing of
Web Event Advertisements?
Tim Furche1, Giovanni Grasso1, Michael Huemer2,
Christian Schallhart1, and Michael Schreﬂ2
1 Department of Computer Science, Oxford University,
Wolfson Building, Parks Road, Oxford OX1 3QD
firstname.lastname@cs.ox.ac.uk
2 Department of Business Informatics – Data & Knowledge Engineering,
Johannes Kepler University, Altenberger Str. 69, Linz, Austria
lastname@dke.uni-linz.ac.at
doc(’http://www.scottfraser.co.uk/’)//select[@id=’search-type’]/{1 /}
2 //input/{click /}/(//div[1]/table//td[4]/a/{click /})*{0,500}
//div[@class=’property-wrapper’]:<record>
4 [? .:<ORIGIN_URL=current-url()>]

DIADEM ›❯ How
OXPath: World-best extraction language (VLDB’11,VLDBJ‘13b)
minimal resource use for cloud extraction; easy to use language
World-ﬁrst fully automatic, full domain extraction system
over 5000 sites in UK real-estate
37

DIADEM ›❯ How
Core Insight: Phenomenology
Monochromatic
Rectangle
Geographic
search facility
Postcode Active map
….
ISA ISA
Occurs in
Price
search facility
….
….
Occurs in
….
Geo-Price Searchbox
ISA
38
Web Object Ontology (domain-parameterized)

DIADEM ›❯ How
Property Search
Facility
Property List
Single Property
Description
Featured
property
part-of
39

Monochromatic
Rectangle
Geographic
search facility
Postcode Active map
….
ISA ISA
Occurs in
Price
search facility
….
….
Occurs in
….
Geo-Price Searchbox
ISA
DIADEM ›❯ How
40
implements Property Search
Facility
Property List
Single Property
Description
Featured
property
part-of

DIADEM ›❯ How
Object creation in Datalog+
41
PRODUCT
Toshiba Protégé cx
Dell 25416
Dell 23233
Acer 78987
PRICE
480
360
470
390
table(T1) &
table(T2) &
sameColor(T1,T2) &
isNeighbourRight(T1,T2) ⟹ "∃ X (tablebox(X) &
" " contains(X,T1) &
" " contains(X,T2)).

DIADEM ›❯ How
42
PRODUCT
Dell 25416
Dell 23233
Acer 78987
table(T1) &
table(T2) &
sameColor(T1,T2) &
PRICE
480
360
470
390
T1 T2

DIADEM ›❯ How
43
PRODUCT
Dell 25416
Dell 23233
Acer 78987
table(T1) &
table(T2) &
sameColor(T1,T2) &
PRICE
480
360
470
390
T1 T2

DIADEM ›❯ How
44
table(T1) &
table(T2) &
sameColor(T1,T2) &
Deduction in Datalog+ undecidable (TGDs)

DIADEM ›❯ How
45
table(T1) &
table(T2) &
sameColor(T1,T2) &
Deduction in Datalog+ undecidable (TGDs)
Datalog± : require guardedness of rule bodies.
Decidable, linear-time data complexity.

DIADEM ›❯ How
46
DIADEM Architecture
OPAL
Form filling &
understanding
AMBER
Object identification
& alignment
BERyL
Block analysis &
object enrichment
OXPath
Efficient extraction
in the cloud
GLUE
Exploration control and integration language

DIADEM: Statistics
48
sites facts modules
sequential
time
avg.
sequential
Rightmove.co.uk 1 < 1M 1098 12 mins —
Oxfordshire 172 98M 127k 1 day < 10 mins
UK RE (capped) 5000 almost 3B 4M 43 days 10 mins

49
per$Task$ per$Page$ per$Site$ TOTAL$
Sec$ 3.19$ 50.40$ 336.30$ 60534.44$
Min$ 0.05$ 0.84$ 5.61$ 1008.91$
1.00$
10.00$
100.00$
1000.00$
10000.00$
1.00$
10.00$
100.00$
1000.00$
10000.00$
100000.00$
Time%per%…%

50
1.00$ 0.98$ 0.98$
0.36$
1.00$
0.38$
0.20$
0.44$
0.26$
0.98$
0.46$
0.42$
0.72$
0.20$
0.16$
0.04$
0.30$
0.04$
0.00$
0.10$
0.20$
0.30$
0.40$
0.50$
0.60$
0.70$
0.80$
0.90$
1.00$
price$
loca5on$
url$
postcode$
descrip5on$
street_address$
city$
tow
n$
county$
im
age$property_type$
property_status$
bedroom
_num
ber$
bathroom
_num
ber$
recep5on_room
_num
ber$
furnishing$period_unit$
branch_loca5on$
Average'a(ributes'per'record'

51
Avg$#$Ac'ons$ Avg$#$Fillings$ Avg$#$Filled$Text$
All$ 2.61$ 0.44$ 0.03$
form$ 11.20$ 3.34$ 0.21$
result$ 1.73$ 0.00$ 0.00$
0.00$
2.00$
4.00$
6.00$
8.00$
10.00$
12.00$

52 firstname.lastname@cs.ox.ac.uk
2 Department of Business Informatics – Data & Knowledge Engineering,
Johannes Kepler University, Altenberger Str. 69, Linz, Austria
lastname@dke.uni-linz.ac.at
[? .//div[@class=’propertyPrice’]/text()[last()-1]:<PRICE=normalize-space
6 [? .//li[@class=’rec’]/span[@class=’value’]/text():<RECEPTION_ROOM_NUMBER
[? .//div[@class=’propertyTitle’]//@href:<URL=string(.)> ]
8 [? .//span[@class=’priceQualifier’]/text():<PERIOD_UNIT=string(.)> ]
[? .//div[@class=’propertyDescription’]/text()[1]:<DESCRIPTION=string(.)>
10 [? .//li[@class=’bed’]/span[@class=’value’]/text():<BEDROOM_NUMBER=string
[? .//li[@class=’bath’]/span[@class=’value’]/text():<BATHROOM_NUMBER=stri
12 [? .//div[@class=’propertyThumbnail’]/a//@src:<IMAGE=string(.)> ]
[? .//div[@class=’propertyTitleWrapper’]//a/text():<LOCATION=string(.)> ]
doc(’http://www.timruss.co.uk/’)//input[@value=’cntrlListingType_Sales’]/{click /
2 //input[@name=’ctl00$ctl14$btnSearch$ctl00’]/{click /}/
(//div[5]//td/following-sibling::td[contains(string(.),’>’)]/a/{click /})*{0,5
4 //div[@id=’ctl00_cntrlCenterRegion_ctl01_pnlPagingFooter’]/preceding-sibling::

53doc(’http://www.scottfraser.co.uk/’)//select[@id=’search-type’]/{1 /}
[? .//div[@class=’propertyPrice’]/text()[last()-1]:<PRICE=normalize-space(.)> ]
6 [? .//li[@class=’rec’]/span[@class=’value’]/text():<RECEPTION_ROOM_NUMBER=string(.)> ]
[? .//div[@class=’propertyDescription’]/text()[1]:<DESCRIPTION=string(.)> ]
10 [? .//li[@class=’bed’]/span[@class=’value’]/text():<BEDROOM_NUMBER=string(.)> ]
[? .//li[@class=’bath’]/span[@class=’value’]/text():<BATHROOM_NUMBER=string(.)> ]
doc(’http://www.timruss.co.uk/’)//input[@value=’cntrlListingType_Sales’]/{click /}
(//div[5]//td/following-sibling::td[contains(string(.),’>’)]/a/{click /})*{0,500}
4 //div[@id=’ctl00_cntrlCenterRegion_ctl01_pnlPagingFooter’]/preceding-sibling::div/div[1]/div:<
[? .:<ORIGIN_URL=current-url()>]
6 [? .//div/following-sibling::h2//text():<PRICE=substring(normalize-space(.),string-length
[? .//div[@class=’ListResultsRooms’]/div[last()]/span/text():<RECEPTION_ROOM_NUMBER=subst
8 [? .//a[.=’Full Details >’]/@href:<URL=string(.)> ]
[? .//div[contains(@class,’SearchText’)]:<DESCRIPTION=string(.)> ]
10 [? .//div[contains(string(.),’Bedrooms:’)]/span/text():<BEDROOM_NUMBER=substring-after(no
[? .//div[contains(string(.),’Bathrooms:’)]/span/text():<BATHROOM_NUMBER=substring-after(
12 [? .//a[@class=’propAdd’]/text():<TOWN=string(.)> ]
[? .//img[@class=’fulldetails-photo-item’]/@src:<IMAGE=string(.)> ]
14 [? .//a[@class=’propAdd’]/text():<LOCATION=string(.)> ]
[? .//div[@class=’propertyPrice’]/text()[last()-1]:<PRICE=normalize-space(.)> ]
6 [? .//li[@class=’rec’]/span[@class=’value’]/text():<RECEPTION_ROOM_NUMBER=string(.)> ]
[? .//div[@class=’propertyDescription’]/text()[1]:<DESCRIPTION=string(.)> ]
10 [? .//li[@class=’bed’]/span[@class=’value’]/text():<BEDROOM_NUMBER=string(.)> ]
[? .//li[@class=’bath’]/span[@class=’value’]/text():<BATHROOM_NUMBER=string(.)> ]
doc(’http://www.timruss.co.uk/’)//input[@value=’cntrlListingType_Sales’]/{click /}
(//div[5]//td/following-sibling::td[contains(string(.),’>’)]/a/{click /})*{0,500}
4 //div[@id=’ctl00_cntrlCenterRegion_ctl01_pnlPagingFooter’]/preceding-sibling::div/div[1]/div:<
[? .:<ORIGIN_URL=current-url()>]
6 [? .//div/following-sibling::h2//text():<PRICE=substring(normalize-space(.),string-length
[? .//div[@class=’ListResultsRooms’]/div[last()]/span/text():<RECEPTION_ROOM_NUMBER=subst
8 [? .//a[.=’Full Details >’]/@href:<URL=string(.)> ]
[? .//div[contains(@class,’SearchText’)]:<DESCRIPTION=string(.)> ]
10 [? .//div[contains(string(.),’Bedrooms:’)]/span/text():<BEDROOM_NUMBER=substring-after(no
[? .//div[contains(string(.),’Bathrooms:’)]/span/text():<BATHROOM_NUMBER=substring-after(
12 [? .//a[@class=’propAdd’]/text():<TOWN=string(.)> ]
[? .//img[@class=’fulldetails-photo-item’]/@src:<IMAGE=string(.)> ]
14 [? .//a[@class=’propAdd’]/text():<LOCATION=string(.)> ]

DIADEM ›❯ How
54
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

DIADEM ›❯ How
55
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

DIADEM ›❯ OPAL
Navigation in DIADEM: OPAL
56
OPAL is DIADEM’s novel framework for
form and interface understanding and
form and interface navigation
previously navigation mostly
crawler-like: navigate all facets of an interface
probing-based: attempts many “blind” submissions
wide applicability beyond data extraction
meta search; automation; assisted/mobile interfaces

DIADEM ›❯ OPAL
56
Furche, Gottlob, Grasso, Guo, Orsi, Schallhart, OPAL:
Automated form understanding for the deep web.
WWW 2012

DIADEM ›❯ OPAL
56
Furche, Grasso, Guo, Orsi, Schallhart, The Ontological
Key: Automatically Understanding and Integrating
Forms to Access the Deep Web. VLDB Journal 2013

DIADEM ›❯ OPAL
Ontological: Constraints for real estate forms
Annotation schema: Λ=(A,<,≺,(isLabela, isValuea: a ∈ A))
set A of annotation types
a transitive, reflexive subclass relation <
a transitive, irreflexive, antisymmetric precedence relation ≺
and two characteristic functions isLabela and isValuea on text
nodes for each a ∈ A.
Domain schema: Σ = (Λ,T,CT ,CΛ)
annotation schema Λ
set of domain types T
CT, CΛ: map domain types to classification & structural constraints
57

DIADEM ›❯ OPAL
58
Location Location Location
Location
Location
Geographic
Area/BranchBuy/Rent
Buy/Rent
Buy/Rent Type of Use
Local National
Location/…
RentingBuying
Ofﬁce
All
Residential Commercial
Min. Bedrooms
Any
Price Range (£)
0
to
700
Submit
Type of Use
Type of Use
Bedroom
Features
Price
Min-Price Max-Price Button
Buy/Rent Form
Real-Estate Form
OPAL Classiﬁcation over Sample Form

59
TEMPLATE basic_concept<C,A> { concept<C>(N) ( N@A{d,e,p} }
2
TEMPLATE concept_by_segment<C,A> {
4 concept<C>(N) ( N@A{e,p} }
6 TEMPLATE concept_minmax<C,CM,A> {
8 N1@A{e,d},(concept<C>(N2) _ N2@A{e,d})
concept<CM>(N1)(child(N1,G),child(N2,G),follows(N2,N1),
10 concept<C>(N1),N2@range_connector{e,d},¬(A1 A, N2@A1{d})
12 N1@A{e,p},N2@A{e,p}, (N1@min{e,p},N2@max{e,p})
_ (N1@max{e,p},N2@min{e,p})
Figure 8: OPAL-TL classification templates
Figure 7: OPAL-TL classification templates
As an example, the following template defines a family of con
straints that associate the domain type D to a node N whenever
is labeled by an exclusive direct and proper annotation of type A.
TEMPLATE basic_concept<D,A> { concept<D>(N) ( N@A{e,d,l} }
A template tpl is instantiated to produce a family of rules whe
the formal template variables D1,...,Dk are instantiated using va
ues vi
1,...,vi
k from a template instantiation expression of the form
INSTANTIATE tpl<D1,...,Dk> using { <v1
1,...,v1
k> ... <vn
1,...,vn
k> }
For example, the following expression instantiates basic_concep
replacing D with type RADIUS and A with annotation type radius
INSTANTIATE basic_concept<D,A> using {<RADIUS, radius>}

Precision Recall F-score
0.94
0.955
0.97
0.985
1
UK Real Estate (100) UK Used Car (100) ICQ (98) Tel-8 (436)

0.94
0.955
0.97
0.985
1
Su et al., TWeb, 2012
with training

0.94
0.955
0.97
0.985
1
0.9
0.92
0.94
0.96
0.98
1
Airfare Auto Book Job US R.E.
with training

0.94
0.955
0.97
0.985
1
0.9
0.92
0.94
0.96
0.98
1
Airfare Auto Book Job US R.E.
Dragut et al., VLDB, 2009
with training

DIADEM ›❯ Inside
61
Real-estate
Used-car
0.6 0.7 0.8 0.9 1
ﬁeld segment layout domain
Contribution of Scopes

Phenomenology: Datalog±
Infer a new form segment if
there is a group of ﬁelds (G) that is not yet classiﬁed
and has at least two children (N1, N2) of type C
Add all children of G of type C to the new segment
62
candidate-segment<C>(∃ X, G) :- ¬segment(G), child(N1, G), child(N2, G),
concept<C>(N1), concept<C>(N2).
child(X, N) :- candidate-segment<C>(X, G), child(N, G), concept<C>(N, G).
segment<C>(X) :- candidate-segment<C>(X, _).

DIADEM ›❯ How
63
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

64
D1
M1,1
M1,2
D2
…
D3
…
M1,3 E
M1,4
Figure 3: Data area identiﬁcation
its of order dominance: The pivot nodes in E are organized rather
regularly, whereas the pivot nodes in D1 vary quite notably. How-
ever, there variation is small enough that M1,1 to M1,4 are depth and
consistent_cluster_members(C, N1, N2, N3) :- pivot(N1), pivot(N2), ...
similar_depth(N1, N2), similar_depth(N2, N3), similar_depth(N1,N3),
similar_tree_distance(N1, N2, N3).
cluster(C,N) :- continuous,
lca,
contains
at
least
one
of
all
mandatories

65
98
98.5
99
99.5
100
data areas records attributes
precision recall
Real Estate
(100 sites)

65
98
98.5
99
99.5
100
precision recall
Real Estate
(100 sites)
90
92.5
95
97.5
100
price postcode location bathroom bedroom reception legal type
precision recall

65
98
98.5
99
99.5
100
precision recall
98
98.5
99
99.5
100
precision recall
Used Car
(100 sites)
Real Estate
(100 sites)
90
92.5
95
97.5
100
price postcode location bathroom bedroom reception legal type
precision recall

66
Page 1
25%
50%
75%
100%
AMBER RR (!) RR (=) MDR AMBER RR (!) RR (=) MDR
precision recall
Real-Estate Used Car
Fig. 23: Comparison with ROADRUNNER and MDR

DIADEM ›❯ How
67
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

Observational Knowledge
comes in three forms
GATE Gazetteer lists
JAPE rules (roughly EBNF + constraints)
domain-independent classiﬁers
to recognise blocks: advertisements, pagination links, etc.
for attribute and entity extraction
Datalog¬,Agg rules for feature extraction and cleaning
68
house
town house
townhouse
corner house
ﬂat
apartment
maisonette
cottage
converted barn
barn conversion
conversion
mews house
mews
farmhouse
farm
penthouse
residence
lodge
parking space
coach house
bungalow
development
villa
Property type
<money> ::= <currency> <numeric_value>
<rental.price> ::= <money> <rental.period>
| <money> where money.value < rental.price.max
Rental price

68
house
town house
townhouse
corner house
ﬂat
apartment
maisonette
cottage
converted barn
barn conversion
conversion
mews house
mews
farmhouse
farm
penthouse
residence
lodge
parking space
coach house
bungalow
development
villa
Property type
Rental price
Aim: Nearly automatic acquisition of such knowledge

68
house
town house
townhouse
corner house
ﬂat
apartment
maisonette
cottage
converted barn
barn conversion
conversion
mews house
mews
farmhouse
farm
penthouse
residence
lodge
parking space
coach house
bungalow
development
villa
Property type
Rental price
Aim: Nearly automatic acquisition of such knowledge
Furche, Grasso, Kravchenko and Schallhart. Turn the Page:
Automated Traversal of Paginated Websites. In Intl Conf.
on Web Engineering (ICWE). 2012

Observational Knowledge: Block
69
ascending_visual_siblings(X) :- numeric(X, ValueX)
direct_visual_sibling(X,Y,left), direct_visual_sibling(X,Z,right),
numeric(Y, ValueY), numeric(Z, ValueZ), ValueY < ValueX < ValueZ.
Siblings in ascending order
Fig. 1: Numeric (1, 3 14) and non-numeric (
neighborhood of links just as well, but although relatively
tures fail to contribute significantly towards high accuracy
combined with content or structural features, as discussed
give an example where some seemingly good heuristics breaks down? In
heuristic which has been employed by the other approaches.
4. Page position features: Pagination links usually appear on
nated information. Thus, a link’s relative position on a page
the first screen (at a typical resolution) might seem to con
ture. Unfortunately, advertisement or navigation headers
these features significantly (and reliably recognizing thos
For simple features, Section 7 again shows that neither a
either content or structural features high accuracy is achie
ample where some seemingly good heuristics breaks down? Has this been
so, can we give an example from their heuristics and show it fail? If no,
Rename: local visual -> page position, global visual -> neighborhood, (secon
Fortunately, BERyL makes it very easy to extract a large
declarative (Datalog) extraction rules. On the extracted feature
block classification:
trade-off between precision, recall, and speed
different block types require different trade-off
flexible framework for block classification: BERyL

BERyL: Navigation Blocks
70
Website n n1 n2 P R Screenshot
Realestate
FindAProperty 370 1 1 1 1
Zoopla 332 1 1 1 1
Savills 234 2 2 1 1
Cars
Autotrader 262 2 2 1 1
Motors 472 2 2 1 1
Autoweb 103 2 2 1 1
Retail
Amazon 448 1 1 1 1
Ikea 290 2 0 1 1
Lands’ End 527 2 2 1 1
Forums
TechCrunch 279 0 1 1 1
TMZ 200 2 2 1 1
Ars Technica 341 2 2 1 1
Table 1: Sample pages

Phenomenology: Datalog±
Infer a new rectangle if
there are two touching boxes (N1, N2) with
same color and same height (or same width)
no visible border (separator line) between them
no existing box contains only N1 and N2 (omitted here)
Set its dimensions to the MBR for the original boxes
71
box(Y, L, T, R, B) :- mon-rect(Y, L, T, R, B).
∃ X mon-rect(X, L, T, R, B) :- box(N1, L1, T1, R1, B1), box(N2, L2, T2, R2, B2),
touches(N1, N2), same-height(N1, N2), same-color(N1, N2),
¬ visible-border-between(N1, N2), ...
∃ X mon-rect(X, ... open geospatial consor
geometric relations

BERyL: Navigation Blocks
feature model: derived from observed facts
through Datalog program with templates
less than two dozen lines of code
72
TEMPLATE annotated_by<Model,AType> {
2 <Model>::annotated_by<AType>(X) ( node_of_interest(X),
gate::annotation(X, <AType>, _). }
4 TEMPLATE in_proximity<Model,Property(Close)> {
<Model>::in_proximity<Property>(X) ( node_of_interest(X),
6 std::proximity(Y,X), <Property(Close)>. }
TEMPLATE num_in_proximity<Model,Property(Close)> {
8 <Model>::in_proximity<Property>(X,Num) ( node_of_interest(X),
std::proximity(Close,X), Num = #count(N: <Property(Close)>). }
10 TEMPLATE relative_position<Model,Within(Height,Width)> {
<Model>::relative_position<Within>(X, (PosH, PosV)) ( node_of_interest(X),
12 css::box(X, LeftX, TopX, _, _), <Within(Height,Width)>,
PosH = 100·LeftX
Width
, PosV = 100·TopX
Height
. }
14 TEMPLATE contained_in<Model,Container(Left,Top,Bottom,Right)> {
<Model>::contained_in<Container>(X) ( node_of_interest(X),
16 css::box(X,LeftX,TopX,RightX,BottomX), <Container(Left,Top,Right,Bottom)>,
Left < LeftX < RightX < Right, Top < TopX < BottomX < Bottom. }
18 TEMPLATE closest<Model,Relation(Closest,X),Property(Closest),Test(Closest)> {
<Model>::closest<Relation>_with<Property>_is<Test>(X) ( node_of_interest(X),
20 <Relation(Closest,X)>, <Property(Closest)>, <Test(Closest)>,
¬(<Relation(Y,X)>, <Property(Y)>, <Relation(Y,Closest)>). }
Fig. 4: BERyL feature templates
In a similar way, the second template deﬁnes a boolean feature that holds for nodes
of interest, if there is another node in their proximity for which Property(Close) is true.
To instantiate it to nodes that are annotated with PAGINATION, we write
INSTANTIATE in_proximity<Model,Property(Close)>
0.95
0.97
0.98
1.00
Real Estate Cars Retail Forums Total
Precision Recall F1

DIADEM ›❯ How
73
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

OXPath » The Language
OXPath = XPath + 4
74
action
iteration
extraction
style

OXPath = XPath + 4
74
action
iteration
extraction
styleFurche, Gottlob, Grasso, Schallhart and Sellers. OXPath: A
Language for Scalable, Memory-efﬁcient Data
Extraction from Web Applications. VLDB, 2011
Furche, Gottlob, Grasso, Schallhart, and Sellers. OXPATH: A
Language for Scalable Data Extraction, Automation,
and Crawling on the Deep Web. In VLDB J. (VLDB 2012
best paper issue) 2013.

OXPath = XPath + 4
74
action
iteration
extraction
style
Silver price @ “Open Source Software World Challenge 2012”

75 Start at kayak.co.uk:
doc("kayak.co.uk")

doc("kayak.co.uk")
To select an airport, type a few letters and select from completion list
//field().destination/{"Sea" /}
//div#smartbox//li[1]/{click /}

doc("kayak.co.uk")
To select an airport, type a few letters and select from completion list
//field().destination/{"Sea" /}
//div#smartbox//li[1]/{click /}
Submit the form

76
Reﬁne the results by unchecking the “2+ stops”:
//*#stops2/{uncheck }

76
On all result pages
/(//a[.=‘Next’]/{click /})*

76
On all result pages
/(//a[.=‘Next’]/{click /})*
and for each ﬂight
//body.resultrow:<flight>

77
Extract the attributes
Mouseover the ! to extract ﬂight quality warnings
//span.qualityWarningIcon/{mouseover /}

77
Extract the attributes
Mouseover the ! to extract ﬂight quality warnings
//span.qualityWarningIcon/{mouseover /}
Click on the details to extract layovers

0
200
400
600
800
1000
1200
1400
1600
0 100 200 300 400 500 600 700 800
timew/opageloading[sec]
Number of pages
OXPath
Lixto
Web Harvest
Chickenfoot
78

0
200
400
600
800
1000
1200
1400
1600
0 100 200 300 400 500 600 700 800
timew/opageloading[sec]
Number of pages
OXPath
Lixto
Web Harvest
Chickenfoot
even faster
78

DIADEM ›❯ How
79
DIADEM Architecture
OPAL
Form ﬁlling &
understanding
AMBER
& alignment
BERyL
Block analysis &
object enrichment
OXPath
in the cloud
GLUE

DIADEM ›❯ Future
Summary
80
Examples of knowledge (and its representation) in DIADEM
observational: clues for price (“looks like a price”) and location
representation: Gazetteers, JAPE rules, WEKA classiﬁers &
Datalog¬,Agg rules
phenomenological: a real estate record and its attributes
representation: Datalog¬,Agg,± rules
ontological: constraints for real estate form
representation: template language on top of Datalog¬,Agg,± rules
script: strategy for exploring post-form pages
representation: modularised Datalog¬,Agg rules

DIADEM ›❯ Partners
Who wants data from us?
81
Threat detection
[Security analytics, London]
Entity extraction in biology
[Oxford Martin institute, Oxford]
Financial data extraction
[Oxford-Man institute, Oxford]
Forum and blog analysis
[Salzburg research, Austria]

DIADEM ›❯ Partners
Collaborations
82

83
Lehmann, Furche, Grasso, et al. DEQA: Deep Web
Extraction for Question Answering. ISWC 2012.

84
Kindergarden_B
White_Road
1,499,950 £
gr:Offering
rdf:type
dd:hasPrice
Kindergarden_Adbp:near
Domain Speciﬁc
Triple Store
Question:
House near a Kindergarden under 2,000,000 £?
OXPath
OXPath
TBSL
White_Road
Answer:
15
dd:bedrooms
1,499,950 £
dd:hasPrice
dbp:near Kindergarden_A
Linking-Metric
OXPath

DIADEM data extraction methodology

Recommended

Recommended

More Related Content

More from Semantic Technology Institute International

More from Semantic Technology Institute International (20)

Recently uploaded

Recently uploaded (20)

DIADEM data extraction methodology