Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn Anderson

Does Zipfian Distribution
Explain SEO?
Presented by: Dawn Anderson
#Pubcon@dawnieando

Does%Zipfian%
Distribution%
Explain%SEO?
@dawnieando

We#Know#
That#SEO#Is#
All#About
Words
Links
Crawling
Indexing
Ranking
Importance
Popularity
Probability;determination
Disambiguation

There%is%a%
phenomenon%
‘hiding%in%plain%
sight’%(blending%in)%
which%impacts%all%of%
these%elements

Zipf’s'Law'+ The'Principle'of'
Least'Effort'is'Everywhere

We#know#words#&#linguistics#have#
unusual#patterns#e.g.
“You#shall#know#a#word#by#the#
company#it#keeps”#(Firth,#1957)#(coE
occurrence)

Zipf’s Law*is*another*
one*of*those*strange*
linguistic*phenomena…*
but*MUCH*more*than*
that

Information*Retrieval*Lecture*2 University*
of*Freiburg*– Lecture*1*– Zipf’s Law

Vsauce Video*– The*Zipf Mystery*5 You*should*watch*it

So…$Just$What$is$Zipf’s Law?

Zipf’s Law*is*about*
frequency*distribution,*
rank*law,*&*is*often*
associated*with linguistics

Zipf’s Law*is*Prevalent*
When*It*Comes*to*
‘Popularity’*(Rank)*
Distributions

Zipf’s Law*is*
Empirical*(Widely*
accepted).*Not*
just*a*theory

Popularised by.George.
Kingsley.Zipf
• George&Kingsley&Zipf (1902&5 1950)
• Linguist&&&philologist
• Popularised ‘Zipf’s Law’&but&did&not&claim&to&have&discovered&it
• Observed&patterns&in&word&frequency&distribution
• Extended&Zipf’s Law&to&data&types&other&than&words
• Developed&‘Principle&of&Least&Effort’&(Path&of&least&resistance)
• Image&attribution:&Freeport&High&School,&Freeport,&Illinois.,&Public&
domain,&via&Wikimedia&Commons

‘Normal’)is)
definitely)
something)
which)Zipf’s
Law)is)NOT

A"‘Standard"Deviation"Curve’"(Normal"Distribution)
This%file%is%licensed%under%the Creative%Commons Attribution%2.5%Generic license.

In#a#‘Normal’#Distribution
Very%few%observations%with%very%low%values
Quite%a%Few%observations%with%%low%values
Many%observations%with%average%values
Quite%a%Few%observations%with%high%values
Very%few%observations%with%very%high%values

A"Zipfian"
Distribution"is"VERY"
Different

A"Zipfian"Distribution"is"Very"
Skewed"&"Heavily"Biased

‘Zipfian’)Distribution
Very%few%observations%with%very%high%values
The%most%frequent%item%(e.g.%word)%is%twice%as%
frequent%as%the%second%most%frequent%item
The%third%most%frequent%item%is%1/3%the%value%of%the%
first%most%frequent%item…%and%so%on
Very,%very%many%observations%with%low%values

Of#Course.#As#SEOs#We#Are#Used#To#Seeing#‘Long#Tail#
Distributions’.#In#Search#Behaviour

But$It$Goes$Much$
Further$Than$Search$
Demand$Curves

In#Linguistics#+ By#Zipfs Law#– The#
Frequency#of#Any#Word#Is#Inversely#
Proportional#To#Its#Rank#In#The#
Frequency#Table

Zipfian'
Distribution
Layman’s(terms(– In(any(count(of(
the(frequency(of(words(in(a(large(
body(of(content,(the(probability(of(
occurrence(of(words(or(other(items(
starts(high(and(tapers(off.(
A"few"occur"very"often"
while"many"others"occur"
rarely.

’The’%is%the%most%used%word%in%the%
English%language%(7%%frequency)
‘The’%is%used%twice%as%often%as%the%next%
most%used%word%in%English%(3.5%)
And%three%times%as%much%as%the%next%
most%used%word
Etcetera,%etcetera,%etcetera%infinitium

The$Most$Used$
Words$In$The$
English$Language$
Follows$a$Zipfian$
Distribution

Brown&Corpus&– 1&Million&Word&Text

It#starts#off#
looking
something
like this
0
10000
20000
30000
40000
50000
60000
70000
80000
The Of And To A In That Is Was He For It With As His On
Occurrence
Occurrence

Till$it’s$this$)
Brown$Corpus$
Words$Ranked$
by$Frequency$
Distribution
'The’

And$it$doesn’t$just$apply$to$
English$either

Even%languages%
untranslated%as%yet

Zipf’s Law*&*
30*Languages*
of*Wikipedia*
(Image*credit*Wikiwand)

By Zipfs Law
A"few"words"
will"be"used"
very"frequently
But"account"for"
a"large"part"of"
the"vocabulary
Many"words"
will"be"used"
infrequently
But"account"for"
a"small"part"of"
the"vocabulary

In#most#
languages#the#
most#popular#
100#words#make#
up around 50%#
of#all text

“In$the$Brown$Corpus,$consisting$of$over$one$
million$words,$half$of$the$word$volume$
consists$of$repeated$uses$of$only$135$words.”$
(Fagan$&$Gencay,$2011)

Historically,,of,course,,
many,of,these,words,
were,considered,to,
add,no,value,(stop,
words)

ASIDE&' Any&value
from most of&these&
words&is&from&
‘contextual&glue’&in&
natural&language&
‘post&BERT&and&
friends’

Zipf’s Law*is*Aligned*with*Pareto’s*80:20*
Rule*Law*(The*Law*of*The*Vital*Few)

By#Pareto#* 20%#of#the#words#
account#for#80%#of#the#vocabulary

Pareto'is'also'EVERYWHERE'too'– Examples
20%$of$drivers$cause$80%$of$all$traffic$accidents.
20%$of$a$company’s$products$represent$80%$of$the$sales.
20%$of$employees$are$responsible$for$80%$of$the$results.

20%$of$pea$pods$produce$
80%$of$the$peas

Even…20%)of)a)
carpet)receives)
80%)of)the)wear

So#Why#Does#This#
‘Zipfyness’#Exist?

Zipf’s'Law'
Appears'to'
Apply'to...Lots'
and'lots'of'
types'of'data

Zipf’s Law*&*The*Population*of*Cities*In*Countries
Appears'to'always'hold'
true'only'when'the'cities'
are'economically'related'
(e.g.'in'the'same'country'
as'each'other)

• Zipf%rank%plot%for%276%
metropolitan%areas%in%the%United%
States,%after%results%of%the%census%in%
2000.%Source:%factfinder.census.gov.%
The%straight%line%has%slope%1.11.

This%biased%spread%holds%true%
for%income%distribution%!

So…$There$are these$same$
predictable$patterns$across$many$
aspects$of$the$web,$within$ccTLDs$
&$even$website$&$webpage$
features$themselves

Pretty&much&every&aspect&follows&a&
Power&Law

And$Zipfs Law$is$a$Power$Law
“A power law*is*a*functional relationship between*two*quantities,*where*a*relative*change*in*one*
quantity*results*in*a*proportional*relative*change*in*the*other*quantity,*independent*of*the*initial*
size*of*those*quantities:*one*quantity*varies*as*a power of*another.”*(Source:*Wikipedia)

These%Laws%Don’t%Change%In%‘Scale%Free’%Networks%Such%
as%The%Web%Grows%or%Ages%Either
This%file%is%licensed%under%the Creative%Commons Attribution6Share%Alike%3.0%Unported license.

A"Small"Example"in"All"The"Words"of"a"Blog"Site

Etymology(Nerd’s(
Study
• 490$page$blog
• Top$40$most$frequently$occurring$words$ordered$by$rank
• Followed$a$Zipfian$distribution

But$It$Goes$Well$Beyond$Just$
The$‘Words’$on$The$Internet

Zipfian'Distribution'Examples'On'The'Internet
Number'of'
inlinks'to'
pages
Number'of'
outlinks from''
pages
Size'of'
websites
No.'of'pages'
in'websites
Number'of'
words'on'a'
page
Number'of'
videos,'images'
&'audio'files

In#word#
distribution#on#
a#ccTLD
In#internal#link#
distribution#on#
within#a#ccTLD
In#external#link#
distribution#
within#a#ccTLD
In#the#
popularity#of#
sites
In#the#
popularity#of#
queries
In#the#
popularity#of#
web#pages

Zipfs Law)and)The)
Internet
Whilst)the)data)is)older)
Zipfs Law)still)prevails
Adamic,)L.A.)and)
Huberman,)B.A.,)2002.)
Zipf's law)and)the)
Internet. Glottometrics,
3(1),)pp.143G150.

I"asked"an"eminent"search"engineer"to"be"sure"
this"was"correct."He"replied"yes

Some%Baeza)Yates%Papers%on Web%Characteristics
Characterization,
of,National,Web,
Domains,(2007)
Link,Analysis,in,
National,Web,
Domains,(2005),
Characteristics,
of,the,Web,of,
Spain,(2005)
Bias,on,The,
Web,(2018)

It#is#true.#All#aspects#of#the#
web#show#this#power#law#
but#it#is#less#precise#on#
webpages#at#the#very#far#
ends#of#the#distribution#
(Baeza<Yates,#2005,2007)

Shame&Law&in&web&
pages&(Minimal&
Shame)&Baeza5Yates,&
2007
Where%authors%or%organisations do%some%
minimal%work%because%they%don’t%want%to%be%
ashamed%of%their%work%(minimal%effort).%Makes%
Zipfs Law%less%precise%on%the%small%pages%or%
websites.

What%About%At%The%
Long,%Long,%Long%Tail?
Ricardo(explained(it’s(because(the(are(too(many(
unique(value(for(it(to(be(exact

ccTLD%Power%
Laws%Versus%a%
‘Whole%
Internet’%
Power%Law
• Each%ccTLD%will%have%a%different%power%law%
(different%exponent)%
• Although%in%general%the%sum%of%ccTLD%power%
laws%is%not%a%power%law,%in%many%cases%it%is%
because%you%have%a%dominant%one.%
• For%example%for%the%whole%Web,%the%USA%will%
dominate.%Experimental%data%shows%this.
Many%thanks%to%Ricardo%BaezaJYates%for%providing%
this%response when questioned

Given&a&known&Power&Law&
Distribution&in&EVERYTHING&on&the&
web&how&might&this&be&reflected&in&
search&engine&systems?

Given&That&By&Zipf’s &&Pareto’s&Law
A"few"sites"will"be"
sites"will"be"visited""
very"frequently
But"account"for"a"
large"part"of"
search"traffic
Many"sites"will"be"
visited"
infrequently
And"account"for"a"
small"part"of"
overall"search"
traffic

They%
Know%ALL%
About%
Zipfs%Law
“No$paper$on$statistics$of$web$pages$
is$complete$without$a$graph$
showing$a$power6law$distribution.”$
(Fetterly,$2005)

Zipf’s'Law'Helps'
with'Search'
Systems'Built'for'
Scale

Since&EVERYTHING&Is&
‘Naturally’&Much&More&
Predictable

Priorities(are(VERY(
obvious(via(Zipf’s
Law

Given&That&By&Zipf’s &&Pareto’s&Law
A"few"sites"will"be"
sites"will"be"visited""
very"frequently
But"account"for"a"
large"part"of"
search"traffic
Many"sites"will"be"
visited"
infrequently
But"account"for"a"
small"part"of"
search"traffic

Some%‘things’%are%simply%NOT%as%important%as%
others Cities,'towns'&'
villages'
(Population)
Many'Low'quality'
content'URLs
Sites:'Think'CNN'
vs'Mom'&'Pop'
Blog
Few'High'vs'Many,'
Many'Low'or'No'
PageRank'URLs
Head'terms'versus'
long'tail'search'
terms

Much%
Literature%
Confirms%
Zipf’s Law%is%
Considered%
in:
Crawling
Indexing
Caching
Ranking
Quality5Determination

Zipfs&Law&&&Crawl&
Frequency&
Scheduling&– AKA&
‘Crawl&Budget’?
Crawl&
budget
John&
Mueller
Gary&
Illyes
Link&
Juice
Meme&
with&
cats
Duplicate&
content

Crawl&Budget&is&
absolutely&tied&to&
Zipfian&Skewed&
‘Demand’

‘Most’'Sites'Will'NOT'Have'High'Crawl'
Budget…'EVER
https://support.google.com/webmasters/answer/9689511?hl=en

The$Rate$of$Change$on$
Sites$Will$Likely$Follow$a$
Power$Law$Too$
(Predictable)

As#Will#The#Rate#of#
Content#Creation#on#
Sites#(Predictable)

‘Quality’*Issues*Can*Arise*as*a*
Result*of*Too*Many*
Automatically*Generated*
Pages*Defying*Zipf’s*Law*(The*
rate*of*creation*is*obviously*
not*by*human*hand)

Zipfian'Quality'Thresholds'
Meeting'will'‘probably’'shift'
for'the'site.'Quality;
impacted;Crawling'will'
‘probably’'be'Impacted

You$Threw$a$‘Long$Tail’$
of$Low,$Low$Importance$
URLs$Into$The$Mix

You$Just$
Bought$
Some$Places$
At$The$Back$
of$The$‘Crawl$
Schedule’$
(Or$Didn’t$
Make$The$
Grade$At$All)

'Crawl'Budget’'Issues'
are'Often'OVERALL'‘Site'
Quality’'Issues

Since&You’re&
Being&Judged&
on&The&‘Whole’&
Cake

Aside&from&’Demand’&&&
Overall&Quality

Crawl&‘budget’&
is mostly
about
‘Importance’(
which&follows&
Zipf’s Law)&
(aside&from&
host&load&
capabilities)
Demand
PageRank+(or+
equivalent)
Inclusion+in+
sitemap
Internal+links
External+links Canonical+tag
Redirections
Quality+
parents

All#of#These#
Things#Likely#
Are#Impacted#
by#Zipf’s Law

The Zipfian Distribution of Many Things

What%About%Zipfyness%in%
Indexing%Systems?

One$Index.$Multiple$
Tiers$Within$It$(Probably)
• “Each&tier&is&the&next&level&of&document&popularity&
and&usually&popularity&is&similar&to&a&Zipf&distribution&in&
its&central&part”&
• (This&is&Ricardo’s&answer&to&me&when&I&asked&about&
this&last&week)

Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn Anderson

Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn Anderson

More Related Content

What's hot

More from Dawn Anderson MSc DigM

Recently uploaded

Zipfs Law & Zipfian Distribution in SEO - Pubcon Virtual Fall 2020 - Dawn Anderson