Corpora, Blogs and Linguistic Variation: Arguments for Using Structured Web Data in Corpus Development Cornelius Puschmann University of Düsseldorf [email_address] University of Paderborn 8 November 2007

Contents of this presentation What counts as evidence in linguistics? System, use and the individual Using the Web for corpus investigations Structured vs. unstructured language data A research example

What counts as evidence in linguistics?

System, use and the individual

Using the Web for corpus investigations

Structured vs. unstructured language data

A research example

What counts as evidence in linguistics?

Four central questions a researcher must answer What is my question? What data can I use? What methods are at my disposal? What are my findings? *0) What are my preliminary assumptions?

What is my question?

What data can I use?

What methods are at my disposal?

What are my findings?

*0) What are my preliminary assumptions?

Different schools of thought ...all have different questions and assumptions! CogSci SocioLing Functionalism

What role does corpus data play? If my aim is figure out what we know when we know language, then corpus data is just one type of evidence among many. If my aim is to describe the social function of language in a speech community, then I'll be interested in some (specific) natural language data. If my aim is to systematically describe, classify and manipulate natural language data for practical reasons, it is likely to be the only thing to qualify as data to me. The relevance of corpus data can range from “somewhat interesting” to “what else is there?”, depending on my perspective.

If my aim is figure out what we know when we know language, then corpus data is just one type of evidence among many.

If my aim is to describe the social function of language in a speech community, then I'll be interested in some (specific) natural language data.

If my aim is to systematically describe, classify and manipulate natural language data for practical reasons, it is likely to be the only thing to qualify as data to me.

The relevance of corpus data can range from “somewhat interesting” to “what else is there?”, depending on my perspective.

System, use and the individual

A totalizing view of language production investigation but... whose system? whose use? system social function cognitive mechanism genetic disposition use cultural transmission

Margin vs. center: what is shared vs. what varies If we're interested in variation, corpus analysis is the way to go shared, recurring & patterned language individual & varying language

A (slightly) different way of looking at language While variation in language use is recognized by linguists, system is generally believed to be an abstract and essentially universal category. Alternately, it is possible to regard system as the sum of all recurring and patterned instances of language use by a single speaker, some of which are shared with other speakers, while others are not. How could similarities and differences between speakers be accurately captured and measured? They could be captured with corpora that allow us to systematically compare different speakers.

While variation in language use is recognized by linguists, system is generally believed to be an abstract and essentially universal category.

Alternately, it is possible to regard system as the sum of all recurring and patterned instances of language use by a single speaker, some of which are shared with other speakers, while others are not.

How could similarities and differences between speakers be accurately captured and measured?

They could be captured with corpora that allow us to systematically compare different speakers.

Using the Web for corpus investigations

The ultimate corpus the indexable Web has more than 11.5 billion pages (2005 study) virtually all (written) languages are represented established forms of writing (fiction, official documents, personal communication) and new genres (blogs, message boards) can be found on the Web

the indexable Web has more than 11.5 billion pages (2005 study)

virtually all (written) languages are represented

established forms of writing (fiction, official documents, personal communication) and

new genres (blogs, message boards) can be found on the Web

Web as Corpus (WaC) WaC treats the vast amount of language data on the WWW as a corpus search engines are used to query this corpus they can be commercial (e.g. Google), or specialized tools for linguistic research (e.g. WebCorp, LSE) but: specialized linguistic search engines limited to post-processing (?) pros: large volume of data, no data acquisition necessary, easy to use cons: no knowledge of the makeup of the dataset, no control over the dataset or the query engine, no tagging, parsing or other specialized linguistic tools (with commercial engines)

WaC treats the vast amount of language data on the WWW as a corpus

search engines are used to query this corpus

they can be commercial (e.g. Google), or specialized tools for linguistic research (e.g. WebCorp, LSE)

but: specialized linguistic search engines limited to post-processing (?)

pros: large volume of data, no data acquisition necessary, easy to use

cons: no knowledge of the makeup of the dataset, no control over the dataset or the query engine, no tagging, parsing or other specialized linguistic tools (with commercial engines)

Broader issues with WaC When we're looking for qualitative data, WaC is relatively unproblematic, but when we're comparing frequencies (i.e. taking a quantitative approach) it has serious issues. The fact that Google controls the data means that both the data and how is processed are unknown and may change at any time ...and most importantly: Google doesn't care!

When we're looking for qualitative data, WaC is relatively unproblematic, but when we're comparing frequencies (i.e. taking a quantitative approach) it has serious issues.

The fact that Google controls the data means that both the

data and

how is processed

are

unknown and

may change at any time

...and most importantly: Google doesn't care!

Web for Corpus Web for Corpus (WfC) extracts data from the Web and stores it locally it is closer to traditional corpus development in the sense that data is consciously added by the researcher following certain criteria (randomness, balance etc) pros: control over makeup of the dataset, precise knowledge of its size, ability to annotate, reuse, share and publish the dataset, ability to parse, tag ... cons: corpus construction is technically more challenging, corpus is smaller than the Web in its entirety (though still larger than traditional corpora)

Web for Corpus (WfC) extracts data from the Web and stores it locally

it is closer to traditional corpus development in the sense that data is consciously added by the researcher following certain criteria (randomness, balance etc)

pros: control over makeup of the dataset, precise knowledge of its size, ability to annotate, reuse, share and publish the dataset, ability to parse, tag ...

cons: corpus construction is technically more challenging, corpus is smaller than the Web in its entirety (though still larger than traditional corpora)

Constructing a corpus using web data Pick a data source (Wikipedia, Project Gutenberg, blogger.com, the entire WWW as indexed by Google, ...) Retrieve the raw HTML data ( spidering or crawling ) Process the HTML data, i.e. separate natural language from formatting instructions. For example <h1> Dear diary </h1> I am <b> really </b> bored &amp; tired right now... <br/> becomes Dear diary I am really bored & tired right now Tag, parse and store

Pick a data source (Wikipedia, Project Gutenberg, blogger.com, the entire WWW as indexed by Google, ...)

Retrieve the raw HTML data ( spidering or crawling )

Process the HTML data, i.e. separate natural language from formatting instructions. For example <h1> Dear diary </h1> I am <b> really </b> bored &amp; tired right now... <br/> becomes Dear diary I am really bored & tired right now

Tag, parse and store

Things to consider ...not a whole lot of language data!

Things to consider ... better, but we need to take register into account

Structured vs. unstructured language data

Structured vs. unstructured language data

A blog is... Blog (n) : a website where entries are written in chronological order and displayed in reverse chronological order (“web log”). blog (v) : to maintain or add content to a blog. Blogs are written by a variety of people (“bloggers”) with a variety of purposes. Every text in a blog can be directly linked to its author and usually has other meta-information (date, keywords etc).

Blog (n) : a website where entries are written in chronological order and displayed in reverse chronological order (“web log”).

blog (v) : to maintain or add content to a blog.

Blogs are written by a variety of people (“bloggers”) with a variety of purposes. Every text in a blog can be directly linked to its author and usually has other meta-information (date, keywords etc).

A few facts Technorati (blog search engine) tracks about 70 million blogs worldwide 120.000 new blogs are created each day – that's 1.4 every second 1.5 million entries (posts) are published every day virtually every (written) language on the planet is represented in blogs blog data is well-structured in the sense that it doesn't contain visual markup The markup of a blog is semantic, meaning it contains meta-information about the content that a machine can understand.

Technorati (blog search engine) tracks about 70 million blogs worldwide

120.000 new blogs are created each day – that's 1.4 every second

1.5 million entries (posts) are published every day

virtually every (written) language on the planet is represented in blogs

blog data is well-structured in the sense that it doesn't contain visual markup

The markup of a blog is semantic, meaning it contains meta-information about the content that a machine can understand.

An example

A research example

Expression of futurity in English: will vs. be going to will origin as a transitive lexical verb (OE willan) with a meaning similar to German wollen; has been grammaticalized to express futurity be going to can be combined with +NP (movement) or +Inf.V (future); movement “away from the speaker, towards a goal” (Perez) “ The notion of satisfying a condition forms one of the major distinctions between the two future expressions will and be going to . A sentence with will relies on a condition evident in the context that will enable the proposition to take place.” (Perez)

will origin as a transitive lexical verb (OE willan) with a meaning similar to German wollen; has been grammaticalized to express futurity

be going to

can be combined with +NP (movement) or +Inf.V (future); movement “away from the speaker, towards a goal” (Perez)

“ The notion of satisfying a condition forms one of the major distinctions between the two future expressions will and be going to . A sentence with will relies on a condition evident in the context that will enable the proposition to take place.” (Perez)

Distribution of will vs. be going to in three blogs light blue = will dark blue = going to

Personal pronoun frequency in the same blogs THOMPSON 01 the DT 2638 02 and CC 1604 03 to TO 1272 04 of IN 1152 05 a DT 1034 06 in IN 821 07 For IN 680 08 on IN 509 09 is VBZ 465 10 with IN 375 J.SCHWARTZ 01 the DT 3077 02 and CC 2120 03 to TO 2072 04 a DT 1862 05 of IN 1486 06 in IN 1008 07 we PP 988 08 I PP 818 09 our PP$ 723 10 It PP 642 H.HAMILTON 01 the DT 3802 02 I PP 3613 03 to TO 2789 04 a DT 2045 05 of IN 1795 06 and CC 1788 07 It PP 1519 08 you PP 1205 09 in IN 1096 10 that IN 1077

THOMPSON

01 the DT 2638

02 and CC 1604

03 to TO 1272

04 of IN 1152

05 a DT 1034

06 in IN 821

07 For IN 680

08 on IN 509

09 is VBZ 465

10 with IN 375

J.SCHWARTZ

01 the DT 3077

02 and CC 2120

03 to TO 2072

04 a DT 1862

05 of IN 1486

06 in IN 1008

07 we PP 988

08 I PP 818

09 our PP$ 723

10 It PP 642

H.HAMILTON

01 the DT 3802

02 I PP 3613

03 to TO 2789

04 a DT 2045

05 of IN 1795

06 and CC 1788

07 It PP 1519

08 you PP 1205

09 in IN 1096

10 that IN 1077

Inanimate subjects with be going to (1) my car is going to get pretty crusty (2) the exhibit is going to be in Seattle (3) the things that are left to do in this house are going to cost some money (4) the work is still going to be there the next day (5) my first post is going to be about troublesome managers

(1) my car is going to get pretty crusty

(2) the exhibit is going to be in Seattle

(3) the things that are left to do in this house are going to cost some money

(4) the work is still going to be there the next day

(5) my first post is going to be about troublesome managers

Observations The assumption that be going to is more frequent with certain subjects is confirmed by the data (1 st pers. -> 2 nd pers. -> human -> animate -> inanimate). This appears to be the strongest conditioning factor – degree of certainty seems less significant. However, where subjects are inanimate, the course of action can always be described as certain. These observations could be tested against a large number of sources (blogs), factoring in individual variation in addition to other factors.

The assumption that be going to is more frequent with certain subjects is confirmed by the data (1 st pers. -> 2 nd pers. -> human -> animate -> inanimate).

This appears to be the strongest conditioning factor – degree of certainty seems less significant.

However, where subjects are inanimate, the course of action can always be described as certain.

These observations could be tested against a large number of sources (blogs), factoring in individual variation in addition to other factors.

Thanks for listening!

Corpora, Blogs and Linguistic Variation: Arguments for Using Structured Web Data in Corpus Development Cornelius Puschmann University of Düsseldorf [email_address] University of Paderborn 8 November 2007