Corpora, Blogs and Linguistic Variation (Paderborn)

Corpora, Blogs and Linguistic Variation: Arguments for Using Structured Web Data in Corpus Development Cornelius Puschmann University of Düsseldorf [email_address] University of Paderborn 8 November 2007

Contents of this presentation
What counts as evidence in linguistics?
System, use and the individual
Using the Web for corpus investigations
Structured vs. unstructured language data
A research example

What counts as evidence in linguistics?

Four central questions a researcher must answer
What is my question?
What data can I use?
What methods are at my disposal?
What are my findings?
*0) What are my preliminary assumptions?

Different schools of thought ...all have different questions and assumptions! CogSci SocioLing Functionalism

What role does corpus data play?
If my aim is figure out what we know when we know language, then corpus data is just one type of evidence among many.
If my aim is to describe the social function of language in a speech community, then I'll be interested in some (specific) natural language data.
If my aim is to systematically describe, classify and manipulate natural language data for practical reasons, it is likely to be the only thing to qualify as data to me.
The relevance of corpus data can range from “somewhat interesting” to “what else is there?”, depending on my perspective.

System, use and the individual

A totalizing view of language production investigation but... whose system? whose use? system social function cognitive mechanism genetic disposition use cultural transmission

Margin vs. center: what is shared vs. what varies If we're interested in variation, corpus analysis is the way to go shared, recurring & patterned language individual & varying language

A (slightly) different way of looking at language
While variation in language use is recognized by linguists, system is generally believed to be an abstract and essentially universal category.
Alternately, it is possible to regard system as the sum of all recurring and patterned instances of language use by a single speaker, some of which are shared with other speakers, while others are not.
How could similarities and differences between speakers be accurately captured and measured?
They could be captured with corpora that allow us to systematically compare different speakers.

Using the Web for corpus investigations

The ultimate corpus
the indexable Web has more than 11.5 billion pages (2005 study)
virtually all (written) languages are represented
established forms of writing (fiction, official documents, personal communication) and
new genres (blogs, message boards) can be found on the Web

Web as Corpus (WaC)
WaC treats the vast amount of language data on the WWW as a corpus
search engines are used to query this corpus
they can be commercial (e.g. Google), or specialized tools for linguistic research (e.g. WebCorp, LSE)
but: specialized linguistic search engines limited to post-processing (?)
pros: large volume of data, no data acquisition necessary, easy to use
cons: no knowledge of the makeup of the dataset, no control over the dataset or the query engine, no tagging, parsing or other specialized linguistic tools (with commercial engines)

Broader issues with WaC
When we're looking for qualitative data, WaC is relatively unproblematic, but when we're comparing frequencies (i.e. taking a quantitative approach) it has serious issues.
The fact that Google controls the data means that both the
data and
how is processed
are
unknown and
may change at any time
...and most importantly: Google doesn't care!

Web for Corpus
Web for Corpus (WfC) extracts data from the Web and stores it locally
it is closer to traditional corpus development in the sense that data is consciously added by the researcher following certain criteria (randomness, balance etc)
pros: control over makeup of the dataset, precise knowledge of its size, ability to annotate, reuse, share and publish the dataset, ability to parse, tag ...
cons: corpus construction is technically more challenging, corpus is smaller than the Web in its entirety (though still larger than traditional corpora)

Constructing a corpus using web data
Pick a data source (Wikipedia, Project Gutenberg, blogger.com, the entire WWW as indexed by Google, ...)
Retrieve the raw HTML data ( spidering or crawling )
Process the HTML data, i.e. separate natural language from formatting instructions. For example <h1> Dear diary </h1> I am <b> really </b> bored & tired right now... <br/> becomes Dear diary I am really bored & tired right now
Tag, parse and store

Things to consider ...not a whole lot of language data!

Things to consider ... better, but we need to take register into account

Structured vs. unstructured language data

A blog is...
Blog (n) : a website where entries are written in chronological order and displayed in reverse chronological order (“web log”).
blog (v) : to maintain or add content to a blog.
Blogs are written by a variety of people (“bloggers”) with a variety of purposes. Every text in a blog can be directly linked to its author and usually has other meta-information (date, keywords etc).

A few facts
Technorati (blog search engine) tracks about 70 million blogs worldwide
120.000 new blogs are created each day – that's 1.4 every second
1.5 million entries (posts) are published every day
virtually every (written) language on the planet is represented in blogs
blog data is well-structured in the sense that it doesn't contain visual markup
The markup of a blog is semantic, meaning it contains meta-information about the content that a machine can understand.

An example

A research example

Expression of futurity in English: will vs. be going to
will origin as a transitive lexical verb (OE willan) with a meaning similar to German wollen; has been grammaticalized to express futurity
be going to
can be combined with +NP (movement) or +Inf.V (future); movement “away from the speaker, towards a goal” (Perez)
“ The notion of satisfying a condition forms one of the major distinctions between the two future expressions will and be going to . A sentence with will relies on a condition evident in the context that will enable the proposition to take place.” (Perez)

Distribution of will vs. be going to in three blogs light blue = will dark blue = going to

Personal pronoun frequency in the same blogs
THOMPSON
01 the DT 2638
02 and CC 1604
03 to TO 1272
04 of IN 1152
05 a DT 1034
06 in IN 821
07 For IN 680
08 on IN 509
09 is VBZ 465
10 with IN 375
J.SCHWARTZ
01 the DT 3077
02 and CC 2120
03 to TO 2072
04 a DT 1862
05 of IN 1486
06 in IN 1008
07 we PP 988
08 I PP 818
09 our PP$ 723
10 It PP 642
H.HAMILTON
01 the DT 3802
02 I PP 3613
03 to TO 2789
04 a DT 2045
05 of IN 1795
06 and CC 1788
07 It PP 1519
08 you PP 1205
09 in IN 1096
10 that IN 1077

Inanimate subjects with be going to
(1) my car is going to get pretty crusty
(2) the exhibit is going to be in Seattle
(3) the things that are left to do in this house are going to cost some money
(4) the work is still going to be there the next day
(5) my first post is going to be about troublesome managers

Observations
The assumption that be going to is more frequent with certain subjects is confirmed by the data (1 st pers. -> 2 nd pers. -> human -> animate -> inanimate).
This appears to be the strongest conditioning factor – degree of certainty seems less significant.
However, where subjects are inanimate, the course of action can always be described as certain.
These observations could be tested against a large number of sources (blogs), factoring in individual variation in addition to other factors.

Thanks for listening!

Corpora, Blogs and Linguistic Variation: Arguments for Using Structured Web Data in Corpus Development Cornelius Puschmann University of Düsseldorf [email_address] University of Paderborn 8 November 2007

http://corpblawg.ynada.com	10
http://www.slideshare.net	2
http://joeandco.blogspot.com	1
http://blogtecologia.blogspot.com	1

Corpora, Blogs and Linguistic Variation (Paderborn)

by Cornelius Puschmann on Nov 16, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 14

Statistics

Corpora, Blogs and Linguistic Variation (Paderborn) — Presentation Transcript