1.
Quantitative Individuated Corpus Linguistics: A Speaker-Centric Approach to Variation Cornelius Puschmann Universität Osnabrück 5 Juni 2007

2.
Preliminaries

3.
A totalizing view of language competence performance production investigation but... whose competence? whose performance? social function learning cognitive basis cultural transmission

4.
Variation or overlap? competence performance Speaker A competence performance Speaker B competence performance Speaker C Observations: 1. A contrastive comparison of performance should give us some insight into shared competence. 2. Speaker-level granularity is preferable to higher levels of segmentation (by gender, social class etc). 3. Instead of generalizing from the outset, we can reach general conclusions after observing the degree of variation or overlap in language production. So how do we do this?

5.
How corpora treat language data <ul><li>any sentence is as good as any other sentence (the data is flat) </li></ul><ul><li>a corpus should be a well-balanced mix of different genres, modes and sources (representativeness) </li></ul><ul><li>textual and compositional coherence cannot be taken into account </li></ul><ul><li>contextual information (who said what, when, where, why, how and to whom) is largely unavailable </li></ul>

6.
Corpora and traditions of text production <ul><li>copora largely consist of “well-established genres” </li></ul><ul><li>the material they contain is produced by “language professionals” (journalists, writers, politicians) </li></ul><ul><li>texts are long and stylistically distant from everyday communication in their level of formality, complexity and elaborateness </li></ul><ul><li>compositional integrity (text structure) is very important but largely ignored </li></ul><ul><li>the text (=collection of words) takes precedent over the speaker </li></ul>

7.
A different view of language data <ul><li>language data and sources of </li></ul><ul><li>variation... </li></ul><ul><li>... vs. speakers and their natural attributes </li></ul>

8.
Blogs as data sources

9.
A new kind of resource <ul><li>estimated 100 million active bloggers in 2007 </li></ul><ul><li>split evenly among genders </li></ul><ul><li>all age groups are represented </li></ul><ul><li>many bloggers provide personal information (age, gender, location) </li></ul><ul><li>use web feeds (Atom and RSS formats) to syndicate blog entries in XML (ideal for building modern corpora) </li></ul><ul><li>“clean” data with minimal interference </li></ul>

10.
Blogs as corpus data: Pros <ul><li>very large bodies of data can be automatically assembled </li></ul><ul><li>data is naturally segmented by </li></ul><ul><ul><li>speaker (+gender, +age, +location, ...) </li></ul></ul><ul><ul><li>length and time of writing </li></ul></ul><ul><li>often include additional meta-data </li></ul><ul><li>produced by a large and growing variety of individuals using it for a wide spectrum of purposes </li></ul>

11.
Blogs as corpus data: Cons <ul><li>only one genre (?) </li></ul><ul><li>CMC as a singular mode (?) </li></ul><ul><li>sampling of speakers not representative (?) </li></ul>

12.
Granularity and natural segmentation of data in a blog-based corpus Modes of investigation: 1. Degree of internal variation among all posts by the same blogger 2. Variation between bloggers 3. Variation between groups (gender, age etc) What I had for breakfast this morning xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx xxxx xxx xxx xx xxxx, xxx xxx xx posted 01/01/2007 by Jane Smith post 1 post 2 post 3 post 4 ...

13.
An example for a blog-based corpus <ul><li>self-built corpus for my research project on corporate blogging </li></ul><ul><li>web feeds (RSS and Atom protocols) used to retrieve, store and analyze language data </li></ul><ul><li>implemented TreeTagger for automated part-of-speech tagging </li></ul><ul><li>156 sources </li></ul><ul><li>25,769 posts </li></ul><ul><li>6.6 million words </li></ul>

14.
Application

15.
Individual variation: word class distribution <ul><ul><li>Heather Hamilton (Microsoft) </li></ul></ul>

16.
Individual variation: word class distribution <ul><ul><li>Irving Wladawsky-Berger (IBM) </li></ul></ul>

17.
Individual variation: pronoun use <ul><li>Heather Hamilton (Microsoft) </li></ul><ul><li>1 the DT 2787 </li></ul><ul><li>2 I PP 2723 </li></ul><ul><li>3 to TO 2088 </li></ul><ul><li>4 a DT 1440 </li></ul><ul><li>5 of IN 1324 </li></ul><ul><li>6 and CC 1254 </li></ul><ul><li>7 It PP 1097 </li></ul><ul><li>8 you PP 854 </li></ul><ul><li>9 in IN 818 </li></ul><ul><li>10 that IN 776 </li></ul><ul><li>11 my PP$ 757 </li></ul><ul><li>12 is VBZ 739 </li></ul><ul><li>13 For IN 580 </li></ul><ul><li>14 n't RB 540 </li></ul><ul><li>15 's VBZ 530 </li></ul><ul><li>16 on IN 498 </li></ul><ul><li>17 are VBP 475 </li></ul><ul><li>18 me PP 450 </li></ul><ul><li>19 with IN 431 </li></ul><ul><li>20 this DT 424 </li></ul><ul><li>Irving Wladawsky-Berger (IBM) </li></ul><ul><li>1 the DT 2788 </li></ul><ul><li>2 and CC 1931 </li></ul><ul><li>3 of IN 1571 </li></ul><ul><li>4 to TO 1562 </li></ul><ul><li>5 in IN 1291 </li></ul><ul><li>6 a DT 1047 </li></ul><ul><li>7 is VBZ 695 </li></ul><ul><li>8 I PP 560 </li></ul><ul><li>9 that IN 439 </li></ul><ul><li>10 For IN 434 </li></ul><ul><li>11 It PP 417 </li></ul><ul><li>12 with IN 401 </li></ul><ul><li>13 as IN 390 </li></ul><ul><li>14 are VBP 380 </li></ul><ul><li>15 we PP 359 </li></ul><ul><li>16 on IN 331 </li></ul><ul><li>17 our PP$ 259 </li></ul><ul><li>18 have VHP 253 </li></ul><ul><li>19 that WDT 248 </li></ul>

18.
Individual variation: collocates preceding instances of believe

19.
Gender, age and variation: Schler et al <ul><li>article Effects of Age and Gender on Blogging (AAAI 2006 ) </li></ul><ul><li>all blogs accessible from blogger.com one day in August 2004 </li></ul><ul><li>downloaded each blog that included author-provided indication of gender and at least 200 appearances of common English words </li></ul><ul><li>the full corpus thus obtained included over 71,000 blogs and over 300 million tokens </li></ul><ul><li>used to predict age and gender of bloggers </li></ul>

20.
Gender, age and variation: common words males <ul><li>token male female </li></ul><ul><li>linux 0.53±0.04 0.03±0.01 </li></ul><ul><li>microsoft 0.63±0.05 0.08±0.01 </li></ul><ul><li>gaming 0.25±0.02 0.04±0.00 </li></ul><ul><li>server 0.76±0.05 0.13±0.01 </li></ul><ul><li>software 0.99±0.05 0.17±0.02 </li></ul><ul><li>gb 0.27±0.02 0.05±0.01 </li></ul><ul><li>programming 0.36±0.02 0.08±0.01 </li></ul><ul><li>google 0.90±0.04 0.19±0.02 </li></ul><ul><li>data 0.62±0.03 0.14±0.01 </li></ul><ul><li>graphics 0.27±0.02 0.06±0.01 </li></ul><ul><li>india 0.62±0.04 0.15±0.01 </li></ul><ul><li>nations 0.25±0.01 0.06±0.01 </li></ul><ul><li>democracy 0.23±0.01 0.06±0.01 </li></ul><ul><li>users 0.45±0.02 0.11±0.01 </li></ul><ul><li>economic 0.26±0.01 0.07±0.01 </li></ul>

21.
Gender, age and variation: common words females <ul><li>token male female </li></ul><ul><li>shopping 0.66±0.02 1.48±0.03 </li></ul><ul><li>mom 2.07±0.05 4.69±0.08 </li></ul><ul><li>cried 0.31±0.01 0.72±0.02 </li></ul><ul><li>freaked 0.08±0.01 0.21±0.01 </li></ul><ul><li>pink 0.33±0.02 0.85±0.03 </li></ul><ul><li>cute 0.83±0.03 2.32±0.04 </li></ul><ul><li>gosh 0.17±0.01 0.47±0.02 </li></ul><ul><li>kisses 0.08±0.01 0.28±0.01 </li></ul><ul><li>yummy 0.10±0.01 0.36±0.01 </li></ul><ul><li>mommy 0.08±0.01 0.31±0.02 </li></ul><ul><li>boyfriend 0.41±0.02 1.73±0.04 </li></ul><ul><li>skirt 0.06±0.01 0.26±0.01 </li></ul><ul><li>adorable 0.05±0.00 0.23±0.01 </li></ul><ul><li>husband 0.28±0.01 1.38±0.04 </li></ul><ul><li>hubby 0.01±0.00 0.30±0.02 </li></ul>

22.
Gender, age and variation: common words by age <ul><li>token teens twens thirties </li></ul><ul><li>maths 1.05±0.06 0.03±0.00 0.02±0.01 </li></ul><ul><li>homework 1.37±0.06 0.18±0.01 0.15±0.02 </li></ul><ul><li>bored 3.84±0.27 1.11±0.14 0.47±0.04 </li></ul><ul><li>sis 0.74±0.04 0.26±0.03 0.10±0.02 </li></ul><ul><li>boring 3.69±0.10 1.02±0.04 0.63±0.05 </li></ul><ul><li>awesome 2.92±0.08 1.28±0.04 0.57±0.04 </li></ul><ul><li>mum 1.25±0.06 0.41±0.04 0.23±0.04 </li></ul><ul><li>mad 2.16±0.07 0.80±0.03 0.53±0.04 </li></ul><ul><li>dumb 0.89±0.04 0.45±0.03 0.22±0.03 </li></ul><ul><li>semester 0.22±0.02 0.44±0.03 0.18±0.04 </li></ul><ul><li>apartment 0.18±0.02 1.23±0.05 0.55±0.05 </li></ul><ul><li>drunk 0.77±0.04 0.88±0.03 0.41±0.05 </li></ul><ul><li>beer 0.32±0.02 1.15±0.05 0.70±0.05 </li></ul><ul><li>student 0.65±0.04 0.98±0.05 0.61±0.06 </li></ul><ul><li>album 0.64±0.05 0.84±0.06 0.56±0.08 </li></ul><ul><li>college 1.51±0.07 1.92±0.07 1.31±0.09 </li></ul><ul><li>someday 0.35±0.02 0.40±0.02 0.28±0.03 </li></ul><ul><li>dating 0.31±0.02 0.52±0.03 0.37±0.04 </li></ul>

23.
Gender, age and variation: common words by age (ii) <ul><li>token teens twens thirties </li></ul><ul><li>marriage 0.27±0.03 0.83±0.05 1.41±0.13 </li></ul><ul><li>development 0.16±0.02 0.50±0.03 0.82±0.10 </li></ul><ul><li>campaign 0.14±0.02 0.38±0.03 0.70±0.07 </li></ul><ul><li>tax 0.14±0.02 0.38±0.03 0.72±0.11 </li></ul><ul><li>local 0.38±0.02 1.18±0.04 1.85±0.10 </li></ul><ul><li>democratic 0.13±0.02 0.29±0.02 0.59±0.05 </li></ul><ul><li>son 0.51±0.03 0.92±0.05 2.37±0.16 </li></ul><ul><li>systems 0.12±0.01 0.36±0.03 0.55±0.06 </li></ul><ul><li>provide 0.15±0.01 0.54±0.03 0.69±0.05 </li></ul><ul><li>workers 0.10±0.01 0.35±0.02 0.46±0.04 </li></ul>

24.
Observations

25.
How an individuated approach to corpus linguistics can benefit the field <ul><li>allow us to take into account individual stylistic preference as a source of variation when making generalizations (syntax, semantics, ...) </li></ul><ul><li>allow us to observe specificities of individual production before making blanket label statements about groups (based on gender, social standing etc) </li></ul><ul><li>inverts the idea of system and variation (“how much overlap is there in language use?” vs. “how much variation can our theories account for?”) </li></ul>

26.
Research possibilities? <ul><li>“personal grammar?”, “personal semantics?” </li></ul><ul><li>Construction Grammar (to what degree are constructions individual?) </li></ul><ul><li>variation over the lifetime </li></ul><ul><li>weighing genre, mode and individual variation </li></ul><ul><li>practical applications for forensic linguistics / language profiling </li></ul>

27.
Thank you for listening!

28.
Quantitative Individuated Corpus Linguistics: A Speaker-Centric Approach to Variation Cornelius Puschmann Universität Osnabrück 5 Juni 2007