Your SlideShare is downloading. ×
0
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

#bigdata in Communication Science
Some ex...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

1 What’s big data?
2 Some examples

Rare ...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

What’s big data?

#bigdata

Damian Trilli...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

What’s big data?

No definition, but . . ....
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

What’s big data?

Some sources
• Social N...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

It’s out there!
You only have to collect ...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Some examples

#bigdata

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

A recent master thesis

Rare...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

A recent master thesis

Rare...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

A recent master thesis

Rare...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

So you’d better collect ever...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

So you’d better collect ever...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

So you’d better collect ever...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

#bigdata

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Rare events

It’s just one line of code!
...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

A recent bachelor thesis
...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

A recent bachelor thesis
...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

A recent bachelor thesis
...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

So you’d better think abo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

So you’d better think abo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

So you’d better think abo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

So you’d better think abo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

#bigdata

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Tone in tweets

#bigdata

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Counting words and n-grams

How often are...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Counting words and n-grams

How often are...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Counting words and n-grams

How often are...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Counting words and n-grams

Again, just o...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Counting words and n-grams

trigrams in O...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Network analysis

Another approach

Netwo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Network analysis

Another approach

Netwo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Network analysis

Another approach

Netwo...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Network analysis

#bigdata

Damian Trilli...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems

#bigdata

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems
You sometimes depend entirely on...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems
You sometimes depend entirely on...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems
You sometimes depend entirely on...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems
You sometimes depend entirely on...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Problems

Not sure if this a problem or a...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

A glimpse in the kitchen

#bigdata

Damia...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

What I’m doing right now

Analyzing #tvdu...
What’s big data?

#bigdata

Some examples

Problems

A glimpse in the kitchen

Questions?

Damian Trilling
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Something you can use?
1 What’s big data?...
What’s big data?

Some examples

Problems

A glimpse in the kitchen

Questions?

Vragen of opmerkingen?

Damian Trilling
d...
Upcoming SlideShare
Loading in...5
×

Guestlecture on #bigdata

1,115

Published on

A guest lecture in the Master elective "The Blind Spot: Tracking Young Media Users" by Susanne Baumgartner

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,115
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Guestlecture on #bigdata"

  1. 1. What’s big data? Some examples Problems A glimpse in the kitchen Questions? #bigdata in Communication Science Some examples from research by me and my students Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net Afdeling Communicatiewetenschap Universiteit van Amsterdam October 2013 #bigdata Damian Trilling
  2. 2. What’s big data? Some examples Problems A glimpse in the kitchen Questions? 1 What’s big data? 2 Some examples Rare events Tone in tweets Counting words and n-grams Network analysis 3 Problems 4 A glimpse in the kitchen 5 Questions? #bigdata Damian Trilling
  3. 3. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? #bigdata Damian Trilling
  4. 4. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? No definition, but . . . • Existing data • Too big to code manually • Sometimes also too big to handle with normal tools • New research questions • Call to revisit the relationship between theory and empirical research #bigdata Damian Trilling
  5. 5. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What’s big data? Some sources • Social Network Sites • RSS-feeds • Databases • Scraping text from the web • ... #bigdata Damian Trilling
  6. 6. What’s big data? Some examples Problems A glimpse in the kitchen Questions? It’s out there! You only have to collect it. #bigdata Damian Trilling
  7. 7. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Some examples #bigdata Damian Trilling
  8. 8. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events #bigdata Damian Trilling
  9. 9. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events Imagine you want to analyze some very rare content. #bigdata Damian Trilling
  10. 10. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events A recent master thesis Rare events Imagine you want to analyze some very rare content. Normal sampling won’t work, that’s for sure. #bigdata Damian Trilling
  11. 11. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  12. 12. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites We collected all articles from nine news sites during a period of two months, resulting in a database with 74.000 articles. Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  13. 13. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events So you’d better collect everything first Getting all news coverage from Dutch news sites We collected all articles from nine news sites during a period of two months, resulting in a database with 74.000 articles. In a second step, we filtered those articles containing specific keywords. Those 292 articles where then manually coded. Pöll, B. (2013). Social media: new sources, new profession? A content analysis of the use of social media as a source for journalists in online news articles. Master Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  14. 14. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events #bigdata Damian Trilling
  15. 15. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Rare events It’s just one line of code! url.txt http://www.gmx.at/themen/wissen/mensch/108g5xi-baeuerlich-schiefe-zaehne http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/408g740-fuermannbittet-um-verzeihung http://www.gmx.at/themen/nachrichten/aufruhr-arabien/268g70u-regierungwill-zuruecktreten http://www.gmx.at/themen/nachrichten/panorama/828g54y-neues-zur-klagegegen-republik http://www.gmx.at/themen/nachrichten/panorama/968g72s-millionstrafewegen-oelpest http://www.gmx.at/themen/unterhaltung/klatsch-tratsch/368g6yc-keinbabybauch-nur-fast-food ... ... ... #bigdata wget-commando wget -i urls.txt Damian Trilling
  16. 16. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets #bigdata Damian Trilling
  17. 17. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets Imagine you want to know something about someone’s behavior on twitter. Or how a specific topic is discussed on Twitter. #bigdata Damian Trilling
  18. 18. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets A recent bachelor thesis Tone in tweets Imagine you want to know something about someone’s behavior on twitter. Or how a specific topic is discussed on Twitter. Do you really want to go through thousands of tweets by hand? #bigdata Damian Trilling
  19. 19. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  20. 20. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  21. 21. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. We used a Python-script to check which type of words were used to refer to opponents. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  22. 22. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets So you’d better think about automating your coding Finding out how negative or positive politicians are towards there opponents We took lists with positive and negative words and with a politician’s opponents. We used a Python-script to check which type of words were used to refer to opponents. For further analysis, the results where imported in SPSS. Schut, L. (2013). Verenigde Staten vs. Verenigd Koningrijk: Een automatische inhoudsanalyse naar verklarende factoren voor het gebruik van positive campaigning en negative campaigning door vooraanstaande politici en politieke partijen op Twitter. Bachelor Thesis, Universiteit van Amsterdam. #bigdata Damian Trilling
  23. 23. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets #bigdata Damian Trilling
  24. 24. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Tone in tweets #bigdata Damian Trilling
  25. 25. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams #bigdata Damian Trilling
  26. 26. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams Imagine you want to know which words or expressions dominate a discourse . #bigdata Damian Trilling
  27. 27. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams How often are specific expressions used? Counting words and n-grams Imagine you want to know which words or expressions dominate a discourse . There are plenty of possibilities to get an answer within minutes, here’s one: #bigdata Damian Trilling
  28. 28. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams Again, just one or two lines of code! For example with STATA • Install the package wordscore (net install http://www.tcd.ie/Political_Science/wordscores/wordscores) • voor wordcounts: wordfreq /home/dami/texts/lab92.txt /home/dami/texts/lab97.txt • voor ngrams (trigrams in dit geval): phrasefreq 3 lab92.txt lab97.txt #bigdata Damian Trilling
  29. 29. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Counting words and n-grams trigrams in Obama-Tweets #bigdata Damian Trilling
  30. 30. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis #bigdata Damian Trilling
  31. 31. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis Imagine you want to know who talks to whom and how networks are interconnected . #bigdata Damian Trilling
  32. 32. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis Another approach Network analysis Imagine you want to know who talks to whom and how networks are interconnected . Use a tool like NodeXL or Gephi! #bigdata Damian Trilling
  33. 33. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Network analysis #bigdata Damian Trilling
  34. 34. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems #bigdata Damian Trilling
  35. 35. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties #bigdata Damian Trilling
  36. 36. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) #bigdata Damian Trilling
  37. 37. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) • It’s rather easy to get (up to 3200) tweets from a specific user (e.g., allmytweets.net), but if you want to capture a #hashtag, you have to record it live #bigdata Damian Trilling
  38. 38. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems You sometimes depend entirely on commercial parties • Services can shut down (GoogleReader) or change their API (Twitter) • It’s rather easy to get (up to 3200) tweets from a specific user (e.g., allmytweets.net), but if you want to capture a #hashtag, you have to record it live • Twitter doesn’t give you all tweets, but just about 1% (+ a bunch of other limits) #bigdata Damian Trilling
  39. 39. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Problems Not sure if this a problem or a great opportunity. . . You cannot rely (only) on ready-made software but shout get ready to use tools like bash-scripts, grep, python, . . . (Which can be fun!) #bigdata Damian Trilling
  40. 40. What’s big data? Some examples Problems A glimpse in the kitchen Questions? A glimpse in the kitchen #bigdata Damian Trilling
  41. 41. What’s big data? Some examples Problems A glimpse in the kitchen Questions? What I’m doing right now Analyzing #tvduell • 570.000 tweets • Identifyig clusters of nouns, verbs and adjectives • Assigning positivity and negativity scores to tweets • See if they can be interpreted as frames ⇒How are Merkel and Steinbrück framed on the Second Secreen during the debate? #bigdata Damian Trilling
  42. 42. What’s big data? #bigdata Some examples Problems A glimpse in the kitchen Questions? Damian Trilling
  43. 43. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Something you can use? 1 What’s big data? 2 Some examples Rare events Tone in tweets Counting words and n-grams Network analysis 3 Problems 4 A glimpse in the kitchen 5 Questions? #bigdata Damian Trilling
  44. 44. What’s big data? Some examples Problems A glimpse in the kitchen Questions? Vragen of opmerkingen? Damian Trilling d.c.trilling@uva.nl @damian0604 www.damiantrilling.net #bigdata Damian Trilling
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×