Downloaded 109 times
![But we pay the price.
Example: search engine improvements in batch
evaluations don’t always predict real user benefits.
[Hersh et al, 2000] Do Batch and User Evaluations Give the Same Results?
[Turpin & Hersh, 2001] Why Batch and User Evaluations do not Give the Same Results
[Turpin, Scholer, 2006] User Performance versus Precision Measures for Simple
Search Tasks
But also see…
[Smucker & Jethani, 2010] Human Performance and Retrieval Precision Revisited](https://image.slidesharecdn.com/webscience-howisitdifferent-140624100930-phpapp01/85/Web-science-How-is-it-different-32-320.jpg)
Uploaded byDaniel Tunkelang
Web science - How is it different?
The document discusses how big data and web science have transformed the scientific method by making experimentation cheaper and more scalable. While traditional methods of hypothesis testing still hold value, the shift to faster, data-driven approaches presents both opportunities and challenges, particularly in recognizing human biases in experimentation. Ultimately, it emphasizes that the scientific method remains relevant, though its economics have shifted dramatically.
More Related Content
Similar to Web science - How is it different?
Web science - How is it different?
- 1. Daniel Web Science: Howis it different? Daniel Tunkelang Head of Query Understanding
- 3. tl;dr: The scientific methodis alive and well. Big data has just changed the economics.
- 4. How have theweb and big data changed science? Let’s ask some of the experts.
- 5. “You have tokiss a lot of frogs to find one prince. So how can you find your prince faster? By finding more frogs and kissing them faster and faster.” Mike Moran Do It Wrong Quickly: How the Web Changes the Old Marketing Rules, 2007 Cited by Kohavi in Online Controlled Experiments at Large Scale, 2013
- 6. Web Science =faster, cheaper experiments.
- 7. “The cost ofexperimentation is now the same or less than the cost of analysis. You can get more value…by doing a quick experiment than from doing a sophisticated analysis.” Michael Schrage Value-Creation, Experiments, and Why IT Does Matter, 2010
- 8. Web Science =more experiments, less analysis?
- 9. “with massive data,this approach to science — hypothesize, model, test — is becoming obsolete… Petabytes allow us to say: "Correlation is enough." We can stop looking for models…analyze the data without hypotheses…throw the numbers into the biggest computing clusters the world…and let… algorithms find patterns where science cannot.” Chris Anderson The End of Theory, 2008
- 10. RIP Scientific Method 1600 BCE – late20th century Killed by Big Data ?
- 11.
- 12.
- 13. What makes itscience?
- 14.
- 15.
- 16.
- 17. The scientific methodstill works today. What’s changed is the economics.
- 18.
- 19.
- 20. It’s the economy,science. Yesterday Experiments are expensive, choose hypotheses wisely. Today Experiments are cheap, do as many as you can!
- 21. What about WebScience?
- 22.
- 23. Google: 20k search experimentsper year
- 24.
- 25. The Myth ofInsight
- 26. Scientists gain insight bystaring at data.
- 27. Big data toolsimprove data exploration.
- 28. In hypothesis generation, quantitytrumps quality.
- 29. Except when itdoesn’t.
- 31. Easier to analyzedata than research humans.
- 32. But we paythe price. Example: search engine improvements in batch evaluations don’t always predict real user benefits. [Hersh et al, 2000] Do Batch and User Evaluations Give the Same Results? [Turpin & Hersh, 2001] Why Batch and User Evaluations do not Give the Same Results [Turpin, Scholer, 2006] User Performance versus Precision Measures for Simple Search Tasks But also see… [Smucker & Jethani, 2010] Human Performance and Retrieval Precision Revisited
- 33. When local optimizationis cheap, you neglect the rest.
- 34. To summarize: howis web science different? • Online testing is cheaper and scalable. • Data exploration tools make hypothesis generation cheaper and easier. • But the experiments that are easy and cheap aren’t always the most valuable. • Easy to forget our biases as scientists.
- 35. Take-Aways • The scientificmethod is alive and well. Big data has just changes the economics. • Cheaper hypothesis testing and generation has already been transformative.That’s why big data matters. • But we neglect the human side of scientific experimentation at our peril.
- 36.
Editor's Notes
- #19 James Lind thought that scurvy was due to putrefaction of the body which could be helped by acids, and thus included a dietary supplement of an acidic quality in the experiment. This began after two months at sea when the ship was afflicted with scurvy. He divided twelve scorbutic sailors into six groups of two. They all received the same diet but, in addition, group one was given a quart of cider daily, group two twenty-five drops of elixir of vitriol (sulfuric acid), group three six spoonfuls of vinegar, group four half a pint of seawater, group five received two oranges and one lemon, and the last group a spicy paste plus a drink of barley water. The treatment of group five stopped after six days when they ran out of fruit, but by that time one sailor was fit for duty while the other had almost recovered. Apart from that, only group one also showed some effect of its treatment.