Presentation about scraping in general, it's techinques and my personal experience with it.

1. Attacking fire with fire
Or how to get an API from any website

2. I am Danielius Visockas
#givingBackToCommunity
Salut!

3. Web harvesting

4. Web harvesting
Go to a
page
Extract the
data
Download a
document

5. Basic diagram of web harvesting

6. Fundamental metrics
◉ Freshness
◉ Age

8. Revisiting policy
Constant Based on freshness

9. “
Edward Coffman et. al. proposed that
a crawler must minimize the fraction
of time pages remain outdated.

10. Aaah, easy

11. curl -i https://delfi.lt

12. No SSL...

13. curl -i http://delfi.lt

14. Doesn’t work....

15. Let’s try mobile
curl -i http://m.delfi.lt

16. ……………….
<script
type="text/javascript">(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(argume
ts)},i[r].l=1*new
Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'scr
pt','//www.google-analytics.com/analytics.js','ga');ga('create','UA-2428893-5','auto');var
__ae=document.getElementsByClassName('delfi-author-name')[0]||document.getElementsByClassName('article-author-name')[0];if('undefi
ed' !== typeof __ae){var
au=__ae.textContent;au=au.replace(/[,;].*/g,'');au=au.replace(/^s+|s+$/g,'');au=au.toLowerCase();ga('set','dimension1',au);}if(m
= navigator.userAgent.match(/Delfi/([0-9.]+)/)){var ua='Other';if(/ip(hone|ad|od)/i.test(navigator.userAgent))ua='iOS App';else
if(/android/i.test(navigator.userAgent))ua='Android App';else if(/(windows|msie)/i.test(navigator.userAgent))ua='Windows
App';ga('set','dimension2',ua);}else if(/FBAV//.test(navigator.userAgent))ga('set','dimension2','FBWV');else
ga('set','dimension2','Browser');ga('set','dimension3',''+(window.__dabd && window.__dabd()));ga('send','pageview');
</script>
<script type="text/javascript">var
t=window.location.hostname.split('.').reverse();if(window._dct)_dct({s:'delfi/mobile',d:'t.'+t[1]+'.'+t[0]});</script>
<script type="text/javascript">
var
__ae=document.getElementsByClassName('delfi-author-name')[0]||document.getElementsByClassName('article-author-name')[0],au='',_sf_
sync_config = {};
if('undefined' !== typeof __ae){var
au=__ae.textContent;au=au.replace(/,.*/g,'');au=au.replace(/^s+|s+$/g,'');au=au.toLowerCase();}
_sf_async_config.uid=46335;_sf_async_config.domain='delfi.lt';_sf_async_config.sections='m.delfi';_sf_async_config.authors=au;_sf_
sync_config.useCanonical=true;
(function(){function loadChartbeat(){window._sf_endpt=(new Date()).getTime();var
e=document.createElement('script');e.setAttribute('language', 'javascript');e.setAttribute('type',
'text/javascript');e.setAttribute('src', '//static.chartbeat.com/js/chartbeat.js');document.body.appendChild(e);}
var oldonload=window.onload; window.onload=(typeof window.onload != 'function') ? loadChartbeat : function() { oldonload();
loadChartbeat(); };
})();
</script>
<script

17. This looks familiar
Let’s use regex and it should be fine

18. Overengineering

19. Basic techniques
Pick a right tool for the job

20. One-time
Your computer is on
Two ways to harvest
Automated
Can be done in a server

21. Copy and paste

22. Client-side scripting

23. Extensions and bookmarks!

24. Online scrapers

25. The fun part
Automated scraping

26. “
Don’t forget to watch the network tab

27. Fetching of websites

28. Extraction of data
Cheerio

29. But then it all changed
When fire nation attacked

30. I found a girl in Kaunas...

31. 7 seconds
Traukiniobilietas.lt response time

32. Thats

33. Five

34. Seconds

35. More

36. Than

37. It

38. Takes

39. To

40. Say

41. Seven

42. Seconds

43. Screenshot
Traukiniobilietas.lt didn’t load...

44. So I decided to learn React
And built an app that helps you to find trips

45. Want big impact?
Use big image.
How do I get the Data?!

46. Headless browsers

47. Brings together the best
Of two worlds

48. I used Casper.js
◉ Runs on PhantomJS
◉ Resource intensive
◉ Can replicate everything
◉ Takes a bit longer
◉ DoS’ed traukiniobilietas…
◉ Works

52. “
So basically
You have to pick
The right tool for the job
#noFreeLunchTheory

53. Legal stuff

54. Security
CAPTCHAS and friends...

55. Interesting ideas
◉ Visual scraping using Machine Learning
◉ Macros + Casper.js (github.com/dvisockas/scrape)

56. Please ask questions!
Thank you!
And if someone from TRAFI could help me with traveling salesman..