It is all about data.
Having the right data at the right time might make the difference between you and your competitor. Google can show you just what it can catch. If you know where to find the data of your interest, let's go deeper and roll your own web crawler framework.
Taking the advantage of the latest cool technologies I will show you how to build your distributed web crawler based on Redis and Mongo