author: Kamil Durski, Senior Ruby Developer at Polcode
Twitter: @kdurski Github: github.com/kdurski
Ruby has built-in support for threads yet it’s barely used, even in situations where it could be very handy, such as crawling the web. While it can be a pretty slow process, the majority of the time is spent on waiting for IO data from the remote server. This is the perfect case to use threads.
You can read more about multi-threaded web crawler in Ruby here: http://www.polcode.com/en/multi-threaded-web-crawler-in-ruby/