Be the first to like this
Trunk.ly launched as a half baked product when Yahoo announced its plan to "sunset" delicious. Within 2 days, 5000 users sign up with the service. In order to deal with the huge demands, we spun off 10+ EC2 instances to crunch through the links. Alex rewrote the crawler architecture using gevent and redis. Reducing the total number of servers to 2. Saving $3000 per month. This talk is about the framework and architecture used in the rewrite.