ニュースパスのクローラーアーキテクチャとマイクロサービス
•
16 likes•10,628 views
This document summarizes Mosa Siru's engineering work at DeNA and Gunosy. It describes Gunosy's RSS/Atom feed crawler system which includes components like a job queue, fetcher, parser, updater, content generator, and indexer. It also discusses technologies used like Python, Celery, MySQL, S3, Elasticsearch. The system is designed to efficiently parse, store, generate content and index feeds at scale.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to ニュースパスのクローラーアーキテクチャとマイクロサービス
Similar to ニュースパスのクローラーアーキテクチャとマイクロサービス (20)
Recently uploaded
Recently uploaded (20)
ニュースパスのクローラーアーキテクチャとマイクロサービス
- 2. @mosa_siru ( ) • • 2
- 3. @mosa_siru as engineer • DeNA • Gunosy • CTO
- 5. • 2016/06 KDDI • •
- 7. • • 匠 • • • • 匠
- 9. • • • RSS • t2small 2 ( ) • • • •
- 10. • XML • 1 • • DB s3
- 11. • RSS2.0, Atom, RDF GunosyFeed Ver. 2 •
- 14. • JobQueue (Python Celery) • • • • Celery Flower
- 15. Celery Flower
- 16. Scheduler
- 17. • 30 • Fetcher enqueue • • HTTP Scheduler
- 18. Fetcher
- 19. • XML • XML hash hash • XML s3 Up Parser enqueue • s3 Fetcher
- 20. Parser(Updater)
- 21. • XML parse Python feedparser • RSS2.0, Atom, RDF parse • XML • ( ) • • etc… Parser
- 22. • parse DB s3 up • DB insert/update/delete • update update • mysql insert on duplicate key update update (1 1 update ) Updater
- 23. • url or guid • hash DB hash • url feed, title, ( guid …)
- 25. • • HTML (js ) • URL URL (./hoge.html ) • css path • • (img ) s3 URL • hash s3 Content Generator
- 27. • s3 DB URL • • hash s3 Enclosure Fetcher
- 28. • HTTP Request Proxy (Squid) • Response Header • IP (Elastic IP) HTTP Proxy
- 29. Image Cropper
- 30. • Microsoft FaceDetection API • crop Image Cropper ///
- 31. Akamai Image Converter • Akamai URL • Smalllight • • • •
- 35. Title Break Calculator • • API ( )
- 36. Indexer
- 37. Indexer • index Indexer API • • Indexer API ElasticSearch
- 38. Classifier
- 42. • • Article DB write/read • API • Article API • Article DB read Article API
- 44. • • • API • •
- 45. Cache Invalidation • Article API memcached invalidation(memd ) • URL, URL Akamai query string https://…/mychild.png?crop=200:200;220,210&output-quality=10 &t=1468557607
- 47. • • • URL • • •
- 49. XML • XML • XML • Crawler API
- 50. XML
- 52. DEMO
- 53. • XML XML • Slack •
- 54. • XML •
- 56. • • • • • 匠