Download as PDF, PPTX
Uploaded bymosa siru
ニュースパスのクローラーアーキテクチャとマイクロサービス
This document summarizes Mosa Siru's engineering work at DeNA and Gunosy. It describes Gunosy's RSS/Atom feed crawler system which includes components like a job queue, fetcher, parser, updater, content generator, and indexer. It also discusses technologies used like Python, Celery, MySQL, S3, Elasticsearch. The system is designed to efficiently parse, store, generate content and index feeds at scale.
In this document
Powered by AI
Overview of crawler operations presented by @mosa_siru, indicating a technical topic in the scope.
@mosa_siru's experience as an engineer at various companies including DeNA and Gunosy, highlighting their role as a CTO.
Brief mention of KDDI in June 2016, setting context for a technical discussion.
Discussion on content syndication standards like RSS2.0 and XML formats utilized in feeds, emphasizing GunosyFeed Ver. 2.
Introduction to Job Queue using Python Celery, detailing Celery Flower for task management.
Technical explanation of a scheduler for managing fetch requests and the fetching process.
Details on parsing XML data using Python feedparser, handling data updates in databases.
Mechanics of database updates using methods like 'insert on duplicate key update' for managing feed data.
Overview of generating content, including HTML structures and resource paths for images and styles.
Detailing how enclosure fetching works, managing URLs and hashes to retrieve media.
Discussion around using an HTTP Request Proxy (Squid) for managing requests and responses.
Image Cropper functionalities including face detection and quality enhancements via Akamai API.
Details on indexing content and classification processes using Indexer API and Classifier API.
Overview of article database interactions through various APIs for reading and writing articles.
Discussion on cache invalidation strategies for effectively managing temporary stored content.
Mention of XML handling capabilities within the Crawler API for structured data acquisition.
Final demonstration of XML integration and its functionalities within the presentation context.
Wrap-up mentioning Gunosy, recognizing the author's contributions and setting up interactions.
More Related Content
![[AKIBA.AWS] AWS Elemental MediaConvertから学ぶコーデック入門](https://cdn.slidesharecdn.com/ss_thumbnails/akibaaws4up-180130002606-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DevFest Strasbourg 2025] - NodeJs Can do that !!](https://cdn.slidesharecdn.com/ss_thumbnails/devfeststrasbourg2025-nodejscandothat-251127142731-da65b6fd-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...](https://cdn.slidesharecdn.com/ss_thumbnails/md-mobileengineerandsoftwareengineerarewestillrelevantsidiqpermana-251127010650-55224ef1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-buildingaisystemsthatusersandcompanieslove-251124030845-038f7732-thumbnail.jpg?width=640&height=640&fit=bounds)
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
ニュースパスのクローラーアーキテクチャとマイクロサービス
- 1.
- 2.
- 3. @mosa_siru as engineer •DeNA • Gunosy • CTO
- 4.
- 5. • 2016/06 KDDI • •
- 7.
- 9. • • • RSS • t2small2 ( ) • • • •
- 10. • XML •1 • • DB s3
- 11. • RSS2.0, Atom,RDF GunosyFeed Ver. 2 •
- 13.
- 14. • JobQueue (PythonCelery) • • • • Celery Flower
- 15.
- 16.
- 17. • 30 • Fetcherenqueue • • HTTP Scheduler
- 18.
- 19. • XML • XMLhash hash • XML s3 Up Parser enqueue • s3 Fetcher
- 20.
- 21. • XML parsePython feedparser • RSS2.0, Atom, RDF parse • XML • ( ) • • etc… Parser
- 22. • parse DBs3 up • DB insert/update/delete • update update • mysql insert on duplicate key update update (1 1 update ) Updater
- 23. • url orguid • hash DB hash • url feed, title, ( guid …)
- 24.
- 25. • • HTML (js) • URL URL (./hoge.html ) • css path • • (img ) s3 URL • hash s3 Content Generator
- 26.
- 27. • s3 DBURL • • hash s3 Enclosure Fetcher
- 28. • HTTP RequestProxy (Squid) • Response Header • IP (Elastic IP) HTTP Proxy
- 29.
- 30. • Microsoft FaceDetection API • crop ImageCropper ///
- 31. Akamai Image Converter •Akamai URL • Smalllight • • • •
- 32.
- 33.
- 34.
- 35. Title Break Calculator • •API ( )
- 36.
- 37. Indexer • index IndexerAPI • • Indexer API ElasticSearch
- 38.
- 39.
- 40.
- 41.
- 42. • • Article DBwrite/read • API • Article API • Article DB read Article API
- 44.
- 45. Cache Invalidation • ArticleAPI memcached invalidation(memd ) • URL, URL Akamai query string https://…/mychild.png?crop=200:200;220,210&output-quality=10 &t=1468557607
- 46.
- 47.
- 49. XML • XML • XML • Crawler API
- 50.
- 52.
- 53. • XML XML • Slack •
- 54.
- 56.
- 57.