NoSQL matters in Catchoom Recognition ServiceDavid Arcos                     http://catchoom.com                          ...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in                Recognition Service     Hi! Im David Arcos     - Python/Django developer (>4yr)     - Web ...
NoSQL matters in                 Recognition Service      Catchoom technology recognizes an      object by searching throu...
NoSQL matters in              Recognition Service       Our customers are leaders in Augmented RealityDavid Arcos | @DZPM ...
NoSQL matters in                Recognition Service      Visual Recognition:      “Identify an object in front of the came...
NoSQL matters in            Recognition Service      Examples of recognized objects:      - CD/DVD and book covers      - ...
NoSQL matters in              Recognition Service       Catchoom Recognition Service:       - Cloud-based Visual Recogniti...
NoSQL matters in             Recognition Service       - Small team of 4 developers, doing SCRUMDavid Arcos | @DZPM       ...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in                Recognition Service      Minimum requirements:      - a public API for the final users to ...
NoSQL matters in              Recognition Service      Must be flexible:      - A customer who does Augmented Reality, and...
NoSQL matters in                 Recognition Service      Must be reliable:      - Images or data should never be lost    ...
NoSQL matters in                        Recognition Service      Must be very fast:      “Layar has been using Catchoom’s ...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in               Recognition Service       Technology stack:       - Development: Python, Django, Tornado, G...
NoSQL matters in                  Recognition Service      The Panel:      - typical customer portal:          - manage yo...
NoSQL matters in     Recognition ServiceDavid Arcos | @DZPM          Catchoom | http://catchoom.com | @catchoom
NoSQL matters in     Recognition ServiceDavid Arcos | @DZPM          Catchoom | http://catchoom.com | @catchoom
NoSQL matters in               Recognition Service      Mobile apps:      - for Android, iOS      - use the Visual Recogni...
NoSQL matters in                Recognition Service      Data models:      - Collection: a set of items. Has at least one ...
NoSQL matters in                Recognition Service      Components:      - the platform is highly modular      - “Do one ...
NoSQL matters in                  Recognition Service      - Frontend:      gets the API request      - Extractor:      ex...
NoSQL matters in            Recognition Service      Required NoSQL features:      - key-value storage      - cache      -...
NoSQL matters in            Recognition Service      Required NoSQL features:      - key-value storage      - cache      -...
NoSQL matters in            Recognition Service      Required NoSQL features:      - key-value storage      - cache      -...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in                Recognition Service      Performance:      - Cant afford writing to disk, or querying slow...
NoSQL matters in                  Recognition Service      Scalability:      - Need to scale different components, separat...
NoSQL matters in                  Recognition Service      Unstructured data: query      - A query object has many optiona...
NoSQL matters in                Recognition Service      Unstructured data: metadata      - Metadata is optional and unstr...
NoSQL matters in                Recognition Service      Availability:      - Avoid single points of failure. Replicate ev...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in                Recognition Service      Do real-time calculations:      - Usage statistics         - tota...
NoSQL matters in                   Recognition Service      Sorted Sets:      - To create indexes and filters      - In ex...
NoSQL matters in                Recognition Service      Cache:      - Redis is compatible with memcached API      - Cache...
NoSQL matters in                Recognition Service      Volatile data:      - Redis can set an expiration time for a valu...
NoSQL matters in                Recognition Service      Messages:      - Redis implements pub/sub and lists.          - P...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in               Recognition Service      Django apps compatibility:      - we use Django and several contri...
NoSQL matters in             Recognition Service         1) Introduction         2) What did we need?         3) How we bu...
NoSQL matters in               Recognition Service      Summary:      - We use a combination of SQL and NoSQL      - Using...
NoSQL matters in                   Recognition Service      Recommendations:      - There is no silver bullet      - Use t...
NoSQL matters in                Recognition Service      Thanks for attending!      - Our beta will be ready soon.      Ge...
catchoom.com | @catchoom
NoSQL matters in                Recognition Service      Thanks for attending!      - Our beta will be ready soon.      Ge...
Upcoming SlideShare
Loading in...5
×

NoSQL matters in Catchoom Recognition Service

7,288

Published on

NoSQL matters 2012: http://2012.nosql-matters.org/bcn/speakers/#david_arcos

Catchoom is a technology startup founded in 2011 in Barcelona. Our platform provides visual recognition as a service, it's designed for applications that need to search through a database of images. Catchoom is empowering leading Augmented Reality companies like Layar and Metaio with scalable and fast visual search.

This presentation will explain the architecture of the Catchoom Platform, emphasizing on the advantages of using NoSQL for the specific requirements of visual recognition.

1) Introduction
- David Arcos
- Catchoom
- Customers
- Visual Recognition
- Examples
- Catchoom Recognition Service
- Team

2) What did we need?
- Minimum requirements
- Flexible
- Reliable
- Fast

3) How we build it
- Technology stack
- The Panel
- Mobile apps
- Data models
- Components
- NoSQL features

4) Advantages of NoSQL
- Performance
- Scalability
- Unstructured data: query
- Unstructured data: metadata
- Availability

5) Cool uses of NoSQL
- Real time calculations (stats, monitoring, rate limiting)
- Sorted sets (indexing)
- Cache
- Volatile data
- Messages (pub/sub, push/pop)

6) Limits
- Django apps compatibility

7) Conclusion
- Summary
- Recommendations
- Thanks for attending
- Q&A

http://catchoom.com

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,288
On Slideshare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
26
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Looks easy?
  • (timestamps, the image index, debug info...)
  • Efficiency Totals, per month, per day, per image, per item, per collection Response times, queue size Redis is compatible with memcached API Avoid hitting the db
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • Efficiency No need to consolidate
  • NoSQL matters in Catchoom Recognition Service

    1. 1. NoSQL matters in Catchoom Recognition ServiceDavid Arcos http://catchoom.com catchoom.com | @catchoom
    2. 2. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    3. 3. NoSQL matters in Recognition Service Hi! Im David Arcos - Python/Django developer (>4yr) - Web backend, distributed systems, databases, scalability, security - Team leader at Catchoom - You can follow me at @DZPMDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    4. 4. NoSQL matters in Recognition Service Catchoom technology recognizes an object by searching through a large collection of images in a fraction of a second. Catchoom targets application developers and integrators.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    5. 5. NoSQL matters in Recognition Service Our customers are leaders in Augmented RealityDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    6. 6. NoSQL matters in Recognition Service Visual Recognition: “Identify an object in front of the camera by comparing it to a huge collection of reference images”David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    7. 7. NoSQL matters in Recognition Service Examples of recognized objects: - CD/DVD and book covers - Newspapers and magazines - Logos and brands - Posters - Packaged goods - Monuments and placesDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    8. 8. NoSQL matters in Recognition Service Catchoom Recognition Service: - Cloud-based Visual Recognition (SaaS) - RESTful API to integrate - “Add VR features to your app/platform”David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    9. 9. NoSQL matters in Recognition Service - Small team of 4 developers, doing SCRUMDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    10. 10. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    11. 11. NoSQL matters in Recognition Service Minimum requirements: - a public API for the final users to perform Visual Recognition - a private API for the customer to manage the Collections and get statistics - a nice website for the customer, providing the functionality of both APIsDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    12. 12. NoSQL matters in Recognition Service Must be flexible: - A customer who does Augmented Reality, and needs a 3D model (binary format) in the item - Another one who needs just the item id - Our data model needs to allow everything (structured and unstructured data)David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    13. 13. NoSQL matters in Recognition Service Must be reliable: - Images or data should never be lost - Avoid single points of failure - We need redundancyDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    14. 14. NoSQL matters in Recognition Service Must be very fast: “Layar has been using Catchoom’s Visual Search technology since the launch of Layar Vision, allowing users to quickly view the AR content placed on top of images by just pointing their camera to the image. We’ve benchmarked Catchoom’s technology in 2011 against 3 of their main competitors and found they had the best results both on speed and on successful matches (including lowest false positives)” Dirk Groten – CTO of LayarDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    15. 15. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    16. 16. NoSQL matters in Recognition Service Technology stack: - Development: Python, Django, Tornado, Gevent - Deployed using: Supervisord, Nginx, gunicorn, Fabric - AWS: EC2, S3, ELBDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    17. 17. NoSQL matters in Recognition Service The Panel: - typical customer portal: - manage your Collections, run Visual Recognition - get usage statistics - and configure the payment method :)David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    18. 18. NoSQL matters in Recognition ServiceDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    19. 19. NoSQL matters in Recognition ServiceDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    20. 20. NoSQL matters in Recognition Service Mobile apps: - for Android, iOS - use the Visual Recognition API - the code will be publishedDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    21. 21. NoSQL matters in Recognition Service Data models: - Collection: a set of items. Has at least one token. - Item: has at least one Image. Has metadata. - Image: you want several images if the item has different sides, logos, flavours... - Token: for authenticating the requests.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    22. 22. NoSQL matters in Recognition Service Components: - the platform is highly modular - “Do one thing, and do it well” - they pass json messages - optimized hardware settingsDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    23. 23. NoSQL matters in Recognition Service - Frontend: gets the API request - Extractor: extracts the visual points - Collector: message exchange - Searcher: looks for matchesDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    24. 24. NoSQL matters in Recognition Service Required NoSQL features: - key-value storage - cache - message lists - message pub/sub - real-time analysis What servers have we chosen?David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    25. 25. NoSQL matters in Recognition Service Required NoSQL features: - key-value storage - cache - message lists - message pub/sub - real-time analysisDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    26. 26. NoSQL matters in Recognition Service Required NoSQL features: - key-value storage - cache - message lists - message pub/sub - real-time analysis - and Filesystem:David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    27. 27. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    28. 28. NoSQL matters in Recognition Service Performance: - Cant afford writing to disk, or querying slow databases - Using Redis, everything stays on memory - One V.R. query takes just 300 msDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    29. 29. NoSQL matters in Recognition Service Scalability: - Need to scale different components, separately - Load balancing using Redis Lists: BLPOP: Remove and get the first element in a list, or block until one is available - But focus on the bottlenecks!David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    30. 30. NoSQL matters in Recognition Service Unstructured data: query - A query object has many optional parameters - each component can add/remove fields dynamically - schema change between versions - Cant fit in a SQL table - We model the query in Redis as a jsonDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    31. 31. NoSQL matters in Recognition Service Unstructured data: metadata - Metadata is optional and unstructed, can be from a json to a binary blob - Cant fit in a SQL table, and would be too slow - Serve the data from Redis, and use S3 as a backup - Warning: in the future, if we have huge metadata files, Redis will get out of memory. Well improve this approachDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    32. 32. NoSQL matters in Recognition Service Availability: - Avoid single points of failure. Replicate everything! - Replicating a SQL server is painful - Redis instances configured as Master/Slave - When the master dies: - promote a slave to be the new master - reconfigure the other slaves to use this new master - Redis Sentinel does this (beta)David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    33. 33. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    34. 34. NoSQL matters in Recognition Service Do real-time calculations: - Usage statistics - total, monthly, daily, hourly - per image, item or collection - Metric monitoring for internal use - response times, queue size, etc - QoS: enforce rate limiting - max hits per minuteDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    35. 35. NoSQL matters in Recognition Service Sorted Sets: - To create indexes and filters - In example, “Most recognized images” (sorted by hits) - Updating the Sorted Set, no need to reconsolidate: ZADD Add one or more members to a sorted set, or update its score if it already existsDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    36. 36. NoSQL matters in Recognition Service Cache: - Redis is compatible with memcached API - Cache everything: - Sessions, metadata, etc - ...although the website is internal: no bottleneck here - Better focus on optimizing other stuff!David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    37. 37. NoSQL matters in Recognition Service Volatile data: - Redis can set an expiration time for a value - Very easy for: - implementing timeouts - removing old queries - adding temporary cappingDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    38. 38. NoSQL matters in Recognition Service Messages: - Redis implements pub/sub and lists. - Publish/Subscribe to a channel - all components get the message - use it for monitoring - List: push/pop messages - only one component gets the message - use the blocking versions for load balancingDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    39. 39. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    40. 40. NoSQL matters in Recognition Service Django apps compatibility: - we use Django and several contrib and external apps. - (“Standing in the shoulder of giants”) - but no support for NoSQL in Django ORM - dropping SQL is not an option! - we use MySQL. South migrations.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    41. 41. NoSQL matters in Recognition Service 1) Introduction 2) What did we need? 3) How we build it 4) Advantages of NoSQL 5) Cool uses of NoSQL 6) Limits 7) ConclusionDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    42. 42. NoSQL matters in Recognition Service Summary: - We use a combination of SQL and NoSQL - Using NoSQL was necessary to meet the requirements - There are a lot of different uses for NoSQLDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    43. 43. NoSQL matters in Recognition Service Recommendations: - There is no silver bullet - Use the best tool for each task - But avoid unneeded complexity! - Try Redis. Dont do a migration, just add it to your stackDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    44. 44. NoSQL matters in Recognition Service Thanks for attending! - Our beta will be ready soon. Get a free trial at http://catchoom.com - Contact me at david.arcos@catchoom.com - Questions?David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    45. 45. catchoom.com | @catchoom
    46. 46. NoSQL matters in Recognition Service Thanks for attending! - Our beta will be ready soon. Get a free trial at http://catchoom.com - Contact me at david.arcos@catchoom.com - Questions?David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×