The Internet in Database: A Cassandra Use Case
•
1 like•623 views
The document discusses using Cassandra and Solr together to build a search engine capable of handling large amounts of unstructured data from the internet. It outlines the goals of collecting and normalizing web data at scale and making it searchable. It then describes the challenges faced with memory, speed, space and data representation when implementing this system and potential solutions and future work.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Internet in Database: A Cassandra Use Case
Similar to The Internet in Database: A Cassandra Use Case (20)
Recently uploaded
Recently uploaded (20)
The Internet in Database: A Cassandra Use Case
- 1. The Internet in a Database A Cassandra Use Case
- 2. Data on the Web DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● 48 billion pages on the Internet ● 56 million GB of data ● Incredibly powerful connections ● 70% of useful data is unstructured ● User generated data + facts
- 3. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Too Much Data…
- 4. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Modern search engines ○ Unstructured data ○ Unconnected data ○ Unnormalized data Search
- 5. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Goals ○ Collect vast amounts of data through web crawling ○ Normalize and deduplicate data ○ Make it searchable and meaningful
- 6. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Speed ● Scale ● Adaptable Needs
- 7. ● Very fast ○ Log-structured storage ● Easily scalable ○ Decentralized rings ● Completely adaptable ○ Schema-less key/value store DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET The Solution
- 8. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET …Almost ● Useful searching was missing ○ Secondary indexes not flexible ○ No free text searches ○ No (reasonable) range queries
- 9. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Pros: Full control over indexing ● Cons: Not scalable What We Needed
- 10. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Reasons to go with DSE ○ Combines Cassandra and Solr ○ Constant refinements and integrations ○ Support Putting It All Together
- 11. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Normalization Cassandra Solr Cassandra Solr Cassandra Solr Load Balancing Our Stack Web Crawling
- 12. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Cassandra / Solr Setup ● 3 column families / 3 cores ○ Locations ○ Products ○ People ● 73,114,909 records
- 13. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● 29,818,644 records ● Interesting data ○ Reviews ○ Revenue ○ Contact information ● Businesses vs. Locations ○ Unique key ○ Location specific user data Data: Locations
- 14. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Data: Products ● 18,470,005 records ● Interesting data ○ Categories ○ Price ○ Reviews ● Challenges ○ Too many unique keys
- 15. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Data: People ● 24,826,260 records ● Interesting data ○ Work History ○ Education History ○ Location ● Challenges ○ Normalization ○ Identification
- 16. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges ● Memory ● Speed ● Space ● Representation
- 17. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Memory ● Multi-minute garbage collection ● Exponential increase in frequency ● Virtual memory confusion ● Solr + Cassandra ● Heap Size vs Buffer Cache ● Bash Scripts
- 18. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Speed ● Upgrade ○ Better memory management ○ Smaller index size ● Reduce index size ● Future: Solaris
- 19. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Speed ● Providing a real-time service ● Issues ○ Solr not inherently real time ○ Search speeds ○ I/O
- 20. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Speed ● Solr Solution: DSE integration leverages ○ Cassandra's speed ○ Cassandra's caches ○ Cassandra's distribution ○ Solr caches less useful
- 21. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Speed ● Search complexity solution ○ Text vs String indexing ○ Uniqueness vs Flexibility ○ Leveraging Cassandra
- 22. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Speed ● I/O Solution ○ Cassandra's built in mapping ○ Increase disk access speeds (SSDs) ■ Not cost effective ○ Future: Solaris
- 23. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Space ● Field corruption ○ Caused by improper encoding ○ Exponential growth ○ Fills up Solr index ● Locate, inspect & remove corrupt records
- 24. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Space ● Solr index issue ○ No compression (vs Cassandra) ○ Must adjust indexing ● Key things to keep in mind ○ Size of fields ○ Scale vs Flexibility ○ Index as little as possible
- 25. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Representation ● Cassandra is flat ● Actual data is not flat ○ Reviews ○ Price information ● Many different output formats ○ CSV, JSON, XML, etc.
- 26. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Solution: Flatten when possible ○ E.g. Address object -> Separate fields ● Internal subgroup representation ○ Composite keys (Occasionally) ■ Known subgroups ■ Non multiple subgroups ○ Dynamic fields ■ Composite field + Dynamic tag ■ E.g. review.text_<tag> Challenges: Representation
- 27. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Challenges: Representation ● Robust and adaptable conversion package ● JSON -> Internal ○ Solr returns JSON ● Internal -> CSV, JSON, XML ○ User defined views ○ Specify field groupings ○ Specify partitioning
- 28. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET
- 29. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET ● Memory Usage ● Speed ● Space ● Containers Future Work
- 30. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Future Work: Memory ● Java 7 G1 (Garbage First) Collector ○ Ideal for large heaps ○ Big Data Sets ○ Bursty Workloads
- 31. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Future Work: Speed ● Solaris Kernel Scheduler > Linux Kernel Scheduler ○ (At large number of cores) ● Drastically increase iops ○ Cache reads (L2ARC) on PCIe SSD (~800 MB/s) ○ Cache writes (ZIL) on PCIe SSD (~800 MB/s) ○ Reduce needed size of SSD ■ More smaller SSDs in ZFS pool ○ Fewer moving parts
- 32. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Future Work: Space ● Caching at PCIe, Storing on SATA III ○ Cheaper larger storage via ZFS pools ○ Easier to grow ● ZFS Compression (LZ4) ○ Replaces Cassandra's Snappy compression ○ Very fast lossless compression (400 Mb/s per core) ○ Scales to multiple CPUs ○ Hits the ram speed limit
- 33. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Future Work: Containers ● OS Level virtualization ○ Resource control ○ Boundary separation ● More control over cassandra resources ● Better snapshots (whole machine) ● Hardware abstracted out ○ Many disks represented as single space ○ Easily add or remove hardware
- 35. DATAFINITI • THE SEARCH ENGINE FOR DATA • WWW.DATAFINITI.NET Addendum 1 ZFS Comparison Name Ratio (MB/s) Compression (MB/s) Decompression (MB/s) LZ4 (r97) 2.084 410 1810 LZO 2.06 2.106 409 600 QuickLZ 1.5.1b6 2.237 373 420 Snappy 1.1.0 2.091 323 1070 LZF 2.077 270 570 zlib 1.2.8 -1 2.730 65 280 LZ4 HC (r97) 2.720 25 2040 zlib 1.2.8 -6 3.099 21 300