The document discusses strategies for indexing big data in the cloud, highlighting the importance of agility, minimalism, and effective use of tools like Solr. It outlines a real project approach, emphasizing the need for quick prototypes and thorough testing while managing large datasets. Key technical details on server setup and scaling methods are also included, along with the potential alternatives for managing cloud resources.

1. Indexing Big Data in the Cloud

2. Me
Scott Stults
Co-Founder of OpenSource Connections
Solr / Lucene
Bash / Python / Java
Indexing Big Data in the Cloud 2

3. Eric
Indexing Big Data in the Cloud 3

4. Big Data
Indexing Big Data in the Cloud 4

5. Big Data Wrangler
Indexing Big Data in the Cloud 5

6. How?
Address a Real Project
Be Agile
Make Small Mistaeks Fast
Succeed BIG
Indexing Big Data in the Cloud 6

7. USPTO Goals
Prototype Search UX
Prove Solr:
Scales
Integrates
Excels
Indexing Big Data in the Cloud 7

8. Scale?
Indexing Big Data in the Cloud 8

9. Our Approach
KISS
YAGNI
(This space intentionally left blank)
Indexing Big Data in the Cloud 9

10. Minimal Flair
Indexing Big Data in the Cloud 10

11. Record Everything!
Indexing Big Data in the Cloud 11

12. Some Numbers
Doc Count 1.1 Million
Zip Files 313
Docs per Zip File 4,000
Zip File Size 75M
File Size 300M
Indexing Big Data in the Cloud 12

13. Testing
Start some servers
Process a batch
Check the clock
Indexing Big Data in the Cloud 13

14. start_nodes
start_nodes() {
ec2-run-instances ami-1b814f72
--block-device-mapping '/dev/sdb=snap-48adde35::true'
--block-device-mapping '/dev/sdi1=:10:false'
--block-device-mapping '/dev/sdi2=:10:false'
--block-device-mapping '/dev/sdi3=:20:false'
--instance-type m1.large
--key uspto-proto
--instance-count $MAX_NODES
--group default > ~/run-output
}
Indexing Big Data in the Cloud 14

15. Gut Check
How fast can we do this?
What can we do in parallel?
Indexing Big Data in the Cloud 15

16. Scaling
Raise our instance limit
xargs -P GNU parallel
Indexing Big Data in the Cloud 16

17. Shortcomings
SSH?
Error recovery
One Solr
Indexing Big Data in the Cloud 17

18. Alternatives
CloudFormation
Puppet / Chef
Multiple Cores / Shards
Hadoop
Indexing Big Data in the Cloud 18

19. Success
Indexing Big Data in the Cloud 19

20. Victory Lap
Indexing Big Data in the Cloud 20

21. Instances / Time
Indexing Big Data in the Cloud 21

22. Thank You
https://github.com/sstults/patent-indexing
@scottstults
#o19s
Indexing Big Data in the Cloud 22