Hivereader.com
I’ve outgrown my basic
stack. Now what?
Thoughts and feelings about growing
with Django and NoSQL
Hivereader.com
Our common stack is built on:
Super awesome and fast (once you learn what knobs to turn).
Lots of cool feat...
Hivereader.com
People are signing up and using your
app/game/site/bread maker/whatever
Hivereader.com
hooray
Hivereader.com
But
Hivereader.com
Things get start to get slow
Hivereader.com
or worse
Hivereader.com
Single server
appapp
DBDB
otherother
Hivereader.com
Find your bottlenecks:
Unless your app is doing something crazy
you’re mostly abusing the DB
Add more cachi...
Hivereader.com
Hivereader.com
pg_bouncer
Hivereader.com
Welcome to Postgres config file
pg_tune is here to help*
*kind of
Hivereader.com
Still Growing?
Hivereader.com
appapp
DBDB
otherother
appapp
DBDB
appapp
DBDB
otherother
appapp
DBDB
otherother
Hivereader.com
Need a solution for lots of data
that is growing quickly.
The solution needs to be targeted
for my problem.
Hivereader.com
NoSQL to the rescue?
Hivereader.com
But wait, can Postgres handle this?
most likely
partitioning
and
sharding
Can have lots of app code sometim...
Hivereader.com
Lots of options
+ tons more
Hivereader.com
Nearly all of them are based on 2 papers
Built on CAP theorem
The theorem began as a conjecture made by Uni...
Hivereader.com
Which one?
Hivereader.com
Super fast
Advanced key-value store
Think of it as super memcached. With union math.
All data must fit in r...
Hivereader.com
Document store
JSON-like documents with dynamic schemas
Ad-hoc queries
Indexing
Load-balancing MongoDB scal...
Hivereader.com
Uses pre-defined column family format
Map Reduce
Used by all people with with ‘big data’ problems
Amazing w...
Hivereader.com
JSON to store data,
JavaScript for MapReduce and HTTP for an API.
Views: embedded map/reduce
Multi-master r...
Hivereader.com
Based hardcore on Amazon's Dynamo paper
Key Value store
Super good about failure, “no downtime”
Map Reduce ...
Hivereader.com
Key value + row-oriented = column family
Linear scalability and fault-tolerance on commodity
hardware or cl...
Hivereader.com
So now what?
Hivereader.com
Things to think about:
Is eventual consistency ok for you?
Do you know your queries you need right now?
Is ...
Upcoming SlideShare
Loading in …5
×

I’ve outgrown my basic stack. Now what?

1,066 views

Published on

Thoughts and feelings about growing
with Django and NoSQL.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,066
On SlideShare
0
From Embeds
0
Number of Embeds
603
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

I’ve outgrown my basic stack. Now what?

  1. 1. Hivereader.com I’ve outgrown my basic stack. Now what? Thoughts and feelings about growing with Django and NoSQL
  2. 2. Hivereader.com Our common stack is built on: Super awesome and fast (once you learn what knobs to turn). Lots of cool features and tools: pg_tune, pg_top, pg_bouncer Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. In-memory key-value store for small chunks of data. Super simple and awesome
  3. 3. Hivereader.com People are signing up and using your app/game/site/bread maker/whatever
  4. 4. Hivereader.com hooray
  5. 5. Hivereader.com But
  6. 6. Hivereader.com Things get start to get slow
  7. 7. Hivereader.com or worse
  8. 8. Hivereader.com Single server appapp DBDB otherother
  9. 9. Hivereader.com Find your bottlenecks: Unless your app is doing something crazy you’re mostly abusing the DB Add more caching or, better yet, smarter caching
  10. 10. Hivereader.com
  11. 11. Hivereader.com pg_bouncer
  12. 12. Hivereader.com Welcome to Postgres config file pg_tune is here to help* *kind of
  13. 13. Hivereader.com Still Growing?
  14. 14. Hivereader.com appapp DBDB otherother appapp DBDB appapp DBDB otherother appapp DBDB otherother
  15. 15. Hivereader.com Need a solution for lots of data that is growing quickly. The solution needs to be targeted for my problem.
  16. 16. Hivereader.com NoSQL to the rescue?
  17. 17. Hivereader.com But wait, can Postgres handle this? most likely partitioning and sharding Can have lots of app code sometimes. What about when you outgrow your shard key “Don’t shard until you have to” - every single talk I’ve seen Slony? “master to multiple slaves” replication
  18. 18. Hivereader.com Lots of options + tons more
  19. 19. Hivereader.com Nearly all of them are based on 2 papers Built on CAP theorem The theorem began as a conjecture made by University of California, Berkeley computer scientist Eric Brewer at the 2000 Symposium on Principles of Distributed Computing. In 2002, Seth Gilbert and Nancy Lynch of MIT published a formal proof of Brewer's conjecture, rendering it a theorem.
  20. 20. Hivereader.com Which one?
  21. 21. Hivereader.com Super fast Advanced key-value store Think of it as super memcached. With union math. All data must fit in ram It is often referred to as a data structure server Keys can contain strings, hashes, lists, sets and sorted sets Not a db solution but more of a helper. This is now a part of our basic stack for most apps.
  22. 22. Hivereader.com Document store JSON-like documents with dynamic schemas Ad-hoc queries Indexing Load-balancing MongoDB scales horizontally using sharding MongoDB uses a readers-writer lock that allows concurrent reads access to a database but gives exclusive access to a single write operation.However, when a write lock exists, a single write operation holds the lock exclusively, and no other read or write operations may share the lock. Global write lock* Uncompressed field names Safe off by default Just google “mongo problems” or “moving off mongodb”
  23. 23. Hivereader.com Uses pre-defined column family format Map Reduce Used by all people with with ‘big data’ problems Amazing workhorse for data You need a sizable cluster Cluster setup can be difficult / hbase I need to personally spend more time with this
  24. 24. Hivereader.com JSON to store data, JavaScript for MapReduce and HTTP for an API. Views: embedded map/reduce Multi-master replication BigCouch, couchbase, Membase? Kind of in a dev rut... ...but just pushed a new huge upgrade
  25. 25. Hivereader.com Based hardcore on Amazon's Dynamo paper Key Value store Super good about failure, “no downtime” Map Reduce / Secondary Indexes Built-in full text search Link walking 2 types of mapreduce Javascript - can be slow as hell Erlang - super fast
  26. 26. Hivereader.com Key value + row-oriented = column family Linear scalability and fault-tolerance on commodity hardware or cloud infrastructure Built by Facebook for Messages Has CQL3 - think SQL, kind of Baked auto cluster AMI Super fast writes Compresses data that’s not accessed a lot Can tie in to Hadoop for big map reduce
  27. 27. Hivereader.com So now what?
  28. 28. Hivereader.com Things to think about: Is eventual consistency ok for you? Do you know your queries you need right now? Is your data complicated or simple? How fast does it grow? How long do you want that data to hang around? Really think about trade offs. Every system has its good and bad There is no “winner”, so stop searching “which is best” Think about which fits your use case http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis http://docs.basho.com/riak/1.2.1/references/appendices/comparisons/ Tons of links out there, just make sure they are relatively new

×