Scaling With Postgres

5,305 views

Published on

Talk given at Percona Performance Conference 2009

Published in: Technology

Scaling With Postgres

  1. 1. Scaling with Postgres <ul><li>Robert Treat </li></ul><ul><li>Percona Performance Conference </li></ul><ul><li>2009 </li></ul>
  2. 2. Disclaimer Every “really big” site uses custom solutions. Don’t expect to get there using out of the box software. You don’t have to reinvent the wheel. But you may have to build the car.
  3. 3. Who am I ? <ul><li>OmniTI </li></ul><ul><li>TB+ databases </li></ul><ul><li>top 100 websites </li></ul><ul><li>Oracle/MySQL/Postgres </li></ul><ul><ul><li>Not necessarily in that order :-) </li></ul></ul><ul><li>P.S. We’re Hiring </li></ul>
  4. 4. Scaling Databases 101 <ul><li>Implementation </li></ul><ul><ul><li>Tuning </li></ul></ul><ul><ul><li>Optimize Queries </li></ul></ul><ul><ul><li>Optimize Schema </li></ul></ul><ul><ul><ul><li>Throw Hardware At It </li></ul></ul></ul><ul><ul><ul><li>Caching </li></ul></ul></ul><ul><ul><ul><ul><li>Replication </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Federation </li></ul></ul></ul></ul></ul>
  5. 5. Implementation <ul><ul><ul><li>Normalization </li></ul></ul></ul><ul><ul><ul><li>Foreign Keys </li></ul></ul></ul><ul><ul><ul><li>Relational Modeling </li></ul></ul></ul>Mostly Irrelevant
  6. 6. Implementation <ul><ul><ul><li>Monitoring </li></ul></ul></ul><ul><ul><ul><ul><li>nagios, munin, etc.. </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>check_postgres </li></ul></ul></ul></ul></ul><ul><ul><li>Trending </li></ul></ul><ul><ul><ul><li>cacti, noit </li></ul></ul></ul><ul><ul><ul><ul><li>watch your hardware! </li></ul></ul></ul></ul>
  7. 7. Tuning <ul><ul><li>Get the Basics Right </li></ul></ul><ul><ul><ul><li>http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server </li></ul></ul></ul><ul><ul><li>Someday </li></ul></ul><ul><ul><ul><li>Xlogs, Checkpoints </li></ul></ul></ul><ul><ul><ul><li>Vacuum Strategy </li></ul></ul></ul><ul><ul><ul><li>Costs and other Minutia </li></ul></ul></ul>
  8. 8. Optimize Queries <ul><ul><ul><li>explain </li></ul></ul></ul><ul><ul><ul><li>explain analyze </li></ul></ul></ul><ul><ul><ul><li>explain analyze verbose </li></ul></ul></ul>
  9. 9. Optimize Queries <ul><ul><ul><li>writing queries tuned to your databases planner </li></ul></ul></ul>
  10. 10. Optimize Schema <ul><ul><ul><li>indexing </li></ul></ul></ul><ul><ul><ul><ul><li>partial, functional </li></ul></ul></ul></ul><ul><ul><ul><li>fix data types </li></ul></ul></ul><ul><ul><ul><li>add/remove tables </li></ul></ul></ul><ul><ul><ul><li>de-normalize (maybe) </li></ul></ul></ul>
  11. 11. Why is this important? <ul><ul><li>Postgres is good at scaling vertically </li></ul></ul><ul><ul><li>Flawed database schema hides the real problems </li></ul></ul>
  12. 12. Why is this unimportant? <ul><ul><li>Optimizing is often a trade off of time vs. money </li></ul></ul><ul><ul><li>It is not an ultimate solution </li></ul></ul>
  13. 13. Throw Hardware At It <ul><ul><li>Find your bottlenecks </li></ul></ul><ul><ul><ul><li>Memory / RAM </li></ul></ul></ul><ul><ul><ul><li>CPU </li></ul></ul></ul><ul><ul><ul><li>I/O, DISKS </li></ul></ul></ul><ul><ul><ul><ul><li>usually the biggest problem </li></ul></ul></ul></ul>
  14. 14. Can you take me higher? <ul><ul><li>32 Cores </li></ul></ul><ul><ul><li>128 GB+ RAM </li></ul></ul><ul><ul><li>Spindles out the wazoo </li></ul></ul><ul><ul><ul><li>tablespaces </li></ul></ul></ul>
  15. 15. Caching <ul><ul><ul><li>Internal </li></ul></ul></ul><ul><ul><ul><ul><li>materialized views </li></ul></ul></ul></ul><ul><ul><ul><li>External </li></ul></ul></ul><ul><ul><ul><ul><li>memcached, et al. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>static files </li></ul></ul></ul></ul>
  16. 16. Oh, the places you’ll go <ul><ul><li>Nothing Fancy or Exciting </li></ul></ul><ul><ul><li>Solid Engineering </li></ul></ul><ul><ul><li>Understanding needs / limits </li></ul></ul>Top 1000+ Website
  17. 17. Replication <ul><ul><li>Ignore the hype </li></ul></ul><ul><ul><ul><li>Triggers won’t destroy your site </li></ul></ul></ul><ul><ul><ul><li>Scripting solves a lot of config issues </li></ul></ul></ul><ul><ul><ul><li>Telling your DBA to do his !@#$ job solves config issues too </li></ul></ul></ul><ul><ul><ul><li>Add-on is FUD for Modular </li></ul></ul></ul>
  18. 18. Replication <ul><ul><ul><li>Truth in advertising </li></ul></ul></ul><ul><ul><ul><ul><li>Oracle does it better </li></ul></ul></ul></ul><ul><ul><ul><ul><li>MySQL does it simpler </li></ul></ul></ul></ul>
  19. 19. Replication <ul><li>Scaling Reads </li></ul><ul><ul><li>Master, many slaves </li></ul></ul><ul><ul><li>Slony, Londiste, Bucardo, PGPoolII </li></ul></ul><ul><li>Scaling Writes </li></ul><ul><ul><li>Mostly impossible </li></ul></ul><ul><ul><li>Bucardo, PGPoolII </li></ul></ul>
  20. 20. What is Federation? <ul><li>A process for eliminating performance and/or scaleability as a requirement for your database </li></ul>
  21. 21. Federation <ul><li>MySQL software has no federation advantage </li></ul><ul><li>MySQL community has large federation advantage </li></ul>
  22. 22. Federation <ul><li>Things to consider </li></ul><ul><ul><li>Horizontal Partitioning </li></ul></ul><ul><ul><li>Application Based Partitioning </li></ul></ul><ul><ul><li>BASE </li></ul></ul><ul><ul><ul><li>golconde, gearman </li></ul></ul></ul>
  23. 23. El Fin <ul><li>www.xzilla.net </li></ul><ul><li>www.omniti.com </li></ul><ul><li>www.postgresql.us </li></ul>
  24. 24. Postgres BOF <ul><li>Tonight @ 7:30 PM </li></ul><ul><li>Location @ Camino Real </li></ul>

×