I ns ide the P o s tg reS Q L
                                       Presentation Title
                                  ...
In the beginning...


       Date: Tue, 23 Apr 1996 16:06:10 -0400 (EDT)
       From: "Marc G. Fournier" <scrappy@ki.net>
...
The first server




                                 Hosted in Toronto, C anada

D ave P age, 25 th M arch 2010       Ins...
E arly services

       • M ailing lists

       • C VS repository

       • FTP site




D ave P age, 25 th M arch 2010  ...
14 Years later...

       • 20+ P hysical servers

       • 35+ Virtual M achines

       • Hosted in:
                –  ...
C urrent services

       •      FTP site
       •      Website
       •      S ource control – C VS and G IT
       •    ...
O S “zoo”

       • P rimarily using FreeB S D jails:
                – E asy to backup
                – E asy to relocat...
P roblems

       • Little FreeBS D experience in the community.

       • FreeB S D P orts are ha rd to upgrade, especial...
New infrastructure

       • R uns D ebian virtual machines under KVM on D ebian
         hosts.

       • P re-built pack...
M onitoring

       • Nagios

       • M unin

       • S mokeping

       • Auto-backup

       • G oogle Analytics




D...
Nagios

       • 64 Hosts

       • 514 S ervices

       • S ervice checks include:
                –    S ervice availab...
Nagios




D ave P age, 25 th M arch 2010   Inside the P ostgreS Q L P roject Infrastructure   S lide: 12
M unin

       • M onitors resource trends:
                –    D isk us age
                –    Network utilisation
   ...
M unin

       • C P U usage – postgresql01.managed.contegix.com




D ave P age, 25 th M arch 2010   Inside the P ostgreS...
S mokeping

       • M onitors network latency to various hosts from the
         C onova C ommunications data centre in S...
Auto-backup

       • Automatically backs up changes to key configuration
         files to S ubversion.

                ...
G oogle Analytics

       • M onitors website utilisation.

       • Helps us understand how the website is used.

       ...
G oogle Analytics




D ave P age, 25 th M arch 2010   Inside the P ostgreS Q L P roject Infrastructure   S lide: 18
FTP S ite

       • P rimary site: ftp.postgresql.org

       • 62 regional mirrors in 39 countries

       • M irrors may...
FTP M irror monitoring

       • All mirrors are checked daily by the 'mirrorbot'

       • The mirrorbot checks that cont...
Website infrastructure

       • D eveloped following the great 8.0 S lashdotting incident.

       • C apable of handling...
wwwmaster.postgresql.org

       • D ynamic, master server.

       • R uns custom-built P HP framework for:
             ...
www.postgresql.org

       • S tatic, slave servers

       • S erve HTM L, C S S , images and files such as P D Fs.

    ...
Website problems

       • P HP framework is complex, and understood by few.

       • Adding new dynamic content can requ...
New website infrastructure

       • S limmed down and vastly simplified framework, built
         using D jango and P yth...
Q uestions?




                                          Thank you.




D ave P age, 25 th M arch 2010   Inside the P ost...
Upcoming SlideShare
Loading in …5
×

Inside the PostgreSQL Project Infrastructure

1,143 views

Published on

Dave Page

This talk will give an insight into the infrastructure behind the PostgreSQL project - what servers do we have and what do they all do? How do we monitor and manage them? How does the website cope with the Slashdot effect on release days? We will also look at how things are likely to change as the Sysadmin and Web teams work to modernize and improve our infrastructure for the future.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,143
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Inside the PostgreSQL Project Infrastructure

  1. 1. I ns ide the P o s tg reS Q L Presentation Title P ro jec t I nfra s truc ture Presentation Sub-Title D ave P age P ostgreS Q L C ore Team S enior S oftware Architect, E nterpriseD B D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 1
  2. 2. In the beginning... Date: Tue, 23 Apr 1996 16:06:10 -0400 (EDT) From: "Marc G. Fournier" <scrappy@ki.net> Subject: Re: [PG95]: postgres95 TODO list posted on the web To: Chad Robinson <chadr@brttech.com> cc: Jolly Chen <jolly@postgres.berkeley.edu>, postgres95@shiloh.vnet.net … ... If it helps, I’d be willing to setup a cvs database, including appropriate accounts for a core few developers that patches can go through. From there, it wouldn’t be too hard to do a weekly "distribution" that is ftpable. I don’t know enough about the server backend to offer much more then that :( D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 2
  3. 3. The first server Hosted in Toronto, C anada D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 3
  4. 4. E arly services • M ailing lists • C VS repository • FTP site D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 4
  5. 5. 14 Years later... • 20+ P hysical servers • 35+ Virtual M achines • Hosted in: – France – P anama C ity – Austria – C anada – US A (4 independent locations) D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 5
  6. 6. C urrent services • FTP site • Website • S ource control – C VS and G IT • M ailing lists • Wiki • M ailing list archives • Website/ archives search • pgFoundry • C ommitfest management server • Buildfarm and Hudson servers • D evelopment servers… and more! D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 6
  7. 7. O S “zoo” • P rimarily using FreeB S D jails: – E asy to backup – E asy to relocate – P er-function jails • Also running: – Ubuntu – S lackware – C entO S – Windows D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 7
  8. 8. P roblems • Little FreeBS D experience in the community. • FreeB S D P orts are ha rd to upgrade, especially with lots of jails. • Hosting companies don't tend to like FreeBS D – and we can't be too picky! • No centralised management or deployment. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 8
  9. 9. New infrastructure • R uns D ebian virtual machines under KVM on D ebian hosts. • P re-built packages setup base hosts and VM s. • M anagement system automates: – VM creation and configuration. – Addition and removal of user accounts and S S H keys. – P ackage installation and upgrades. – D etection of unexpected user accounts or unauthorised services. – S etup and configuration of Nagios and M unin monitoring. – Auto-backup configuration D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 9
  10. 10. M onitoring • Nagios • M unin • S mokeping • Auto-backup • G oogle Analytics D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 10
  11. 11. Nagios • 64 Hosts • 514 S ervices • S ervice checks include: – S ervice availability – NTP, S S H, HTTP, FTP, R S YNC etc. – Utilisation – disk usage, logged in us ers, proces ses, mail queue – M anagement – software update availability – “O ur stuff” - buildfarm status, search indexer, database backups D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 11
  12. 12. Nagios D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 12
  13. 13. M unin • M onitors resource trends: – D isk us age – Network utilisation – P rocesses – S endmail/ ostfix stats P – C P U/ emory utilisation M – Apache stats D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 13
  14. 14. M unin • C P U usage – postgresql01.managed.contegix.com D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 14
  15. 15. S mokeping • M onitors network latency to various hosts from the C onova C ommunications data centre in S alzburg, Austria D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 15
  16. 16. Auto-backup • Automatically backs up changes to key configuration files to S ubversion. – G ives us a simple backup of config files – Allows us to trace the history of changes to a file • Alerts the sysadmins to changes to monitored files – Helps us see what the other team members are doing – Acts as a simple Intrusion Detection System D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 16
  17. 17. G oogle Analytics • M onitors website utilisation. • Helps us understand how the website is used. • C an be hampered by disabled scripting support in browsers, common with computer geeks! D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 17
  18. 18. G oogle Analytics D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 18
  19. 19. FTP S ite • P rimary site: ftp.postgresql.org • 62 regional mirrors in 39 countries • M irrors may also serve content via: – HTTP (supported) – R S YNC (unsupported) • C ontent includes main FTP site, and pgFoundry downloads D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 19
  20. 20. FTP M irror monitoring • All mirrors are checked daily by the 'mirrorbot' • The mirrorbot checks that content is up to date: – Fresh mirrors have a D NS hostname, e.g. ftp.uk.postgresql.org – Fresh mirrors are listed on the website for users to choose • O ut of date or broken mirrors: – Are automatically removed from the website and D NS . – Are reported to their maintainers via email. – Are automatically purged from the system if un-fixed. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 20
  21. 21. Website infrastructure • D eveloped following the great 8.0 S lashdotting incident. • C apable of handling high-load scenarios on release days.. • M inimised points of failure. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 21
  22. 22. wwwmaster.postgresql.org • D ynamic, master server. • R uns custom-built P HP framework for: – S tatic page rendering (general content) – D ynamic page rendering (docs, news, events etc) – Form processing • D ynamic content stored in P ostgreS QL. • S tatic version of content generated hourly by a spider, and pushed via R S YNC to the static servers. • Users redirected back to static servers where possible. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 22
  23. 23. www.postgresql.org • S tatic, slave servers • S erve HTM L, C S S , images and files such as P D Fs. • C urrently 2 servers. • G eographically diverse. • R ound-robin load balanced via D NS . • M onitoring system dynamically removes servers from D NS within minutes of a failure. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 23
  24. 24. Website problems • P HP framework is complex, and understood by few. • Adding new dynamic content can require significant effort to build administration pages. • The framework includes lots of features and code we thought we needed, but then never used. • S pider can take hours to process the entire site. • S pidering the site is very inefficient. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 24
  25. 25. New website infrastructure • S limmed down and vastly simplified framework, built using D jango and P ython. • D jango's administration module makes it easy to add and manage content. • S pider and static slaves will be replaced with Varnish cache s ervers: – P ages dynamically cached from wwwmaster on first request. – Last available content served if wwwmaster goes down. – C ache invalidation of individual or groups of pages as changed on wwwmaster. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 25
  26. 26. Q uestions? Thank you. D ave P age, 25 th M arch 2010 Inside the P ostgreS Q L P roject Infrastructure S lide: 26

×