SSU Email Infrastructure

799 views
676 views

Published on

Slides for a talk I gave for the Computer Science Colloquium at Sonoma State University on 2009-10-08

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
799
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • First a couple disclaimers:
    * It’s all my own opinions. Definitely nothing here represents an official university statement of any kind
    * Many people have been involved with the design and operation of this system, which has evolved over time. I’m making no attempt to give credit; assume any bad ideas were mine and any good ideas were somebody else’s.
    * I’m a Linux Server guy. The network is essential, but I’m ignoring it.
    * I have a life. And furloughs. I originally signed up for this a year ago, and in the meantime, life got complicated.
  • First a couple disclaimers:
    * It’s all my own opinions. Definitely nothing here represents an official university statement of any kind
    * Many people have been involved with the design and operation of this system, which has evolved over time. I’m making no attempt to give credit; assume any bad ideas were mine and any good ideas were somebody else’s.
    * I’m a Linux Server guy. The network is essential, but I’m ignoring it.
    * I have a life. And furloughs. I originally signed up for this a year ago, and in the meantime, life got complicated.
  • First a couple disclaimers:
    * It’s all my own opinions. Definitely nothing here represents an official university statement of any kind
    * Many people have been involved with the design and operation of this system, which has evolved over time. I’m making no attempt to give credit; assume any bad ideas were mine and any good ideas were somebody else’s.
    * I’m a Linux Server guy. The network is essential, but I’m ignoring it.
    * I have a life. And furloughs. I originally signed up for this a year ago, and in the meantime, life got complicated.
  • First a couple disclaimers:
    * It’s all my own opinions. Definitely nothing here represents an official university statement of any kind
    * Many people have been involved with the design and operation of this system, which has evolved over time. I’m making no attempt to give credit; assume any bad ideas were mine and any good ideas were somebody else’s.
    * I’m a Linux Server guy. The network is essential, but I’m ignoring it.
    * I have a life. And furloughs. I originally signed up for this a year ago, and in the meantime, life got complicated.
  • It’s Magic!
  • It’s Magic!
  • It’s Magic!
  • Too complicated
  • Still too complicated, and still missing some details
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • RFC 821; SMTP; August 1982 (port 25)
    RFC 1869; ESMTP; November 1995; revised
    RFC 1870; ESMTP SIZE; November 1995
    RFC 2476; MSP (mail submission); December 1998 (port 587)
    RFC 2920; ESMTP PIPELINING; September 2000
    RFC 3207; ESMTP STARTTLS; February 2002
    RFC 5321; SMTP rewritten; October 2008 (consolidates; updates; clarifies; doesn’t change)
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • Example blatantly stolen from http://en.wikipedia.org/wiki/Post_Office_Protocol
  • How email works *here*
    Note ssuMailHomeDirectory
  • No Locking
    time.pseudorandom.hostname,flags
    psuedorandom(P=pid, H=microseconds)
  • Sent January 21, 2004
    Guaranteed Delivery
  • 2003/11 (approx): 300G
    2007/05: 700G
    2009/01: 1.05T
  • What’s our current peak?

    Back in 2007 I said “12 per second”. What is it now?

    https://groundwork.sonoma.edu/cacti/graph_image.php?local_graph_id=271&rra_id=0&view_type=tree&graph_start=1154415600&graph_nolegend=true&graph_height=300&graph_width=900

    https://groundwork.sonoma.edu/cacti/graph_image.php?local_graph_id=4407&rra_id=0&view_type=tree&graph_start=1154415600&graph_nolegend=true&graph_height=300&graph_width=1200
  • What’s our current peak?

    Back in 2007 I said “12 per second”. What is it now?

    https://groundwork.sonoma.edu/cacti/graph_image.php?local_graph_id=271&rra_id=0&view_type=tree&graph_start=1154415600&graph_nolegend=true&graph_height=300&graph_width=900

    https://groundwork.sonoma.edu/cacti/graph_image.php?local_graph_id=4407&rra_id=0&view_type=tree&graph_start=1154415600&graph_nolegend=true&graph_height=300&graph_width=1200
  • Note logarithmic scale
    Moore’s Law: chip density doubles every 18 months
    related: chip speed, memory sizes, transfer speeds
  • Kryder’s Law: magnetic disk areal storage density doubles annually
  • Diagram drive now
    Disk access/seek time: time to move head + time to rotate drive
    rotational speed is key
    Big jump soon with SSD? Still too expensive.
  • History
  • “No Layoff” unofficial policy at SSU relates to things/stuff being easier to buy than people. (stuff is up front; people are long term commitment)
    The amount of money isn’t the only limit, but the nature of how it flows in relation to the project. Large (capital) amounts are always special request.
    Email == cost center (not money maker)
  • “No Layoff” unofficial policy at SSU relates to things/stuff being easier to buy than people. (stuff is up front; people are long term commitment)
    The amount of money isn’t the only limit, but the nature of how it flows in relation to the project. Large (capital) amounts are always special request.
    Email == cost center (not money maker)
  • * Reliable - Highly Available, Fault Tolerant, Redundant components (just enough), no staff available outside 6:30am-6pm
    * Cost: prefer to spend money once, not every year
    * Concurrent: lots of users all at once. Locks = Evil
  • * Reliable - Highly Available, Fault Tolerant, Redundant components (just enough), no staff available outside 6:30am-6pm
    * Cost: prefer to spend money once, not every year
    * Concurrent: lots of users all at once. Locks = Evil
  • * Reliable - Highly Available, Fault Tolerant, Redundant components (just enough), no staff available outside 6:30am-6pm
    * Cost: prefer to spend money once, not every year
    * Concurrent: lots of users all at once. Locks = Evil
  • * Reliable - Highly Available, Fault Tolerant, Redundant components (just enough), no staff available outside 6:30am-6pm
    * Cost: prefer to spend money once, not every year
    * Concurrent: lots of users all at once. Locks = Evil
  • Diagram VM infrastructure
    5 hosts; NetApp + EMC storage
    VMotion
  • beginnings: wiggum, dir1->dir2, 2*IMAP, 2*exim, 2*spam
  • beginnings: wiggum, dir1->dir2, 2*IMAP, 2*exim, 2*spam
  • beginnings: wiggum, dir1->dir2, 2*IMAP, 2*exim, 2*spam
  • beginnings: wiggum, dir1->dir2, 2*IMAP, 2*exim, 2*spam
  • beginnings: wiggum, dir1->dir2, 2*IMAP, 2*exim, 2*spam
  • Summer 2005; draw pegged graph
    Major error: didn’t get metrics going
  • Fall 2007; talk about EMC
  • diagram multiple shared block storage configs
  • Growth: not just “infinite”, but have any idea how much, how fast, etc...
  • SSU Email Infrastructure

    1. 1. Sonoma State’s Email SSU CS Colloquium, October 8, 2009 Eric Eisenhart <eric.eisenhart@sonoma.edu> http://eric.eisenhart.name/presentations/
    2. 2. Disclaimers
    3. 3. Disclaimers • My opinions
    4. 4. Disclaimers •My opinions •No Credit
    5. 5. Disclaimers •My opinions •No Credit •Network == Air
    6. 6. Disclaimers •My opinions •No Credit •Network == Air •Life
    7. 7. Too Simple
    8. 8. Too Simple A Mail Server #1
    9. 9. Too Simple A B Mail Server Mail Server #1 #2
    10. 10. Big Picture Alice Bob Load Balancer #1 ns authns mail ldaps webmail ns cluster mail cluster authns cluster moltar zorak blinky snorky ns-a ns-b ns-c chalmers clancy cletus krusty milhouse authns-a authns-b authns-c NetApp Cluster mojo ns3.csu.net flanders heckle jeckle heckle- jeckle- shelf-1 shelf-1 heckle- jeckle- shelf-2 shelf-2
    11. 11. Big Picture Chuck Red Condor #1 Red Condor #2 Alice Bob Red Condor Offsite Load Balancer #1 ns authns mail ldaps webmail ns cluster mail cluster authns cluster moltar zorak blinky snorky ns-a ns-b ns-c chalmers clancy cletus krusty milhouse authns-a authns-b authns-c NetApp Cluster mojo ns3.csu.net flanders heckle jeckle heckle- jeckle- shelf-1 shelf-1 heckle- jeckle- shelf-2 shelf-2
    12. 12. Simpler Picture?
    13. 13. How Email Works
    14. 14. An Email Return-path: <root@eisenhae> Envelope-to: eisenhae@sonoma.edu Delivery-date: Wed, 30 Sep 2009 09:57:52 -0700 Received: from exim by nelson with spam-scanned (Exim 4.63) (envelope-from <root@eisenhae>) id 1Mt2Ut-0007cJ-M2 for eisenhae@sonoma.edu; Wed, 30 Sep 2009 09:57:52 -0700 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on nelson X-Spam-Level: * X-Spam-Status: No, score=1.6 required=5.0 tests=MISSING_DATE,MISSING_HEADERS, MISSING_MID autolearn=no version=3.2.5 Received: from eisenhae ([1.2.3.4] helo=eisenhae) by nelson with esmtp (Exim 4.63) (envelope-from <root@eisenhae>) id 1Mt2Uf-0007cC-FN for eric.eisenhart@sonoma.edu; Wed, 30 Sep 2009 09:57:47 -0700 Subject: test message Message-Id: <E1Mt2Ut-0007cJ-M2@nelson> From: root@eisenhae Date: Wed, 30 Sep 2009 09:57:52 -0700 just a test.
    15. 15. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol)
    16. 16. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
    17. 17. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae
    18. 18. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP
    19. 19. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu
    20. 20. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK
    21. 21. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu
    22. 22. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted
    23. 23. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA
    24. 24. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA 354 Enter message, ending with "." on a line by itself
    25. 25. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA 354 Enter message, ending with "." on a line by itself Subject: test message just a test. .
    26. 26. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA 354 Enter message, ending with "." on a line by itself Subject: test message just a test. . 250 OK id=1Mt2Uf-0007cC-FN
    27. 27. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA 354 Enter message, ending with "." on a line by itself Subject: test message just a test. . 250 OK id=1Mt2Uf-0007cC-FN QUIT
    28. 28. SMTP + MSP (Simple Mail Transport Protocol + Mail Submission Protocol) 220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700 EHLO eisenhae 250-nelson Hello eisenhae [1.2.3.4] 250-SIZE 20971520 250-PIPELINING 250-STARTTLS 250 HELP MAIL FROM:root@eisenhae.sonoma.edu 250 OK RCPT TO:eric.eisenhart@sonoma.edu 250 Accepted DATA 354 Enter message, ending with "." on a line by itself Subject: test message just a test. . 250 OK id=1Mt2Uf-0007cC-FN QUIT 221 nelson.sonoma.edu closing connection
    29. 29. POP3 (Post Office Protocol)
    30. 30. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
    31. 31. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose
    32. 32. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted
    33. 33. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass
    34. 34. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted
    35. 35. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT
    36. 36. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320
    37. 37. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST
    38. 38. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets)
    39. 39. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120
    40. 40. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200
    41. 41. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 .
    42. 42. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1
    43. 43. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets
    44. 44. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1>
    45. 45. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1> .
    46. 46. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1> . DELE 1
    47. 47. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1> . DELE 1 +OK message 1 deleted
    48. 48. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1> . DELE 1 +OK message 1 deleted QUIT
    49. 49. POP3 (Post Office Protocol) +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> USER mrose +OK User accepted PASS mrosepass +OK Pass accepted STAT +OK 2 320 LIST +OK 2 messages (320 octets) 1 120 2 200 . RETR 1 +OK 120 octets <the POP3 server sends message 1> . DELE 1 +OK message 1 deleted QUIT +OK dewey POP3 server signing off (maildrop empty)
    50. 50. IMAP (Internet Message Access Protocol)
    51. 51. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information.
    52. 52. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
    53. 53. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok.
    54. 54. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX
    55. 55. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX * FLAGS (Draft Answered Flagged Deleted Seen Recent) * OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited * 751 EXISTS * 1 RECENT * OK [UIDVALIDITY 1201564895] Ok * OK [MYRIGHTS "acdilrsw"] ACL 2 OK [READ-WRITE] Ok
    56. 56. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX * FLAGS (Draft Answered Flagged Deleted Seen Recent) * OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited * 751 EXISTS * 1 RECENT * OK [UIDVALIDITY 1201564895] Ok * OK [MYRIGHTS "acdilrsw"] ACL 2 OK [READ-WRITE] Ok 3 FETCH 1 BODY[]
    57. 57. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX * FLAGS (Draft Answered Flagged Deleted Seen Recent) * OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited * 751 EXISTS * 1 RECENT * OK [UIDVALIDITY 1201564895] Ok * OK [MYRIGHTS "acdilrsw"] ACL 2 OK [READ-WRITE] Ok 3 FETCH 1 BODY[] * 1 FETCH (BODY[] {2390} <server sends full message with headers> 3 OK FETCH completed.
    58. 58. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX * FLAGS (Draft Answered Flagged Deleted Seen Recent) * OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited * 751 EXISTS * 1 RECENT * OK [UIDVALIDITY 1201564895] Ok * OK [MYRIGHTS "acdilrsw"] ACL 2 OK [READ-WRITE] Ok 3 FETCH 1 BODY[] * 1 FETCH (BODY[] {2390} <server sends full message with headers> 3 OK FETCH completed. 4 LOGOUT
    59. 59. IMAP (Internet Message Access Protocol) * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information. 1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop= 1 OK LOGIN Ok. 2 SELECT INBOX * FLAGS (Draft Answered Flagged Deleted Seen Recent) * OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited * 751 EXISTS * 1 RECENT * OK [UIDVALIDITY 1201564895] Ok * OK [MYRIGHTS "acdilrsw"] ACL 2 OK [READ-WRITE] Ok 3 FETCH 1 BODY[] * 1 FETCH (BODY[] {2390} <server sends full message with headers> 3 OK FETCH completed. 4 LOGOUT * BYE Courier-IMAP server shutting down 4 OK LOGOUT completed
    60. 60. LDAP (Lightweight Directory Access Protocol) dn: uid=eisenhae,ou=people,o=sonoma,o=edu uid: eisenhae ssuMailHomeDirectory: /nfs/jeckle/maildirs/92/eisenhae ssuMailLocalAddress: eisenhae@sonoma.edu ssuMailLocalAddress: eric.eisenhart@sonoma.edu ssuMailRoutingAddress: eisenhae@smtp-vip.sonoma.edu ssuMailSpamAssassinPrefs: required_hits 5 ssuMailSpamAssassinPrefs: whitelist_from *@example.com ssuMailDiskQuota: 1024M
    61. 61. Maildir (++) /nfs/jeckle/maildirs/92/eisenhae/ courierimapsubscribed courierimapuiddb tmp/ new/ new/1254971281.H995082P22240.cletus.sonoma.edu,S=3593 cur/ cur/1251296264.H25153P29370.krusty.sonoma.edu,S=3045:2,RS .GrayMai/cur/ .GrayMail/new/ .GrayMail/tmp/ .Lists/ .Lists.puppet/
    62. 62. The aka: requirements
    63. 63. Email == “Sonoma State University will employ these email accounts for official University communications to students effective immediately.” http://www.sonoma.edu/it/get_connected/ provost_letter.shtml
    64. 64. Reliance
    65. 65. Reliance Assumed
    66. 66. Reliance Assumed Addicted
    67. 67. Growth
    68. 68. Population
    69. 69. 1500 1125 GB 750 375 0 2004 2005 2006 2007 2008 2009 ~300G 700G 1.35T Storage Usage
    70. 70. SPA
    71. 71. SPA
    72. 72. Traffic 2006-08 to now-
    73. 73. Peak Traffic 2006-08 to now-
    74. 74. Moore’s Law: Friend or Foe?
    75. 75. The aka: what we did
    76. 76. Limits
    77. 77. Limits • Money
    78. 78. Limits •Money •Time / Staff
    79. 79. Goals
    80. 80. Goals • Reliable
    81. 81. Goals •Reliable •Minimal Update Costs
    82. 82. Goals •Reliable •Minimal Update Costs •Concurrency
    83. 83. Goals •Reliable •Minimal Update Costs •Concurrency •Asynchronous
    84. 84. What We Run
    85. 85. What we Run Hardware
    86. 86. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
    87. 87. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD • LDAP: 2 + master + standby. 2GB, 2 Intel 3GHz
    88. 88. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD • LDAP: 2 + master + standby. 2GB, 2 Intel 3GHz • webmail: 2 VMs. 2GB
    89. 89. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD • LDAP: 2 + master + standby. 2GB, 2 Intel 3GHz • webmail: 2 VMs. 2GB • lists: 1 VM. 1GB
    90. 90. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD • LDAP: 2 + master + standby. 2GB, 2 Intel 3GHz • webmail: 2 VMs. 2GB • lists: 1 VM. 1GB • MySQL: 1 VM. 1GB.
    91. 91. What we Run Hardware • MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD • LDAP: 2 + master + standby. 2GB, 2 Intel 3GHz • webmail: 2 VMs. 2GB • lists: 1 VM. 1GB • MySQL: 1 VM. 1GB. • storage: NetApp FAS 3070c cluster; 30*136GB 15k
    92. 92. Virtual Machines?
    93. 93. What we Run “MAIL”
    94. 94. What we Run “MAIL” •exim
    95. 95. What we Run “MAIL” •exim •clamav
    96. 96. What we Run “MAIL” •exim •clamav •SpamAssassin
    97. 97. What we Run “MAIL” •exim •clamav •SpamAssassin •courier-auth
    98. 98. What we Run “MAIL” •exim •clamav •SpamAssassin •courier-auth •courier-imap
    99. 99. What we Run Storage
    100. 100. What we Run Storage •2 * NetApp FAS 3070s in active-active cluster
    101. 101. What we Run Storage •2 * NetApp FAS 3070s in active-active cluster •10 shelves of 15 disks
    102. 102. What we Run Storage •2 * NetApp FAS 3070s in active-active cluster •10 shelves of 15 disks •EMC NAS (going, going, gone)
    103. 103. What we Run “webmail”
    104. 104. What we Run “webmail” •apache httpd
    105. 105. What we Run “webmail” •apache httpd •php
    106. 106. What we Run “webmail” •apache httpd •php •squirrelmail + plugins
    107. 107. What we Run “webmail” •apache httpd •php •squirrelmail + plugins •up-imapproxy
    108. 108. What we Run “ldaps”
    109. 109. What we Run “ldaps” •apache httpd
    110. 110. What we Run “ldaps” •apache httpd •OpenLDAP (slapd)
    111. 111. What we Run “ldaps” •apache httpd •OpenLDAP (slapd) •Lots of custom code
    112. 112. What we Run “flanders”
    113. 113. What we Run “flanders” •mysqld
    114. 114. What we Run “lists”
    115. 115. What we Run “lists” •apache httpd
    116. 116. What we Run “lists” •apache httpd •mailman
    117. 117. What we Run Secret Sauces
    118. 118. What we Run Secret Sauces •Tsunami
    119. 119. What we Run Secret Sauces •Tsunami •puppet
    120. 120. What we Run Secret Sauces •Tsunami •puppet •Registry
    121. 121. SPAM
    122. 122. SPAM • SpamAssassin
    123. 123. SPAM •SpamAssassin •Red Condor
    124. 124. Changes (so far) • storage backend (wiggum; otto; bruno; heckle+jeckle) • Directory layout (f/foo -> ac/foo) • Red Condor • IMAP + SMTP + SPAM + clamav (same servers now) • SPAM prefs from MySQL to LDAP • Virtualization
    125. 125. The Past
    126. 126. The Past • My First: one box
    127. 127. The Past • My First: one box • VAX; cluster
    128. 128. The Past • My First: one box • VAX; cluster • 2003: VAX + student server
    129. 129. The Past • My First: one box • VAX; cluster • 2003: VAX + student server • Later 2003: wiggum -> otto
    130. 130. The Past • My First: one box • VAX; cluster • 2003: VAX + student server • Later 2003: wiggum -> otto • Summer 2005: performance?
    131. 131. Failures aka: what went wrong
    132. 132. Spring 2006
    133. 133. Fall 2007-Spring 2008
    134. 134. Problems
    135. 135. Problems • Cache
    136. 136. Problems •Cache •SPAM accuracy
    137. 137. Problems •Cache •SPAM accuracy •Local SPAM
    138. 138. Problems •Cache •SPAM accuracy •Local SPAM •Quotas
    139. 139. Alternatives / Future • Smarter Load Balancers • IMAP server with local caching • Shared block storage (mini clusters?) • Exchange • Outsourced (Google)
    140. 140. Building your own? • Plan for Growth • Understand how to scale everything O(1) > O(log(n)) > O(n)
    141. 141. Questions!

    ×