Raiffeisen OnLine implemented an open-source mail cluster to replace their existing Windows solution. They initially set up two servers running Postfix, Spamassassin, ClamAV, AmaVis, and MySQL in a redundant configuration. Over time, they enhanced the cluster by adding new open-source components like Milter and policy daemons to improve performance and security. The cluster now processes high mail volumes with IPv6 support, SSL/TLS encryption, and a front-end interface for customer support.

1. 18.11.2014
Raiffeisen OnLine‘s OSS mail cluster

2. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Purpose + requirements
•implementation of a new spam filter system
•replacement of the present Windows solution (3 servers)
•redundant setup
•Active/Active preferred
•easily scalable and flexible
•give our “Service Center” access to
•account configuration setting
•log data for customer support purposes
•should allow further expansion in order to implement new features and improve hit rate

3. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Which solution should we use?
•Evaluation of manufacturers
•major differences in price/performance ratio
•not all solutions grant actual redundancy (only standby setup)
•Test installation by a manufacturer works OK …
•… until Friday afternoon, when a sudden deadlock occurs!
•After having spent 3 days searching for the error, the manufacturer states that:
•…there must be a power outage in the datacenter -> I’m sure there was no outage 

4. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
We finally decided to try out a self-made solution
•But why OSS?
•The experience told us it’s preferable to adopt a solution on which we are able to investigate by ourselves in an emergency situation.
•We were aiming at a solution that would allow us to meet new requirements on our own in the future.
•We wanted to be able to develop own new components if necessary.
•Our objective was to reduce dependency on one single manufacturer.

5. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
We found all components in the open source world very quickly
•First implementation in the middle of 2008:
•Postfix (MTA) -> on the front line
•Spamassassin + ClamAV -> as content filter
•AmaVis -> interface between postfix + content filter
•MySQL –> configuration + quarantine (Spam/Virus)
•MySQL -> central logging for “ServiceCenter” + customers
•PHP -> administration front end for “ServiceCenter” + customers

6. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
First phase of implementation
•2x bare metal servers (8GB RAM, QuadCore, SAS)
•„extreme“ container-virtualization
•Each service had its own container
•2 Postfix front end MTAs
•2 AmaVis (Spamassassin, ClamAV) content filters
•2 MySQL Master/Master Replications (configuration)
•2 MySQL Master/Master Replications (quarantine)
•2 MySQL Master/Master Replications (logdatabase)
•2 DNS Resolvers
•2 Postfix Backend MTAs (delivering e-mails to storage back ends)

7. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Each node could go down at any time
•Redundant SMTP paths
IN ---> Postfix---> AmaVis---> Postfix---> OUT

8. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
What are the main problems?
•OSS components are available in heaps, but it was difficult to
•have all different components work together
•process large amounts of log data in order for users to understand them
•We had not much experience with high mail traffic.
•It took us some time to identify the correct performance and configuration parameters.

9. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Technical support
•Google 
•knows about 90% of all the problems we came across
•Community
•We posted our own questions to mailing lists and forums.
•We kept reading newsgroups.
•Common sense
•logical + analytical approach
•“Don’t worry, it’s just numbers and letters.”*
* cit.: Thomas Gelf

10. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Continuous enhancements in the last years
•Perl -> small policy daemons we developed in 2012 and 2013
•selective grey listing
•performance improvements, storing non important data into the RAM
•ClamAv content filter replaced(July 2014)
•by ClamAv-Milter: scans viruses directly on front end servers and rejects with 550 in real time instead of storing in quarantine
•SPF-Policyd (Juli 2014)
•supports Sender Policy Framework in either direction (In/Out)
•OpendDKIM (Juli 2014)
•supports Domain Key Identified Mail in either direction (In/Out)
•OpenDMARC (Juli 2014)
•supports Domain-based Message Authentication, Reporting & Conformance in either direction (In/Out)

11. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Current mail traffic (Postfix 1)

12. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Current mail traffic (Postfix 2, Content filter)

13. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Some more highlights:
•SSL/TLS availableon all front end servers
•IPv6
•active on all front end servers in both directions
•(partial) communication between components
•now facing IMAP/POP3 + SMTP submission
•IMAP + POP3 + SMTP failover
•front end displays processed log data for „ServiceCenter“ and customers

14. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
IMAP + POP3 failover
•outward: virtialIP + Multicast MAC
•basedon IPTablesCluster-IP Module
•[imap,pop3].rolmail.net (Dovecot)
•smtp.rolmail.net (Postfix)
•2 proxyservers
•imap1.rolmail.net
•imap2.rolmail.net
•morepossible
•Mailstore:
•manybackend servers(Dovecot)
•Load balancingofI/O

15. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Processed data for „ServiceCenter“ and customers

16. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
Future plans
•to force SSL/TLS for IMAP/POP3 + SMTP submission clients
•to protect mail traffic with DNS-based Authentication of Named Entities (called DANE)
•protection of zones with DNSSEC is needed
•to improve quarantine mechanism
•rejecting all e-mails tagged as spam directly on the frond end servers with 550 -> no mail will be stored in quarantine
•maybe to implement the mailstore backend server pool as an active/active setup

17. Raiffeisen OnLine‘sOSS mail cluster
urban.loesch@raiffeisen.net
18.11.2014
### It was a pleasure to be here ###
Many thanks for your attention!
Urban Lösch
Raiffeisen OnLine GmbH
urban.loesch@raiffeisen.net