7 Secrets of Wells Fargo SOA platform's high availability

1. Ram Lakshmanan
Sr. Systems Architect
Java One – 9/23/2013
secrets of Wells Fargo SOA
platform's 99.99% availability

2. About Wells Fargo
 Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages

3. About Wells Fargo
 Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages
 Global SIFI (Systemically Important
Financial Institution)

4. About Wells Fargo
 Just two data points
– Serving 1 in 3 US households
– Originating 1 in 3 US mortgages
 Global SIFI (Systemically Important
Financial Institution)
 4th in assets & 1st in market value
among US peers
 No. 1 most valuable financial services
brand in the world (Brand Finance, The
Banker)

5. Agenda
 High Level Architecture
 Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. 240k
5. Cockpit
6. Wackamole
7. *
 Meta-Secret

6. High Level Architecture

7. High Level Architecture
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)

8. High Level Architecture
SOR-2
SOR-60
SOR-5
SOR-3
SOR-4
SOR-6
SOR-1
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)

9. High Level Architecture
SOR-2
SOR-60
SOR-5
SOR-3
SOR-4
SOR-6
SOR-1
Online App
Mobile App
Stores App
Call Center App
Other LOB Apps (100+)
SOA
<SOAP/>
HTTP
JDBC
JMS
HTTPS
LDAP

10. Agenda
 High Level Architecture
 Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. Cockpit
5. 240k
6. Wackamole
7. *
 Meta-Secret

11. Secret 1: AAA…
datacenters: active-active-active…

12. Secret 2: One 9

13. Secret 2: One 9
 With “intelligent” Software that is running on one 9 (i.e.
90%) hardware availability you can achieve Four 9s (i.e.
99.99%) availability
SORSOA
SORSOA
Data Center 1
Data Center 2

14. Secret 2: One 9
 With “intelligent” Software that is running on one 9 (i.e.
90%) hardware availability you can achieve Four 9s (i.e.
99.99%) availability
SORSOA
SORSOA
Data Center 1
Data Center 2

15. Secret 2: One 9
 With “intelligent” Software that is running on one 9 (i.e.
90%) hardware availability you can achieve Four 9s (i.e.
99.99%) availability
SORSOA
SORSOA
Data Center 1
Data Center 2

16. Secret 2: One 9
 With “intelligent” Software that is running on one 9 (i.e.
90%) hardware availability you can achieve Four 9s (i.e.
99.99%) availability
SOA JMS
Server

17. Secret 2: One 9
 With “intelligent” Software that is running on one 9 (i.e.
90%) hardware availability you can achieve Four 9s (i.e.
99.99%) availability
SOA JMS
Server
storage

18. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA

19. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA

20. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA

21. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA

22. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA

23. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA

24. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA

25. Secret 3: Lord protect me from my friends
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA
SOR
2
SOA

26. Secret 3: Lord protect me from my friends
 Fault Containment
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA
SOR
2
SOA

27. Secret 3: Lord protect me from my friends
 Fault Containment
 Fail Fast (Exponential Backoff)
SOR
2
SOR
60
SOR
3
SOR
1
SOA
SOR
2SOA
SOR
2
SOA

28. Secret 4: 240k

29. Secret 4: 240k
 80K regression tests

30. Secret 4: 240k
 80K regression tests
 3x production load testing

31. Secret 4: 240k
 80K regression tests
 3x production load testing
 EDS – Enterprise Data Simulator

32. Secret 5: Cockpit
 Pilot*
* - if one thing you want to take away from this session

33. Secret 5: Cockpit
 Pilot*
 Switches (installing without outage)
* - if one thing you want to take away from this session

34. Secret 6: Wackamole
http://www.youtube.com/watch?v=6apVZWu_1LU

35. Secret 6: Wackamole
 Out of control – CPU will spike, memory will leak,
JVM will crash.
sometimes
^
http://www.youtube.com/watch?v=6apVZWu_1LU

36. Secret 6: Wackamole
 Out of control – CPU will spike, memory will leak,
JVM will crash.
 Strong monitoring & instant reactions
sometimes
^
http://www.youtube.com/watch?v=6apVZWu_1LU

37. Secret 7: .*

38. Secret 7: .*
 Dangerous species

39. Secret 7: .*
 Dangerous species
 Restart Button

40. Secret 7: .*
 Dangerous species
 Restart Button Factory Reset

41. Secret 7: .*
 Dangerous species
 Restart Button
 SODA – Service Oriented Development & Assembly
Factory Reset

42. Secret 7: .*
 Dangerous species
 Restart Button
 SODA – Service Oriented Development & Assembly
 Not ‘So Cool’
– Never chosen technology for the sake of “Coolness”
– But full stack is running almost on Open Source
technologies
Factory Reset

43. Agenda
 High Level Architecture
 Secrets
1. AAA…
2. One 9
3. Lord protect me from my friends
4. 240k
5. Cockpit
6. Wackamole
7. *
 Meta-Secret

44. Meta-Secret: Our Team

45. Thank you!
 Email: ram.lakshmanan@wellsfargo.com
 Blog: http://tier1app.wordpress.com
45