Downloaded 56 times
Uploaded byRyan Hoegg
Building Performant, Reliable, and Scalable Integrations with Mule ESB
The document discusses best practices and case studies for building performant, reliable, and scalable integrations using Mule ESB. Key topics include service level agreements (SLAs) focused on reliability, scalability, and performance tuning, alongside real-world applications in various case studies such as gaming and retail. It emphasizes performance management through observation, tuning, and load generation to meet SLA requirements.
Building Performant, Reliable, and Scalable Integrations with Mule ESB
- 1.
- 2. 2 Building Performant, Reliable, ScalableIntegrations with Mule ESB Ryan Hoegg, IntegrationArchitect,Confluex RupeshRamachandran,SolutionsArchitect, MuleSoft
- 3. 3 Service Level Agreement FirstClass Requirements Precision Matters
- 4.
- 5. 5 Reliability SLAs Availability Uptime Time toRecovery MessageLoss Maximum Guarantee Detection Recovery
- 6.
- 7.
- 8. 8 Performance Tuning: Bigpicture Mule ESB JVM Operating System File System Network Downstream Systems
- 9. 9 Performance: Best Practices Asynchronousvs Synchronous Real-time vs Batch Stateful vs Stateless (web scale) On-Premise vs iPaaS
- 10.
- 11. 11 Performance: Case Study UseCase: API Gateway XML to JSON transforms Mixed Payload Sizes
- 12. 12 Performance: Case Study TestCase Mule ESB 3.5 EE as API Gateway Infrastructure Amazon EC2 with 10GbE network Throughput ~8000tps Latency ~5ms Scale Linear scale out, to 6 boxes*
- 13.
- 14. 14 Making SLAs aReality Prioritize Model Measure
- 15. 15 Tuning: Focusing onWhat Matters Observe Identify hot spot Generate load Compare with SLA
- 16. 16 Tuning: Improve Scalability ScaleUp, Scale Out SEDA Store and Forward Message Oriented Middleware
- 17. 17 Case Study: GamingPlatform Public Beta Launch “Code Complete” Players are unforgiving
- 18. 18 Case Study: GamingPlatform 1. Catalog Services, Estimate Load 2. Prioritize 3. Isolate Muleand Instrument 4. Generate Load 5. Observe 6. Tune
- 19. 19 Case Study: GamingPlatform
- 20. 20 Case Study: GamingPlatform
- 21. 21 Tuning: Improve Reliability ReliableAcquisition Pattern Transactions Retry Delegate
- 22. 22 Case Study: Retail BusinessCritical Integration “Code Complete” Losing Purchase Orders
- 23. 23 Case Study: Retail 1.Determine failure modes 2. Decide how torespond 3. Induce and observe 4. Apply reliability patterns
- 24.
- 25. 25 Questions? Please visit Confluexand MuleSoft experts in the Expo Hall
- 26.
Editor's Notes
- #3 Integration Software is Software
- #4 Someone’s going to define them, intentionally or not Personal Story – emergent properties -> discover in production Instead, determine these requirements along with the functional ones Define terms: Service Level Agreement, originally from telecom, informally used in integration work to describe precise non functional requirements at system boundaries. OLA another term you may see used.
- #5 Opposing Forces Tuning for reliability can negatively impact performance and scalability, … Complexity is the currency
- #6 Risk Management
- #7 Degraded – optional, when useful Introduce Rupesh Ramachandran, Solutions Architect, MuleSoft
- #8 Primary: Response Time TPS Secondary: How many concurrent users can be handled Is performance sustainable over time?
- #9 Peak throughput and response times require tuning of the entire stack. For instance: JVM (heap size, GC algorithm for low GC pause) OS (Linux ulimits, tcp_ip stack, HugePages, etc) File System (avoid excess logging, SSD vs HDD vs NFS) Network (1gig ethernet cannot handle over 125MBps per channel and quickly becomes the bottleneck) Downstream systems: backend systems like DB, SOAP WS, JMS Server, Websphere MQ, etc being integrated by Mule need to be tuned/scaled to perform equally or better than the ESB or it becomes the limiting factor
- #10 Synchronous: Request-response with fast response times – synchronous allows use of single thread per request and avoid context switching overhead. If request takes too long, thread held up for long time, not always on CPU. Sub-optimal core usage. Non-blocking synchronous HTTP (Use Jetty) Asynchronous: Leverages SEDA, provides better CPU utilization for longer running processes. Threads not held up. Real-time: Typically synchronous, but not necessary. All records read into memory and processed at once. Problematic if occasional spikes or large payloads. Batch: Records processed in stages/batches. Like ETL jobs. Stateful: Typically required for long running processes, for H/A via in-memory grid, for state based flow controls like aggregator, scatter-gather, etc Stateless: No state means no persistence overhead. Helps achieve web scale.
- #11 Managing SLA’s depending on consumer Get SLA metrics from API Analytics Works for all HTTP endpoints
- #12 Business critical API. Back end .NET WS and REST service. Payloads 10k – 500k
- #13 8000tps = 650+ million records per day (could go higher if not for 10GbE saturation) Average Latency of 5ms is round-trip from test client. Mule added latency is approx 1ms Scale test was done on smaller boxes. Stopped at 6 because the 10g ethernet was saturated.
- #14 Single MuleESB. 2G heap on a 36GB RAM box. 50% cpu used on a 24 vcpu box. SOAP Proxy use case with data transformation from SOAP to REST.
- #15 Prioritize: Establishing baseline one SLA at a time allows you to protect the most important things first Model: How to generate load? How realistic? Measure: Instrument, Record, apply statistics
- #16 <60 seconds, describe iterative process
- #17 Scale up = additive, Scale out = multiplicative SEDA: focuses threading controls on hot spots Store and Forward: supports out of process SEDA MOM: supports scale out
- #19 Work with project manager to determine the list of services that would go live, and what realistic and peak load might be Rank them by importance and by volume, so we work on the right one first Mock outbound services, instrument with JMX, profiler, network monitoring Model request patterns in a JMeter test plan, execute against QA environment Observe Response Time, Error Rate, CPU, Memory, IO, Threads, Network See next slides