Sonic 7 Hentchel Performance Tuning

3,107 views

Published on

Sonic ESB performance tuning

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,107
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
86
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Hello. A word about my background. Logistics: .
  • Sonic 7 Hentchel Performance Tuning

    1. 1. SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance David Hentchel Principal Solution Engineer
    2. 2. Agenda <ul><li>Methodology : </li></ul><ul><li>Review the recommended approach to project and procedures </li></ul><ul><li>Analysis: </li></ul><ul><li>Understand how to characterize performance requirements and platform capacity </li></ul><ul><li>Testing: </li></ul><ul><li>Learn how to simulate performance scenarios using the Sonic Test Harness </li></ul><ul><li>Tuning: </li></ul><ul><li>Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance
    3. 3. Setting Performance Expectations <ul><li>System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment. </li></ul><ul><li>Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach. </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging D I S C L A I M E R D I S C L A I M E R
    4. 4. Agenda <ul><li>Methodology : </li></ul><ul><li>Review the recommended approach to project and procedures </li></ul><ul><li>Analysis: </li></ul><ul><li>Understand how to characterize performance requirements and platform capacity </li></ul><ul><li>Testing: </li></ul><ul><li>Learn how to simulate performance scenarios using the Sonic Test Harness </li></ul><ul><li>Tuning: </li></ul><ul><li>Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance
    5. 5. Performance Concepts and Methodology <ul><li>Terms and definitions </li></ul><ul><li>Performance engineering concepts </li></ul><ul><li>Managing a performance analysis project </li></ul><ul><ul><li>Skills needed for the project </li></ul></ul><ul><ul><li>Performance Tools </li></ul></ul><ul><ul><li>Project timeline </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    6. 6. Performance Engineering Terms SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging “ System Under Test” “ Test Harness” “ Load” = “Sessions” * “Delivery Rate” “ Latency” = ReceiveTime – SendTime “ Test Components” “ External Components” “ Platform” “ System Metric” R V <ul><li>“ Variable” = </li></ul><ul><ul><li>client param, </li></ul></ul><ul><ul><li>app param, </li></ul></ul><ul><ul><li>system param </li></ul></ul>V V Expert Tip: Limit scope to those test components that are critical to performance and under your control
    7. 7. Concepts: Partitioning Resource Usage SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging <ul><li>Partitionable resources can be broken down as the sum of the contributions of each test component on the system </li></ul><ul><li>Total resource usage is limited by system capacity </li></ul><ul><ul><li>Becomes the bottleneck as it nears 100% </li></ul></ul><ul><ul><li>Goal is linear scalability as additional resource is added </li></ul></ul><ul><ul><li>Vertical versus Horizontal scalability </li></ul></ul><ul><li>Total latency is the sum across all resource latencies, i.e.: </li></ul><ul><ul><li>Latency = CPU_time + Disk_time + Socket_time + wait_sleep </li></ul></ul>% CPU ( Overhead X Message rate ) = Load ∑ (Writes/sec) (msg/sec) (writes/msg) svcs Bottom-Up Rule: Test and tune each component before you test and tune the aggregate.
    8. 8. Concepts: Computer Platform Resources SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Favorite tools: task mgr, perfmon, top, ping –s, traceroute Disk I/O (read/write) Network I/O (send/receive) CPU time Memory (in use, swap) # Threads <ul><li>Use level of detail appropriate to the question being asked </li></ul><ul><li>Machine resource (such as %CPU) artifacts: </li></ul><ul><ul><li>side effects from uncontrolled applications </li></ul></ul><ul><ul><li>timing of refresh interval </li></ul></ul><ul><ul><li>correlation with test intervals </li></ul></ul><ul><li>Scaling across different platforms and resource types </li></ul>
    9. 9. The Performance Engineering Project SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging The Project is Goal Driven Analyze Test Tune <ul><li>For each iteration </li></ul><ul><li>Test performance vs goal </li></ul><ul><li>Identify likeliest area for gain </li></ul><ul><li>Startup tasks </li></ul><ul><li>Define project completion goals </li></ul><ul><li>Staff benchmarking skills </li></ul><ul><li>Acquire test harness </li></ul>Expert Tip: Schedule daily meetings to share results and reprioritize test plan.
    10. 10. Performance Project Skills <ul><li>Requirements Expert </li></ul><ul><ul><li>SLA/QoS levels – minimal & optimal </li></ul></ul><ul><ul><li>Predicted load – current & future </li></ul></ul><ul><ul><li>Distribution topology </li></ul></ul><ul><li>Integration Expert </li></ul><ul><ul><li>Allowable design options </li></ul></ul><ul><ul><li>Cost to deploy </li></ul></ul><ul><ul><li>Cost to maintain </li></ul></ul><ul><li>Testing Expert </li></ul><ul><ul><li>Load generation tool </li></ul></ul><ul><ul><li>Bottleneck diagnosis </li></ul></ul><ul><ul><li>Tuning and optimization </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging SOLUTION I.E. T.E. R.E. Cost/Benefit Load/Distribution Design Options
    11. 11. Tools for a Messaging Benchmark <ul><li>Configurator – creates conditions to bring system under test into known state </li></ul><ul><li>Harness – the platforms and components whose performance response is being measured </li></ul><ul><li>Analyzer – tools and procedures to make meaningful conclusions based on result data. </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging System Under Test Test Harness Test Configurator Test Analyzer Platform Component
    12. 12. Performance Project Timeline SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Development Project Week Process Dev Service Dev Sizing System Test Deployment Plan Launch Performance Project Perf Prototype 1 2 3 4 5 6 7 8 Design Application Tuning
    13. 13. Agenda <ul><li>Methodology : </li></ul><ul><li>Review the recommended approach to project and procedures </li></ul><ul><li>Analysis : </li></ul><ul><li>Understand how to characterize performance requirements and platform capacity </li></ul><ul><li>Testing: </li></ul><ul><li>Learn how to simulate performance scenarios using the Sonic Test Harness </li></ul><ul><li>Tuning: </li></ul><ul><li>Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance
    14. 14. Performance Analysis <ul><li>Performance scenarios : requirements and goals </li></ul><ul><ul><li>Some generic performance scenarios </li></ul></ul><ul><li>System characterization : platforms and architecture </li></ul><ul><li>Test cases : specification for benchmark </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Utilization (% total) Capacity (units/sec) = Efficiency (units/msg) X Load (msg/sec)
    15. 15. Performance Scenario Specification <ul><li>First, triage performance-sensitive processes </li></ul><ul><ul><li>substantial messaging load and latency requirements </li></ul></ul><ul><ul><li>impact of resource-intensive services </li></ul></ul><ul><li>Document only process messaging & services </li></ul><ul><ul><li>leave out Application specific logic – this is a prototype </li></ul></ul><ul><li>Set specific messaging performance goals </li></ul><ul><ul><li>Message rate and size </li></ul></ul><ul><ul><li>Minimum and average latency required </li></ul></ul><ul><li>Try to quantify actual value and risk </li></ul><ul><ul><li>Why this use case matters </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec, and service loads over 10,000 per hour.
    16. 16. Generic Scenario: Decoupled process <ul><li>Asynchronous, loosely coupled, distributed services. </li></ul><ul><li>Assumptions: </li></ul><ul><li>Services allow concurrent, parallel distribution </li></ul><ul><li>Messaging is lightweight, pub sub </li></ul><ul><li>End-to-end process completes in real time </li></ul><ul><li>May be part of a Batch To Real Time pattern </li></ul><ul><li>Factors to analyze: </li></ul><ul><li>Speed and scalability of invoked services </li></ul><ul><li>Distributed topology </li></ul><ul><li>Quality of Service </li></ul><ul><li>Aggregate Latency over time across batched invocations </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: The easiest way to manage decoupled SOA processes is the ESB Itinerary.
    17. 17. Generic Scenario: Real time data feed <ul><li>High speed distribution of real time events </li></ul><ul><li>Assumptions: </li></ul><ul><li>Read-only data pushed dynamically to users </li></ul><ul><li>Messages are small </li></ul><ul><li>Service mediation is simple and fast </li></ul><ul><li>Latency is very important, but QoS needs are modest </li></ul><ul><li>Factors to analyze: </li></ul><ul><li>Quality of Service, esp. worst case for outage and message loss </li></ul><ul><li>Message rate and fanout (pub sub) </li></ul><ul><li>Scalability of consumers </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    18. 18. Generic Scenario: Simple request reply <ul><li>Typical web service call that waits for response </li></ul><ul><li>Assumptions: </li></ul><ul><li>client is blocked, pending response </li></ul><ul><li>small request message, response is larger </li></ul><ul><li>latency is critical </li></ul><ul><li>Factors to analyze: </li></ul><ul><li>Latency of each component service </li></ul><ul><li>Load balancing of key services </li></ul><ul><li>Recovery strategy if loop is interrupted </li></ul><ul><li>Client network, protocol and security specs </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    19. 19. Example: Performance scenario specification <ul><li>Overall project scope: </li></ul><ul><ul><li>Project goals and process </li></ul></ul><ul><ul><li>Deployment environment </li></ul></ul><ul><ul><li>System architecture </li></ul></ul><ul><li>For each Scenario: </li></ul><ul><ul><li>Description of business process </li></ul></ul><ul><ul><li>Operational constraints (QoS, recovery, availability) </li></ul></ul><ul><ul><li>Performance goals, including business benefits </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    20. 20. Characterizing platforms and architecture <ul><li>Scope current and future hardware options available to the project </li></ul><ul><li>Identify geographical locations, firewalls and predefined service hosting restrictions </li></ul><ul><li>Define predefined Endpoints and Services </li></ul><ul><li>Define data models and identify required conversions and transformations. </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    21. 21. Platform configuration specification SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging DMZ Field DMZ CPU number type speed Network bandwidth latency speed Disk type speed Firewall cryptos latency Memory size speed Rule of Thumb: LAN 15 – 150 MBytes / second Disk: 2 – 10 MBytes / second XSLT: 200 – 300 KBytes / second
    22. 22. Platform Profile: Real-time messaging SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging % capacity System resource limitations 90% 70% 20% 5% Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.
    23. 23. Platform Profile: Queued requests SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging % capacity System resource limitations 50% 20% 40% 85% Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.
    24. 24. Architecture Spec: Service distribution <ul><li>Identify services performance characteristics </li></ul><ul><li>Identify high-bandwidth message channels </li></ul><ul><li>Decide which components can be modified </li></ul><ul><li>Annotate with message load estimates </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging ESB ESB ESB Partner ESB Back Office Partner Field Front Office CRM Finance SFA CRM SCM Tracking Service PoS ERP Adapter SCM Adapter Integration Broker
    25. 25. Architecture Spec: Data Integration <ul><li>Approximate the complexity of data schemas </li></ul><ul><li>Identify performance critical transformation events </li></ul><ul><li>Estimate message size </li></ul><ul><li>Identify acceptable potential services </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Data Schemas 1…n Data Schemas 2…m ? Expert Tip: Transform tools vary in efficiency: XSLT – slowest (but most standard) Semantic modeler – generally faster (e.g. Sonic DXSI) Custom text service – fastest, but not as flexible
    26. 26. DEMO: Test lab setup <ul><li>Test hardware </li></ul><ul><ul><li>guidelines for lab computers </li></ul></ul><ul><ul><li>setting up the lab network </li></ul></ul><ul><li>Test architecture </li></ul><ul><ul><li>location of test components </li></ul></ul><ul><ul><li>installation of brokers </li></ul></ul><ul><ul><li>configuration of service containers </li></ul></ul><ul><li>Test design assets </li></ul><ul><ul><li>sample service definitions & wsdls </li></ul></ul><ul><ul><li>sample test documents </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    27. 27. Specifying Test Cases <ul><li>Factors to include: </li></ul><ul><ul><li>Load, sizes, complexity of messaging </li></ul></ul><ul><ul><li>Range of scalability to try (e.g. min/max msg size) </li></ul></ul><ul><ul><li>Basic ESB Process model </li></ul></ul><ul><ul><li>Basic distribution architecture </li></ul></ul><ul><li>Details to avoid: </li></ul><ul><ul><li>Application code (unless readily available) </li></ul></ul><ul><ul><li>Detailed transformation maps </li></ul></ul><ul><li>Define relevant variables: </li></ul><ul><ul><li>Fixed factors </li></ul></ul><ul><ul><li>Test Variables </li></ul></ul><ul><ul><li>Dependent measures </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    28. 28. Typical test variables <ul><li>JMS Client variables: </li></ul><ul><ul><li>Connection / session usage </li></ul></ul><ul><ul><li>Quality of Service </li></ul></ul><ul><ul><li>Interaction mode: </li></ul></ul><ul><ul><li>Message size and shape </li></ul></ul><ul><li>ESB container variables </li></ul><ul><ul><li>Service provisioning and parameters </li></ul></ul><ul><ul><li>Endpoint / Connection parameters </li></ul></ul><ul><ul><li>Process implementation and design </li></ul></ul><ul><ul><li>Routing branch or key field for lookup </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: External JMS client variables are easily managed with the Test Harness.
    29. 29. Example: Test Case Specification <ul><li>For each identified Test Case there is a section specifying the following: </li></ul><ul><li>Overview of test </li></ul><ul><ul><li>How this use case relates to the scenario </li></ul></ul><ul><ul><li>Key decision points being examined </li></ul></ul><ul><li>Functional definition </li></ul><ul><ul><li>How to simulate the performance impact </li></ul></ul><ul><ul><li>Description of ESB processes and services </li></ul></ul><ul><ul><li>Samples messages </li></ul></ul><ul><ul><li>Design alternatives that will be compared </li></ul></ul><ul><li>Test definition </li></ul><ul><ul><li>Variables manipulated </li></ul></ul><ul><ul><li>Variables measured </li></ul></ul><ul><li>Completion criteria </li></ul><ul><ul><li>Throughput and latency goals </li></ul></ul><ul><ul><li>Issues and options that may merit investigation </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    30. 30. Agenda <ul><li>Methodology : </li></ul><ul><li>Review the recommended approach to project and procedures </li></ul><ul><li>Analysis: </li></ul><ul><li>Understand how to characterize performance requirements and platform capacity </li></ul><ul><li>Testing : </li></ul><ul><li>Learn how to simulate performance scenarios using the Sonic Test Harness </li></ul><ul><li>Tuning: </li></ul><ul><li>Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance
    31. 31. Testing Performance <ul><li>Staging test services in the test bed </li></ul><ul><li>Staging brokers and containers </li></ul><ul><li>Configuring the Sonic Test Harness </li></ul><ul><li>Running performance tests and gathering data </li></ul><ul><li>Evaluating results for each test case </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    32. 32. Staging Test Services Deploying existing services <ul><li>Appropriate to use actual implementation of a service IF </li></ul><ul><ul><li>robust implementation exists </li></ul></ul><ul><ul><li>minimal effort to set up in test environment </li></ul></ul><ul><ul><li>no side effects with other test components </li></ul></ul><ul><li>Production ready services merit special treatment: </li></ul><ul><ul><li>perform unit load tests to get baseline </li></ul></ul><ul><ul><li>document possible tuning / scaling options </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    33. 33. Staging Test Services Prototyping proposed new services <ul><li>Prototype should include: </li></ul><ul><ul><li>Correct routing logic for Use Case process </li></ul></ul><ul><ul><li>Approximately correct resource usage </li></ul></ul><ul><ul><li>Generic data </li></ul></ul><ul><li>Prototype does not need: </li></ul><ul><ul><li>Detailed business logic </li></ul></ul><ul><ul><li>Exception handling code </li></ul></ul><ul><ul><li>Invocation of non-critical library calls </li></ul></ul><ul><li>It’s a prototype . Just keep it simple </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    34. 34. Staging Test Services Simulating non-essential services <ul><li>Use ‘stub’ service as placeholder for service step that are not performance-sensitive </li></ul><ul><li>Can return generic data </li></ul><ul><li>Ensures ESB process for target use case will run correctly </li></ul><ul><li>Useful stub services: </li></ul><ul><ul><li>Transform service </li></ul></ul><ul><ul><li>GetFile service </li></ul></ul><ul><ul><li>PassThrough service </li></ul></ul><ul><ul><li>Enrichment service </li></ul></ul><ul><ul><li>Prototype service (version 7.5.2 or later) </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    35. 35. Demo: Provisioning test services SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Status Info WSI: Address Svc Status Request Status Query XForm: Build query DBSvr: Query Test Harness Web Portal Adapter: M/F Callout Addr Info Query Result Enriched Result WSI: Address Svc Status Query Status Query PassThru: DBSvr: Query PassThru: Sleep Status Query Query Result Query Result Business Use Case Performance Test Case
    36. 36. Provisioning test brokers <ul><li>Test broker must be similar to production </li></ul><ul><ul><li>Correct Sonic release and JVM </li></ul></ul><ul><ul><li>Expected deployment heap size and settings </li></ul></ul><ul><li>CAA configuration </li></ul><ul><ul><li>Optimize network for replication channels </li></ul></ul><ul><ul><li>Locate on separate host to avoid bottlenecks </li></ul></ul><ul><ul><li>If failover testing is part of plan: </li></ul></ul><ul><ul><ul><li>define fault tolerant (JNDI) connections </li></ul></ul></ul><ul><li>DRA configuration </li></ul><ul><ul><li>Set up subset of clusters and WAN simulations </li></ul></ul><ul><ul><li>Measure local broker configs first, then expand </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: For each client connection scenario define a named JNDI Connection Factory and document its characteristics.
    37. 37. Provisioning ESB containers <ul><li>Use ESB Container to manage service groups </li></ul><ul><ul><li>name accordingly to service group role </li></ul></ul><ul><ul><li>plan to reshuffle services during tuning phase </li></ul></ul><ul><ul><li>provision jar files out of sonicfs </li></ul></ul><ul><li>Use MF Container to control distribution </li></ul><ul><ul><li>name according to domain/host location </li></ul></ul><ul><ul><li>configure Java heap appropriately </li></ul></ul><ul><ul><ul><li>for IBM jdk make -Xms = -Xmx </li></ul></ul></ul><ul><ul><ul><li>for caching services (e.g. Transform, XML Server), add extra memory for locally-cached data </li></ul></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Default heap size is fine for most ESB containers; if memory is limited, reduce size, but no smaller than: 8MB + ∑ (service jar size) + (#Listeners) * (200KB)
    38. 38. Demo: Setting up containers for test <ul><li>Workbench view of containers </li></ul><ul><ul><li>Coding and debugging the prototype </li></ul></ul><ul><li>Runtime view of containers </li></ul><ul><ul><li>managing the distributed environment </li></ul></ul><ul><ul><li>reinitializing back to a known state </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    39. 39. Simulating endpoint producers and consumers <ul><li>Endpoint protocols and performance </li></ul><ul><ul><li>Test Driver options for various protocols </li></ul></ul><ul><li>Simulating process/thread configuration </li></ul><ul><li>Implementing endpoint interaction modes </li></ul><ul><li>Configuring client Quality of Service (QoS) </li></ul><ul><li>Generating message content </li></ul><ul><li>Demo of client/endpoint simulation </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging System Under Test Test Harness
    40. 40. Endpoint protocols and performance <ul><li>JMS </li></ul><ul><ul><li>fastest client protocol </li></ul></ul><ul><ul><li>strongest range of QoS and Failover </li></ul></ul><ul><li>HTTP </li></ul><ul><ul><li>moderate performance and QoS </li></ul></ul><ul><ul><li>rigid connection model (requires client or router reconnect logic) </li></ul></ul><ul><li>Web Services </li></ul><ul><ul><li>slowest performance </li></ul></ul><ul><ul><li>QoS and recovery depend on WS-* extensions </li></ul></ul><ul><li>File-based </li></ul><ul><ul><li>flat file pickup / dropoff, FTP </li></ul></ul><ul><ul><li>limited to disk speeds (i.e. 1 to 5 MB / sec) </li></ul></ul><ul><ul><li>appropriate for batch processing scenarios </li></ul></ul><ul><li>JCA </li></ul><ul><ul><li>appropriate for EJB server scenarios </li></ul></ul><ul><ul><li>limited to EJB transaction speeds (i.e. 100 to 1000 msg / sec) </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: HTTP acceptors typically achieve ½ to ¼ the performance of comparable JMS/TCP acceptors.
    41. 41. Client session configuration <ul><li>Broker performance depends on scalability of connections and sessions </li></ul><ul><ul><li>JMS best-practice is one thread per session </li></ul></ul><ul><ul><li>JMS sessions can efficiently share a connection </li></ul></ul><ul><ul><li>Use session pool for clients and app servers </li></ul></ul><ul><li>For test simulation: </li></ul><ul><ul><li>determine allowable range of client threads </li></ul></ul><ul><ul><li>test connection/session numbers up to max # threads </li></ul></ul><ul><ul><li>distribute client processes / drivers across multiple machines, if needed, to avoid client-side bottleneck </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Create one connection for every 10 to 20 sessions
    42. 42. Configuring client Quality of Service (QoS) <ul><li>HTTP and Web Services clients </li></ul><ul><ul><li>best possible QoS is at least once </li></ul></ul><ul><ul><li>even with WS-Reliability </li></ul></ul><ul><li>JMS Client: </li></ul><ul><ul><li>CAA w NonPersistentReplicated -> exactly once </li></ul></ul><ul><ul><li>Many shared sub’s versus one durable sub </li></ul></ul><ul><ul><li>NonPersistent Request/Reply -> at least once </li></ul></ul><ul><ul><li>Discardable send to avoid queue backup </li></ul></ul><ul><ul><li>Flow to disk to prevent blocked senders </li></ul></ul><ul><li>ESB service: </li></ul><ul><ul><li>Exactly once uses JMS transaction </li></ul></ul><ul><ul><li>At least once uses client ack </li></ul></ul><ul><ul><li>Best effort uses dups_ok ack </li></ul></ul><ul><li>Broker: </li></ul><ul><ul><li>Sync (default) versus Async disk i/o </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.
    43. 43. Generating message content <ul><li>Simulate message size / distribution for accurate results </li></ul><ul><li>Message content may trigger ESB routing rules </li></ul><ul><li>Some services depend on message content </li></ul><ul><ul><li>key values must match existing data / rules </li></ul></ul><ul><ul><li>duplicate key value could cause error </li></ul></ul><ul><ul><li>services that cache content require accurate key distribution </li></ul></ul><ul><li>Simulating content in the client / driver </li></ul><ul><ul><li>file-based message pool </li></ul></ul><ul><ul><li>message template generation </li></ul></ul><ul><ul><li>Java / object message generator </li></ul></ul><ul><ul><li>Message properties </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Sonic Test Harness supports all these gen random int gen sample xml Addr svc lookup transform rule Status Request priority=2 org=1 Cust Info
    44. 44. Demo: Simulating clients with Test Harness <ul><li>JNDI connection configuration </li></ul><ul><li>Producer / Consumer parameters </li></ul><ul><li>Message generation </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging System Under Test Test Harness Connection Broker JNDI Msg Pool Reply Request Message Generation
    45. 45. Running performance test iterations <ul><li>Logistics of test orchestration </li></ul><ul><ul><li>managing multiple Test Harness clients </li></ul></ul><ul><ul><li>configuring test intervals </li></ul></ul><ul><ul><li>test warm-up and cool-down </li></ul></ul><ul><li>Data collection correlation </li></ul><ul><li>Ensuring repeatability of results </li></ul><ul><li>Demo of Test Harness iterations </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    46. 46. Logistics of test orchestration <ul><li>Managing multiple Test Harness clients </li></ul><ul><ul><li>Simplest option is multiple command windows </li></ul></ul><ul><ul><ul><li>use telnet sessions for remote hosts </li></ul></ul></ul><ul><ul><ul><li>initiate test and warmup </li></ul></ul></ul><ul><ul><ul><li>hit <enter> key in each window </li></ul></ul></ul><ul><ul><li>Advanced environments can use distributed driver: </li></ul></ul><ul><ul><ul><li>Grinder, SLAMD, JMeter, LoadRunner, … </li></ul></ul></ul><ul><li>Configuring test intervals </li></ul><ul><ul><li>long enough to detect trend effects </li></ul></ul><ul><ul><li>short enough to allow fast iteration across tests </li></ul></ul><ul><li>Test warm-up and cool-down </li></ul><ul><ul><li>helps eliminate first-time test artifacts </li></ul></ul><ul><ul><li>ensures steady-state performance numbers </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Start with 10-20 intervals of 30 secs; if steady state is soon reached, reduce to 10 intervals of 10 seconds, plus warmup .
    47. 47. Data collection correlation <ul><li>Test Harness output: </li></ul><ul><ul><li>Throughput (msg/sec) </li></ul></ul><ul><ul><li>Latency (msecs per round trip) </li></ul></ul><ul><ul><li>Message size (bytes) </li></ul></ul><ul><li>System measures: </li></ul><ul><ul><li>CPU usage (%usr, %sys) </li></ul></ul><ul><ul><li>Disk usage (writes/sec) </li></ul></ul><ul><li>Broker metrics: </li></ul><ul><ul><li>Messaging rate (bytes/second) </li></ul></ul><ul><ul><li>Peak queue size (bytes) </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    48. 48. Ensuring repeatability of results <ul><li>Experimental method requirement </li></ul><ul><ul><li>critical in measuring impact of change </li></ul></ul><ul><ul><li>validate by rerunning identical test </li></ul></ul><ul><li>Most common artifacts impacting repeatability </li></ul><ul><ul><li>messages left in queue </li></ul></ul><ul><ul><li>duplicate files dropped in file system </li></ul></ul><ul><ul><li>growing database size / duplicate keys </li></ul></ul><ul><ul><li>disconnected Durable subscribers </li></ul></ul><ul><ul><li>cached Service artifacts (ESB default) </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: Run initial tests twice in succession; if results differ by more than 3% or so, investigate why.
    49. 49. Demo of Test Harness iterations <ul><li>Baseline test </li></ul><ul><li>Change test harness properties </li></ul><ul><li>Rerun test </li></ul><ul><li>Show spreadsheet across tests </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    50. 50. Evaluating performance Measurement against goal <ul><li>Short of goal </li></ul><ul><ul><li>Perform bottleneck analysis / attempt tuning </li></ul></ul><ul><ul><li>Review option of scaling up resources </li></ul></ul><ul><ul><li>Review design change options </li></ul></ul><ul><ul><li>Give up and re-think goal </li></ul></ul><ul><li>Meet or exceed goal </li></ul><ul><ul><li>Continue scaling up and tuning ‘til it breaks </li></ul></ul><ul><ul><li>Give up and declare success </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    51. 51. Evaluating performance Bottleneck analysis <ul><li>Review of resource consumption </li></ul><ul><ul><li>Determine cpu-bound, disk-bound, net-bound </li></ul></ul><ul><ul><li>Identify components using the resource </li></ul></ul><ul><ul><li>Possibility of offloading to other hosts </li></ul></ul><ul><li>Examine trends in scalability tests </li></ul><ul><ul><li>Possibility of improving throughput by adding more client sessions, ESB listeners, clustered brokers, etc. </li></ul></ul><ul><ul><li>Option of rebalancing (# threads, java heap, priority </li></ul></ul><ul><li>Go through Top Ten tuning tips and others… </li></ul><ul><li>Last resort: recode or redesign to save cycles </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    52. 52. Evaluating performance Compare across test runs <ul><li>Carefully planned test runs yield fertile comparisons: </li></ul><ul><ul><li>estimate cost/benefit of a feature or option </li></ul></ul><ul><ul><li>estimate incremental overhead of a tunable parameter </li></ul></ul><ul><ul><li>narrow the field of concerns and alternatives </li></ul></ul><ul><li>Advice in collating and analyzing test runs </li></ul><ul><ul><li>collect test summary results in spreadsheet </li></ul></ul><ul><ul><li>save raw data and logs in a separate place </li></ul></ul><ul><ul><li>save test config, so you can replicate later </li></ul></ul><ul><ul><li>schedule ad hoc review after each test sequence </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: Verify key conclusions by replicating key test runs that led to that conclusion
    53. 53. Demo: Example test result matrix SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Update Scenario Query Scenario Msg Size DB Svc Thruput * Latency ** Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 * kbytes / second ** milliseconds ORX 1 KB 112 331 ORX 10 KB 146 460 ORX 100 KB 152 1197 XSVR 1 KB 121 68 XSVR 10 KB 632 139 XSVR 100 KB 2113 688
    54. 54. Agenda <ul><li>Methodology : </li></ul><ul><li>Review the recommended approach to project and procedures </li></ul><ul><li>Analysis: </li></ul><ul><li>Understand how to characterize performance requirements and platform capacity </li></ul><ul><li>Testing: </li></ul><ul><li>Learn how to simulate performance scenarios using the Sonic Test Harness </li></ul><ul><li>Tuning : </li></ul><ul><li>Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Analyzing, testing and tuning ESB/JMS performance
    55. 55. Performance Tuning with Sonic ESB ® <ul><li>Diagnostics </li></ul><ul><ul><li>Review of ESB architecture </li></ul></ul><ul><ul><li>Factors influencing message throughput </li></ul></ul><ul><ul><li>Factors influencing message latency </li></ul></ul><ul><ul><li>Factors influencing scalability </li></ul></ul><ul><li>Top Ten Tuning Tips </li></ul><ul><li>Other tuning issues </li></ul><ul><ul><li>Broker parameters </li></ul></ul><ul><ul><li>Java tuning </li></ul></ul><ul><ul><li>ESB tuning </li></ul></ul><ul><ul><li>Specialized ESB services </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    56. 56. Review ESB Architecture SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging SOA view System view Router/Switch Router/Switch SERVICES COMMUNICATION BACKBONE NETWORK SERVICE CONTAINER ESB CONTROL Message Log DBMS Cache / Swap Threads VM Threads VM Threads VM Threads VM I/O Controller Memory Network Interface
    57. 57. ESB System Usage: CPU SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging <ul><li>Sources of CPU cost: </li></ul><ul><li>Application code </li></ul><ul><li>XML Parsing </li></ul><ul><li>CPU cost of i/o </li></ul><ul><ul><li>Network sockets </li></ul></ul><ul><ul><li>Log/Data disks </li></ul></ul><ul><li>Web Service protocol </li></ul><ul><ul><li>Web Service security </li></ul></ul><ul><li>Security </li></ul><ul><ul><li>Authorization </li></ul></ul><ul><ul><li>Message or channel encryption </li></ul></ul>
    58. 58. ESB System Usage: Disk SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging <ul><li>Sources of Disk overhead: </li></ul><ul><li>Database services </li></ul><ul><li>Large File / Batch services </li></ul><ul><li>Message recovery log </li></ul><ul><ul><li>Might not be used if other guarantee mechanisms work </li></ul></ul><ul><li>Message data store </li></ul><ul><ul><li>Disconnected consumer </li></ul></ul><ul><ul><li>Queue save threshold overflow </li></ul></ul><ul><li>Flow to disk overflow </li></ul><ul><li>Message storage overhead depends on disk sync behavior </li></ul><ul><ul><li>Explicit file synchronization ensures data retained on crash </li></ul></ul><ul><ul><li>Tuning options at disk, filesystem, JVM and Broker levels </li></ul></ul>Expert Tip: To estimate best-case message log performance, use the DiskPerf utility bundled with Test Harness.
    59. 59. ESB System Usage: Network SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging <ul><li>Sources of Network overhead: </li></ul><ul><li>Client application messages and replies </li></ul><ul><li>Service to service steps within the ESB (except intra-container ) </li></ul><ul><li>ESB Web Service callouts </li></ul><ul><li>CAA broker replication messages </li></ul><ul><li>Metadata (JMX, cluster, DRA) messages (normally < 1%) </li></ul><ul><li>Computing network bandwidth: </li></ul><ul><ul><li>Network card: 100 mbit ~= 12 MB/sec, 1 gbit ~= 120 MB/sec </li></ul></ul><ul><ul><li>Network switches are individually rated </li></ul></ul><ul><li>Computing network load: </li></ul><ul><ul><li>message rate X message size </li></ul></ul><ul><ul><li>include response messages and intermediate steps </li></ul></ul><ul><ul><li>add ack packets (very small) for each message send </li></ul></ul>Expert Tip: To estimate best-case network performance, use the DiskPerf utility bundled with Test Harness.
    60. 60. Tip #1: Increase sender and listener threads to make service & client scale <ul><li>Increase # Listeners for key entry Endpoints </li></ul><ul><li>Add more Service/Process instances </li></ul><ul><li>Warning: Intra-Container ignores endpoint </li></ul><ul><ul><li>split scalable service into separate container </li></ul></ul><ul><ul><li>turn intra-container messaging off </li></ul></ul><ul><ul><li>note: sub-Process is always intra-container </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Container2 ServiceX … Container1 ServiceX … … Expert Tip: Stop increasing Listeners when CPU usage nears maximum acceptable.
    61. 61. Tip #2: Implement optimal QoS to balance speed versus precision SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging (Based on CAA brokers and fault-tolerant connections) ESB QoS MQ Setting Message Loss Events Duplicate Msg Events N/A Discardable Buffer overflow, Any crash Never Best Effort NonPersistent, DupsOK ack Broker crash, Client crash Never At Least Once Persistent, DupsOK ack Never Never Exactly Once Transacted Never Never Expert Tip: With CAA configured, Best Effort service is equivalent to At Least Once, with substantially lower overhead.
    62. 62. Tip #3: Re-use JMS objects to reduce setup costs <ul><li>Objects with client and broker footprint: </li></ul><ul><ul><li>Connection </li></ul></ul><ul><ul><li>Session </li></ul></ul><ul><ul><li>Sender/Receiver/Publisher/Subscriber </li></ul></ul><ul><ul><li>Temporary destination </li></ul></ul><ul><li>Tuning strategies: </li></ul><ul><ul><li>Reuse JMS objects in client code </li></ul></ul><ul><ul><li>Share each Connection across sessions </li></ul></ul><ul><ul><li>Share Sessions across Producers and Consumers </li></ul></ul><ul><ul><ul><li>but not across JVM Threads </li></ul></ul></ul><ul><ul><li>For low-load topics/queues </li></ul></ul><ul><ul><ul><li>Use Anonymous Producer </li></ul></ul></ul><ul><ul><ul><li>Use wildcard or multi-topic subscription </li></ul></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Up to 500 queues per Broker and 10,000 topics per broker.
    63. 63. Tip #4: Use intra-container service calls to avoid broker hops SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Inter-Container Messaging Intra-Container Messaging v7.5: better! faster! Broker Dispatch Outbox Marshall Msg Send Msg … … Dispatch Outbox Unmarshall Msg Marshall Msg Call onMessage Send Msg … Receive Msg Unmarshall Msg Call onMessage Instantiate Proc …
    64. 64. Tip #5: Use NonPersistentReplicated mode to reduce disk overhead <ul><li>Normal broker mechanisms require disk sync </li></ul><ul><ul><li>contributes to latency across the board </li></ul></ul><ul><ul><li>interferes with batching of packets </li></ul></ul><ul><ul><li>limited bandwidth </li></ul></ul><ul><li>Disabling disk sync eliminates this overhead </li></ul><ul><ul><li>Send mode NonPersistentReplicated </li></ul></ul><ul><ul><li>Optional broker params to disable entirely </li></ul></ul><ul><ul><li>WARNING: Log-based recovery will lose recent messages </li></ul></ul><ul><ul><li>BUT: CAA failover will not </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Network overhead for Replication channel is about ½ the Persistent Msg load of the broker.
    65. 65. Tip #6: Use XCBR instead of CBR to eliminate Javascript overhead <ul><li>CBR rules implemented via Javascript </li></ul><ul><ul><li>dynamic change with complex rules </li></ul></ul><ul><ul><li>very high overhead for runtime engine </li></ul></ul><ul><li>XCBR rules extract data fields for comparison </li></ul><ul><ul><li>only simple comparisons supported </li></ul></ul><ul><ul><li>no script engine overhead </li></ul></ul><ul><ul><li>use message property data key for best effect </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: Invocation of the Rhino javascript engine requires about 100 milliseconds cpu time on a typical platform.
    66. 66. Tip #7: Use message batching to accelerate message streams <ul><li>Message transfer overhead is generally fixed </li></ul><ul><li>Hidden ack messages amenable to tuning: </li></ul><ul><ul><li>AsyncNonPersistent mode decouples ack latency </li></ul></ul><ul><ul><li>Transaction Commit allows 1 ack per N messages </li></ul></ul><ul><ul><li>DupsOK ack mode allows ‘lazy’ ack from consumer </li></ul></ul><ul><ul><li>Pre-Fetch Count allows batched transmit to consumer </li></ul></ul><ul><li>ESB Design option: send one multi-part message instead of N individual messages </li></ul><ul><li>XML transforms and other services handle multi-record data efficiently </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Producer Consumer Broker Msg Msg … Ack Msg Msg Ack …
    67. 67. Tip #8: Minimize XML/SOAP operations to avoid parsing overhead <ul><li>Bypass SOAP and Web Services processing </li></ul><ul><ul><li>Use HTTP Direct Basic instead of SOAP or WS </li></ul></ul><ul><ul><li>Risk of invalid XML if source is unreliable </li></ul></ul><ul><li>Combine multiple XML parsing steps into one </li></ul><ul><ul><li>Save target XPath results as Message props </li></ul></ul><ul><ul><li>Also relevant for BPEL correlation ID’s </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Input Message XML Transform Custom JAXB XCBR JAXB Service Input Message propY=9 propX=1 Input Message JAXB Obj Msg part
    68. 68. Tip #9: Use high-speed encryption to reduce security overhead <ul><li>Default SSL encryption uses old RCA stack </li></ul><ul><ul><li>At least 2X slower than more modern options </li></ul></ul><ul><li>Change to any JSSE compliant stack: </li></ul><ul><ul><li>modify client DSSL_PROVIDER_CLASS to: progress.message.net.ssl.jsse.jsseSSLImpl </li></ul></ul><ul><ul><li>change broker SSL provider from RSA to JSSE </li></ul></ul><ul><li>Use more efficient cipher suites: </li></ul><ul><ul><li>RSA_With_Null_MD5 is the smallest and fastest </li></ul></ul><ul><li>Reduce broker memory overhead by deleting any unused ciphers </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    69. 69. Tip #10: Use stream API's to improve large message performance <ul><li>SonicMQ Recoverable File Channels </li></ul><ul><ul><li>Uses JMS layer to manage large file xfer </li></ul></ul><ul><ul><li>Queue-based initiation of transfer </li></ul></ul><ul><ul><li>High-speed JMS pipeline for blocks of data </li></ul></ul><ul><ul><li>Recovery continues at point interrupted </li></ul></ul><ul><li>Sonic ESB open-source Large Message Service </li></ul><ul><ul><li>Provides dynamic provisioning </li></ul></ul><ul><ul><li>Interacts with ESB processes </li></ul></ul><ul><li>SonicStream API (version 7.5 or later) </li></ul><ul><ul><li>Topic-based, pipeline into Java Stream api </li></ul></ul><ul><ul><li>No recovery </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    70. 70. Broker Tuning Parameters <ul><li>Core Resources: </li></ul><ul><ul><li>JVM heap size </li></ul></ul><ul><ul><li>JVM thread, stack limits </li></ul></ul><ul><ul><li>DRA, HTTP and Transaction threads </li></ul></ul><ul><ul><li>TCP settings </li></ul></ul><ul><li>Message storage: </li></ul><ul><ul><li>Queue size and save threshold </li></ul></ul><ul><ul><li>Pub/sub buffers </li></ul></ul><ul><ul><li>Message log and store </li></ul></ul><ul><li>Message management </li></ul><ul><ul><li>Encryption </li></ul></ul><ul><ul><li>Flow control and flow-to-disk </li></ul></ul><ul><ul><li>Dead message queue management </li></ul></ul><ul><li>Connections </li></ul><ul><ul><li>Mapping to NIC’s </li></ul></ul><ul><ul><li>Timeouts </li></ul></ul><ul><ul><li>Load balancing </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: For non-trivial queues, multiply default settings by 10 to 100.
    71. 71. Java Tuning Options <ul><li>‘ Fastest’ JVM depends a little on the application and a lot on the platform </li></ul><ul><li>VM heap needs to be large enough to process load, but small enough to avoid system swapping </li></ul><ul><li>Garbage Collection: </li></ul><ul><ul><li>default settings good for optimal throughput </li></ul></ul><ul><ul><li>use advanced (jdk4 or later) GC options to optimize worst-case latency </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Rule of Thumb: On windows platforms, the Sun 1.5.0 JVM is 10% to 50% slower than the default IBM 1.4.2 JVM.
    72. 72. ESB Tuning Options <ul><li>Load balancing and scalability of services: </li></ul><ul><ul><li>number of distributed service instances </li></ul></ul><ul><ul><li>number of service listener threads </li></ul></ul><ul><li>Container Java VM heap size </li></ul><ul><li>Intra-Container messaging </li></ul><ul><li>Endpoint and connection parameters </li></ul><ul><ul><li>same principles as JMS client </li></ul></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: Start with small Java heap and only increase -Xms size if it improves performance.
    73. 73. Discussion of Service tuning <ul><li>Transformations </li></ul><ul><li>XML Server </li></ul><ul><li>BPEL Server </li></ul><ul><li>Database Service </li></ul><ul><li>DXSI Service </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
    74. 74. Other fun things you can tune <ul><li>Database: indexing, query optimization </li></ul><ul><li>SOA patterns: federated query, temporal aggregation, split/join, caching </li></ul><ul><li>XML: DOM, SAX, XStream </li></ul><ul><li>Disk: Device balancing, RAID, mount params </li></ul><ul><li>Network: Nagle algorithm, timeouts </li></ul>SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Expert Tip: If you save service resources in sonicfs , the ESB container will dynamically load and cache them.
    75. 75. <ul><li>SonicMQ Performance Tuning Guide </li></ul><ul><li>Benchmarking Enterprise Messaging Systems white paper </li></ul><ul><li>Sonic Test Harness User Guide </li></ul><ul><li>Progress Professional Services </li></ul><ul><ul><li>Developer training courses </li></ul></ul><ul><ul><li>Sonic Performance Analysis package </li></ul></ul>Other Performance Engineering Resources SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging No “Magic Bullet”, but plenty of places you can go for info:
    76. 76. SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Questions?
    77. 77. SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging Thank you for your time
    78. 78. SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging

    ×