Performance Test Driven Development with Oracle Coherence
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Performance Test Driven Development with Oracle Coherence

on

  • 3,484 views

Slides from Coherence SIG London,18 July 2013

Slides from Coherence SIG London,18 July 2013

Statistics

Views

Total Views
3,484
Views on SlideShare
1,551
Embed Views
1,933

Actions

Likes
4
Downloads
37
Comments
0

16 Embeds 1,933

http://blog.ragozin.info 1743
http://orana.info 124
http://cloud.feedly.com 35
http://www.inoreader.com 6
http://www.newsblur.com 5
http://digg.com 3
http://feedly.com 2
http://reader.aol.com 2
http://newsblur.com 2
http://inoreader.com 2
http://7735872642513631302_ed89ccdf3335f3c17e92a5ed92b23872a9f351ca.blogspot.com 2
http://127.0.0.1 2
https://reader.aol.com 2
http://www.goread.io 1
http://www.ranksit.com 1
http://plus.url.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Performance Test Driven Development with Oracle Coherence Presentation Transcript

  • 1. Test Driven Development with Oracle Coherence Alexey Ragozin London, 18 Jul 2013
  • 2. Presentation outline • Motivation • PTDD philosophy • Oracle Coherence under test  Coherence Vs. Testing  Small cluster Vs. Big cluster  Areas to keep an eye on • Automation challenge • Common pitfalls of performance testing
  • 3. This code works Filter keyFilter = new InFilter(new KeyExtractor(), keySet); EntryAggregator aggregator = new Count(); Object result = cache.aggregate(keyFilter, aggregator); ValueExtractor[] extractors = { new PofExtractor(String.class, TRADE_ID), new PofExtractor(String.class, SIDE), new PofExtractor(String.class, SECURITY), new PofExtractor(String.class, CLIENT), new PofExtractor(String.class, TRADER), new PofExtractor(String.class, STATUS), }; MultiExtractor projecter = new MultiExtractor(extractors); ReducerAggregator reducer = new ReducerAggregator(projecter); Object result = cache.aggregate(filter, reducer);
  • 4. This code also works public static class MyNextExpiryExtractor implements ValueExtractor { @Override public Object extract(Object obj) { MyPorfolio pf = (MyPorfolio) obj; long nextExpiry = Long.MAX_VALUE; for(MyOption opt: pf.getOptions()) { if (opt.getExpiry() < nextExpiry) { nextExpiry = opt.getExpiry(); } } return nextExpiry; } @Override public String toString() { return getClass().getSimpleName(); } }
  • 5. And this also looks Ok @LiveObject public static class MyLiveObject implements PortableObject { // ... @EventProcessorFor(events={EntryInsertedEvent.class}) public void inserted( EventDispatcher dispatcher, EntryInsertedEvent event) { DefaultCommandSubmitter.getInstance() .submitCommand( "subscription", new MySubscribeCommand(event.getEntry().getKey())); } }
  • 6. Another slide to scare you API Cache service Packet publisher Packet speaker OS Packet listener Packet receiver OS Service thread Worker thread Packet receiver Packet publisher Packet speaker Packet listener Packet receiver Service thread Cache service Packet publisher Packet speaker Packet listener API Service thread Packet receiver Packet listener OS OS Packet speaker Packet publisher Service thread Worker thread Serialization Deserialization Client thread Approximate sequence diagram for cache get operation
  • 7. Functional Vs. Fast  You have paid for Coherence  You have paid for gigabytes of RAM  You have spent time developing solution and you want to be REALLY FAST  Do not be a hostage of your beliefs  Test early  Test often
  • 8. PTTD Philosophy Working cycle  Write simplest functional code  Benchmark it  Improve based on test measurements Never optimize unless you can measure outcome Never speculate, measure Saves time and improves work/life balance 
  • 9. Testing Coherence Challenges  Cluster required  Sensitive to network  Database is usually part of solution  Massive parallel load generation required
  • 10. Testing Coherence Benefits  Pure Java, less hurdle with OS tweaking etc  Nodes are usually plain J2SE processes  you can avoid app server setup pain  No disk persistence  managing data is usually hardest part of test setup
  • 11. Benchmarking and cluster size Single node cluster may reveal  server side processing issues Small cluster 2-8 physical servers  latency related problems  scalability anomalies  partitioning anomalies Large cluster > 8 physical servers  my terra incognita, so far
  • 12. Areas to keep eye on Extractors, queries, indexes • query index usage • query plans for complex filters • POF extractors Server side processing • backing map listeners • storage side transformations • cross cache access Network • effective network throughput Capacity • large messages in cluster • Coherence*Extend buffers Mixed operation loads • cache service thread pool saturation • cache service lock contention Scale out • broadcast requests • hash quality • 100% utilization of network thread
  • 13. Automation “Classic” approach  bash + SSH + log scraping Problems  not reusable  short “half-live” of test harness  Java and bash/awk is totally different skill set
  • 14. Automation Stock performance test tools  Deployment are not covered  Often “web oriented”  Insufficient performance of tool
  • 15. Automation – Game changer cloud = CloudFactory.createSimpleSshCloud(); cloud.node("cbox1"); cloud.node("cbox2"); cloud.node("cbox3"); cloud.node("**").touch(); // Say hello cloud.node("**").exec(new Callable<Void>() { @Override public Void call() throws Exception { String jvmName = ManagementFactory.getRuntimeMXBean().getName(); System.out.println("My name is '" + jvmName + "'. Hello!"); return null; } });
  • 16. Automation – Game changer NanoCloud - http://code.google.com/p/gridkit/wiki/NanoCloudTutorial • Managing slave nodes  in-process, local JVM process, remote JVM process • Deploy free remote execution • Classpath management  automatic master classpath replication  include/exclude classpath elements • Pain free master/slave communications • Just works! 
  • 17. Automation – Game changer Full stack • Maven – ensure test portability • Nanocloud – painless remote execution • JUnit – test enter point • Java – all test logic staring nodes, starting clients, load generation, result processing … •Java – all test logic • Jenkins – execution scheduling
  • 18. Simple microbench mark @Before public void setupCluster() { // Simple cluster configuration template // Single host cluster config preset cloud.all().presetFastLocalCluster(); cloud.all().pofEnabled(true); cloud.all().pofConfig("benchmark-pof-config.xml"); // DSL for cache config XML generation DistributedScheme scheme = CacheConfig.distributedSheme(); scheme.backingMapScheme(CacheConfig.localScheme()); cloud.all().mapCache("data", scheme); // Configuring roles cloud.node("storage*").localStorage(true); cloud.node("client").localStorage(false); // Storage nodes will run as separate processes cloud.node("storage*").outOfProcess(true); } *https://github.com/gridkit/coherence-search-common/blob/master/src/test/java/org/gridkit/coherence/search/bench/FilterPerformanceMicrobench.java
  • 19. Simple microbench mark @Test public void verify_full_vs_index_scan() { // Tweak JVM arguments for storage nodes JvmProps.addJvmArg(cloud.node("storage-*"), "|-Xmx1024m|-Xms1024m|-XX:+UseConcMarkSweepGC"); // Generating data for benchmark // ... cloud.node("client").exec(new Callable<Void>() { @Override public Void call() throws Exception { NamedCache cache = CacheFactory.getCache("data"); System.out.println("Cache size: " + cache.size()); calculate_query_time(tagFilter); long time = TimeUnit.NANOSECONDS.toMicros(calculate_query_time(tagFilter)); System.out.println("Exec time for [tagFilter] no index - " + time); // ... return null; } }); } *https://github.com/gridkit/coherence-search-common/blob/master/src/test/java/org/gridkit/coherence/search/bench/FilterPerformanceMicrobench.java
  • 20. Monitoring  Server CPU usage  Process CPU usage  Network bandwidth usage  Coherence threads CPU usage  Packet Publisher/Speaker/Receiver  Cache service thread  Cache service thread pool  Coherence MBeans  Cache service task backlog  TCMP, *Extend IO throughput etc
  • 21. Flavors of testing  Distributed micro benchmarks  Performance regression tests  Bottlenecks analyzing  Performance sign off
  • 22. Flavors of testing  Distributed micro benchmarks • Micro benchmark using real cluster • Proving ideas • To be run manually be developer  Performance regression tests  Bottlenecks analyzing  Performance sign off
  • 23. Flavors of testing  Distributed micro benchmarks  Performance regression tests • To be run by CI • Execute several stable test scenarios • Fixed load scenarios, not for stress testing • GOAL: track impact of code changes • GOAL: keep test harness compatible with code base  Bottlenecks analyzing  Performance sign off
  • 24. Flavors of testing  Distributed micro benchmarks  Performance regression tests  Bottlenecks analyzing • Testing through N-dimensional space of parameters • Fully autonomous execution of all test grid !!! • Analyzing correlation to pin point bottle neck • To be performed regularly to prioritize optimization • Also used to measure/prove effect of optimizations  Performance sign off
  • 25. Flavors of testing  Distributed micro benchmarks  Performance regression tests  Bottlenecks analyzing  Performance sign off • Execution performance acceptance tests aligned to release goals • Activity driven by QA • Can share infrastructure with dev team owned tests
  • 26. Flavors of testing  Distributed micro benchmarks  Performance regression tests  Bottlenecks analyzing  Performance sign off
  • 27. Common pit falls  Measuring “exception generation” performance  always validate operation results  write functional test on performance tests  Fixed user Vs. Fixed request rate  serious problems may go unnoticed  Ignoring environment health and side load
  • 28. Common pit falls Fixed user Vs. Fixed request frequency Fixed users  5 threads  5 operations out of 50k will fall out of time envelop  99 percentile would be ~1ms Fixed request rate  300k operation in total  250k around 1 ms  50k between 1ms and 10 s  99 percentile would be ~9.4 s Case:  Operation mean time: 1ms  Throughput: 5k ops/s  Test time: 1 minute  GC pause 10 seconds in middle of run
  • 29. Links • Nanocloud  http://code.google.com/p/gridkit/wiki/NanoCloudTutorial • ChTest – Coherence oriented wrapper for Nanocloud  http://code.google.com/p/gridkit/wiki/ChTest  http://blog.ragozin.info/2013/03/chtest-is-out.html  https://speakerdeck.com/aragozin/chtest-feature-outline • GridKit @ GitHub  https://github.com/gridkit
  • 30. Thank you Alexey Ragozin alexey.ragozin@gmail.com http://blog.ragozin.info - my blog about JVM, Coherence and other stuff