Tuning and Troubleshooting OpenSplice DDS Applications

3,410 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,410
On SlideShare
0
From Embeds
0
Number of Embeds
29
Actions
Shares
0
Downloads
80
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Tuning and Troubleshooting OpenSplice DDS Applications

  1. 1. Tuning and Troubleshooting OpenSplice DDS ApplicationsOpenSplice DDS Angelo Corsaro, PhD Hans van’t Hag. Chief Technology Officer Product Manager OMG DDS Sig Co-Chair OMG DDS Co-Author PrismTech PrismTech angelo.corsaro@prismtech.com hans.vanthag@prismtech.com
  2. 2. OpenSplice DDS US OSUM 2013http://www.prismtech.com/events/opensplice-user-meeting-usa-2013 Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  3. 3. Part I -- Setting the ContextOpenSplice DDS
  4. 4. Kinds of Issues When building distributed applications there are several kinds of issues you can Copyright  2011,  PrismTech  –    All  Rights  Reserved. encounter. Some of the most common issues are often: ☐ Connectivity: “things” don’t seem to talk to each other ☐ Performance:the latency is too high or throughput is too lowOpenSplice DDS ☐ Scalability: your system does not scale as necessary ☐ Resource Utilization: your application is using way too much CPU, Memory, and/or Network This webcast explores common issues in DDS applications and demonstrates how the tools provided by OpenSplice DDS can their diagnose and resolution
  5. 5. OpenSplice DDS Connectivity
  6. 6. Connectivity Issues Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ DDS provides dynamic discovery thus as there is network connectivity applications that are intended to communicate should discover each other and just work!OpenSplice DDS ☐ What may be the cause of connectivity issues in DDS? ☐ The network is not working (did you plug the wire?) ☐ Your applications does not share the same domain (thus your mistake) ☐ The QoS of the DataWriter and the DataReader don’t match
  7. 7. OpenSplice DDS QoS Mismatch?
  8. 8. Tip #8 Understand the QoS Model DURABILITY LIVELINESS DEST. ORDER TIME-BASED FILTER ☐ DDS defines 22 QoS HISTORY OWENERSHIP PARTITION RESOURCE LIMITS policies that can be LIFESPAN OWN. STRENGTH PRESENTATION Copyright  2011,  PrismTech  –    All  Rights  Reserved. RELIABILITY DW LIFECYCLE applied to USER DATA DEADLINE DR LIFECYCLE communication entities TOPIC DATA LATENCY BUDGET ENTITY FACTORY to control their local as GROUP DATA TRANSPORT PRIOOpenSplice DDS well as end-to-end behaviour RxO QoS Local QoS Most of the QoS Policies that control an end-to-end property follow the so- called Request vs. Offered (RxO) Model based on which the QoS requested by the Consumer should not exceed the QoS Provided by the Producer. If there is an RxO mismatch no communication will take place
  9. 9. OpenSplice DDS
  10. 10. OpenSplice DDS Performance
  11. 11. Latency / Throughput ☐ DDS provides several QoS Policies to control latency and Copyright  2011,  PrismTech  –    All  Rights  Reserved. throughput ☐ OpenSplice provides extensions such as Streams to efficiently distribute small data (e.g. several millions samples per second)OpenSplice DDS ☐ OpenSplice Tools such TimeBasedFilter Deadline as the Tuner can help find the right QoS [Inbound] setting for achieving Throughput LatencyBudget Latency the desired level or [Outbound] latency/throughput TransportPriority
  12. 12. OpenSplice DDS Scalability
  13. 13. Federated / Standalone Deployment Application Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. OpenSplice DDS OpenSplice DDS OpenSplice DDS Operating OperatingOpenSplice DDS Systems Systems Federated Deployment Standalone Deployment ☐ Performance ☐ Usability ☐ Scalability ☐ Simple Configuration ☐ Determinism ☐ Simple Deployment
  14. 14. Deployment Configuration Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The deployment mode can be changed by a simple configuration parameter. No With OpenSplice DDS recompilation or re-linking! you can choose the deployment ☐ <SingleProcess>true</SingleProcess> architecture that makesOpenSplice DDS the most sense for your ☐ The deployment options can be mixed at use case. will (even within a single computing node) ☐ The same application can be deployed in “Simple when Sufficient. Performant when Required” federated and standalone mode (even on the same system)
  15. 15. Deployment Configuration Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The deployment mode can be changed by a simple configuration parameter. No With OpenSplice DDS recompilation or re-linking! you can choose the deployment ☐ <SingleProcess>true</SingleProcess> architecture that makesOpenSplice DDS the most sense for your ☐ The deployment options can be mixed at use case. will (even within a single computing node) ☐ The same application can be deployed in “Simple when Sufficient. Performant when Required” federated and standalone mode (even on the same system)
  16. 16. Resource UtilizationOpenSplice DDS
  17. 17. Shared Memory SizingOpenSplice DDS
  18. 18. Shared Memory Size ☐ OpenSplice DDS shared memory size is defined in its Copyright  2011,  PrismTech  –    All  Rights  Reserved. configuration file. The size defined by the default configuration file is 10MBytes Beware that different OS have different limitations w.r.t.OpenSplice DDS ☐ the maximum shared memory segment that can be allocated ☐ If you want to go beyond the OS limits you need to change the configuration of your kernel
  19. 19. Linux ☐ The default value for the maximum shared memory segment is 32MBytes Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ This default can be changed in several ways (1) Adding this line to your /etc/rc.d/rc.local file:OpenSplice DDS echo “your_max_shared_memory_size” > /proc/sys/kernel/shmmax (2) Changing the settings for the sys-limits (save changes on /etc/ sysctl.conf to maintain them across reboots): $ sysctl -w kernel.shmmax=yourMaxValue
  20. 20. Windows Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The default maximum size for Shared Memory segments on Windows is 2GB ☐ To exend it, say to 3GB, add the /3GB the boot.ini as shown below:OpenSplice DDS [boot loader] timeout=30 default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS [operating systems] multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Windows NT Workstation Version 4.00" /3GB
  21. 21. Understand Topic KeysOpenSplice DDS
  22. 22. Topic Types & Keys Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Topic Types can define some of their attributes as keysOpenSplice DDS ☐ Yet, even when a Topic type does not define a key the keylist directive has to be provided -- just to tell the IDL compiler that this is a topic
  23. 23. Topic Types & Keys Keyful Keyless Copyright  2011,  PrismTech  –    All  Rights  Reserved. enum TemperatureScale { CELSIUS, FAHRENHEIT, struct UniversalConstants { KELVIN double PI; }; double e; // Nepero Number double g; // Gravitational Constant double NA; // Avogadro NumberOpenSplice DDS struct TempSensorType { short id; double F; // Faraday Constant float temp; double K; // Boltzman Constant float hum; double c; // Light Speed TemperatureScale scale; }; }; #pragma keylist UniversalConstants #pragma keylist TempSensorType id
  24. 24. Topic Instances Copyright  2011,  PrismTech  –    All  Rights  Reserved. enum TemperatureScale { CELSIUS, FAHRENHEIT, Writer Reader KELVIN };OpenSplice DDS 2 26.0 70.0 CELSIUS 1 26.0 70.0 CELSIUS struct TempSensorType { short id; 1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS float temp; float hum; TemperatureScale scale; }; #pragma keylist TempSensorType
  25. 25. Topic Instances Copyright  2011,  PrismTech  –    All  Rights  Reserved. enum TemperatureScale { CELSIUS, FAHRENHEIT, Writer Reader KELVIN };OpenSplice DDS struct TempSensorType { 2 26.0 70.0 CELSIUS 1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS short id; 1 26.0 70.0 CELSIUS float temp; float hum; TemperatureScale scale; }; #pragma keylist TempSensorType id
  26. 26. Instance Management ☐ Each topic instance takes Copyright  2011,  PrismTech  –    All  Rights  Reserved. resources (e.g. memory) Writer Reader ☐ If you don’t choose carefully your key, you may end-up 2 26.0 70.0 CELSIUSOpenSplice DDS 1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS creating an instance per 1 26.0 70.0 CELSIUS sample... ☐ In addition, instances can be explicitly disposed in order to reclaim resources
  27. 27. Example struct Counter { short cid; long count; }; #pragma keylist Counter ???OpenSplice DDS
  28. 28. Example struct Counter { short cid; long count; }; #pragma keylist Counter cidOpenSplice DDS
  29. 29. Example struct SpatialSensor { short x; short y; float v; };OpenSplice DDS #pragma keylist SpatialSensor ???
  30. 30. Example struct SpatialSensor { short x; short y; float v; };OpenSplice DDS #pragma keylist SpatialSensor
  31. 31. Topic Instances Lifecycle struct VehiclePosition { string plate; long x; long y; Copyright  2011,  PrismTech  –    All  Rights  Reserved. }; com.myco.VPos #pragma keylist VehiclePosition plateOpenSplice DDS struct VehiclePosition { string plate; long x; long y; }; #pragma keylist VehiclePosition com.myco.VPos
  32. 32. Topic Instances Lifecycle struct VehiclePosition { string plate; New long x; “A01” 100 200 long y; Copyright  2011,  PrismTech  –    All  Rights  Reserved. }; com.myco.VPos #pragma keylist VehiclePosition plateOpenSplice DDS struct VehiclePosition { string plate; long x; long y; New }; #pragma keylist VehiclePosition “A01” 100 200 com.myco.VPos
  33. 33. Topic Instances Lifecycle struct VehiclePosition { string plate; New long x; “A01” 100 200 Copyright  2011,  PrismTech  –    All  Rights  Reserved. long y; }; New “B41” 57 31 #pragma keylist VehiclePosition plate com.myco.VPosOpenSplice DDS struct VehiclePosition { string plate; long x; long y; }; #pragma keylist VehiclePosition “B41” 57 31 “A01” 100 200 com.myco.VPos
  34. 34. Topic Instances Lifecycle struct VehiclePosition { string plate; New long x; “A01” 110 210 “A01” 100 200 long y; Copyright  2011,  PrismTech  –    All  Rights  Reserved. }; New #pragma keylist VehiclePosition “B41” 57 31 plate com.myco.VPosOpenSplice DDS struct VehiclePosition { string plate; long x; long y; }; #pragma keylist VehiclePosition “A01” 110 210 “B41” 57 31 “A01” 100 200 com.myco.VPos
  35. 35. Topic Instances Lifecycle New “A01” 120 220 “A01” 110 210 “A01” 100 200 Copyright  2011,  PrismTech  –    All  Rights  Reserved. New “B41” 57 31 com.myco.VPosOpenSplice DDS “A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200 com.myco.VPos
  36. 36. Topic Instances Lifecycle New “A01” 120 220 “A01” 110 210 “A01” 100 200 Copyright  2011,  PrismTech  –    All  Rights  Reserved. New “B41” 47 19 “B41” 57 31 com.myco.VPosOpenSplice DDS “B41” 47 19 “A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200 com.myco.VPos
  37. 37. Topic Instances Lifecycle New “A01” 120 220 “A01” 110 210 “A01” 100 200 Copyright  2011,  PrismTech  –    All  Rights  Reserved. New d Dispose “B41” - - “B41” 47 19 “B41” 57 31 com.myco.VPosOpenSplice DDS “B41” is ed D47pos19 “A01” 120 220 “A01” 110 210 “B41” 57 31 “A01” 100 200 com.myco.VPos
  38. 38. Part II -- DemoesOpenSplice DDS
  39. 39. Content☐ Tools ☐ Configuration: OpenSplice Configurator ☐ Black-box System (regression) Testing: OpenSplice Tester ☐ White-box application Inspection & Tuning: OpenSplice Tuner Copyright  2013,  PrismTech  –    All  Rights  Reserved. ☐ Recording & Replay: OpenSplice RnR ☐ Shared-memory and Network monitoring: OpenSplice mmStat and ‘Wireshark’ ™☐ Common Issues ☐ System Configuration (deployment architecture, networking-setup) ☐ System Deployment (resource usage/watermarks, statistics, reports, tracing) ☐ Tool Deployment (intrusiveness: local versus remote operation)☐ Demo’s ☐ Tester: system browser (SHMEM vs. SP) and QoS conflict detection (Reliability) ☐ Tuner: Statistics monitoring & QoS Tuning (Latency Budget) ☐ Tester: 1-click capturing & real-time Charting (virtual attributes, scripting) ☐ RnR: scripted recording & replay with RnR-service and Tester ☐ RnR: graphical control with RnRManager
  40. 40. OpenSplice Tools
  41. 41. Total Control: OpenSplice Configurator™ Copyright  2013,  PrismTech  –    All  Rights  Reserved.☐ The reference tool for configuring OpenSplice DDS☐ Rich online guide to configuration options☐ Context help and parameter validation☐ 100% Java
  42. 42. OpenSplice Tester™☐ “Black-Box” DDS system testing☐ Automated testing of DDS systems ☐ Dynamic Discovery of DDS Entities ☐ Domain-specific scripting languages Copyright  2013,  PrismTech  –    All  Rights  Reserved. ☐ Batch execution of regression tests☐ Debugging of distributed DDS systems ☐ System browser of DDS participants ☐ Connectivity & QoS conflict monitoring ☐ One-click definition of monitoring timeline ☐ Analysis/comparison of topic data ☐ Virtual topic-attributes to ease analysis ☐ Statistics monitoring ☐ 1-click spawning of Tuner to ‘attach’ to a remote process / federation☐ Integrated IDE ☐ Syntax highlighting editor ☐ One-click relations between script/logs/timeline
  43. 43. Total Control: OpenSplice Tuner™☐ “Whitebox” debug/tuning Tool ☐ Looking ‘inside’ a federation and/or application ☐ Different perspectives Copyright  2013,  PrismTech  –    All  Rights  Reserved. (participant, topic, partition)☐ Monitoring & Tuning ☐ Inspect and Tune the app’s DDS entities ☐ Make snapshots of reader-caches ☐ Detect and resolve QoS Mismatch ☐ Inspect Statistics☐ Reading & Writing ☐ Read/Write data for arbitrary topics☐ Import & Export ☐ Inject Topic Definitions ☐ export and import XML-based reader/writer snapshots
  44. 44. OpenSplice RnR☐ Dynamic recording of any topic- Copyright  2013,  PrismTech  –    All  Rights  Reserved. data in a DDS system☐ Selective replay with variable speed☐ Distributed control by topic-based API☐ Seamless integration with OpenSplice Tester (topic-based API) RnR RnR RnR Service Service(s) Manager☐ Dedicated RnR-Manager graphical GUI for scenario-definition and data Any topic Record/Replay command & status topics Record/Replay command & status topics import/analysis OpenSplice DDS
  45. 45. Wireshark☐ WireShark Packet Dissector Copyright  2013,  PrismTech  –    All  Rights  Reserved.☐ Watch what goes on the wire☐ Inspect DDSI-RTPS and/or RT-Networking packets being exchanged between applications☐ Native RTnetworking dissector support available in source-distribution ☐ OpenSpliceDDSV6.2.1HDEx86.win32toolswireshark-pluginsospl (see README)
  46. 46. mmStat Copyright  2013,  PrismTech  –    All  Rights  Reserved.mmstat -hmmstat [-M|m] [-e] [-a] [-i interval] [-s sample_count] [URI]mmstat [-t|T] [-i interval] [-s sample_count] [-l limit] [-f filter_expression] [URI] Mode: -m Show memory statistics (default mode) -M Show memory statistics difference -t Show meta object references -T Show meta object references difference -h Show this help -e Extended mode, shows bar for allocated memory -a Show pre-allocated memory as well. -i interval Display interval (in milliseconds) -s sample_count Stop after sample_count samples -l limit Show only object count >= limit -f filter_expr Show only meta objects which name passes the filter expression
  47. 47. Common Issues: Examples & Demo’s
  48. 48. 11Common issues: System Configuration☐ Selecting the proper deployment architecture ☐ Federated / shared-memory (for scalability and data/application lifecycle decoupling) Copyright  2013,  PrismTech  –    All  Rights  Reserved. ☐ Standalone / single-process (for simplicity yet at the cost of ‘network-scheduling determinism’)☐ Selecting the proper networking architecture ☐ RTNative networking: ‘push’ based architecture using pre-configured (multicast) network-partitions ☐ Pro : zero discovery times (optional topology-discovery only needed for fast disconnect-reaction times) ☐ Con : push implies data might show-up at places where there’s no actual/current interest .. Yet tunable in OpenSplice ☐ DDSI networking: ‘pull’ based architecture using advertized locators ☐ Pro : selective distribution to ONLY the set of discovered participants, using unicast whenever just 1 destination ☐ Con : (standardized) discovery-protocol has scalability limitations (amount of heartbeats) .. Yet tunable in OpenSplice☐ Configuring your domain ☐ Using “OpenSplice Configurator” to aid in correct configurations ☐ Assure system-wide consistent values ☐ Domain-ID’s and in case of RTNetworking the related port-numbers (channels and discovery) ☐ RTNetworking only: NetworkPartition definitions (names and related [multicast-] addresses)
  49. 49. Common issues: System Deployment 13☐ How to Prevent Issues ☐ Using the Configurator to validate configurations [demo] Copyright  2013,  PrismTech  –    All  Rights  Reserved. ☐ Assuring system-wide consistent networking/durability setup ☐ Using ResourceLimits policies in your applications☐ How to detect Issues ☐ Setting resource-usage ‘watermarks’ with the configurator [demo] ☐ Checking ospl_info.log and ospl_error.log files [demo] ☐ Checking QoS RxO conflicts with the Tester and/or Tuner [demo] ☐ Capturing behavior with Tester (and [auto-]creation of scripts to reproduce) ☐ Recording of information using RnR (manually and/or scripted) [demo]☐ How to resolve issues ☐ Runtime Tuning of QoS policies with the Tuner [demo] ☐ Exploring reader/writer/service statistics with Tester & Tuner [demo]
  50. 50. Demo’s
  51. 51. Demo’s☐ Environment ☐ Perf1/2/3 reference-system (shmem versus SP) ☐ Ishapes demo-application ☐ Configurator / Tester / Tuner / RnR tools Copyright  2013,  PrismTech  –    All  Rights  Reserved.☐ Configurator ☐ Used configuration-files (SHMEM/RnR, SHMEM/Statistics, SP/Resource-limits ☐ Demo-config: perf1: RnR added, perf2: statistics added perf3: resource-limits☐ Tuner/Tester ‘ishapes’ demos ☐ Tester: system browser (SHMEM vs. SP) ☐ Tester: QoS conflict detection (Reliability) ☐ Tuner: QoS Tuning (Latency Budget) ☐ Tuner: Status & Statistics (Writer/Reader, perf2: Network-queues, perf3: Network receiving) ☐ Tester: real-time Charting (virtual attributes, scripting)☐ Recording & Replay ☐ Scripting with Tester ☐ Controlling with RnRManager
  52. 52. Demo setup remote Tester-2 Remote Remote Splice- Tuner 1 Tuner 2 Hans Office-net Copyright  2013,  PrismTech  –    All  Rights  Reserved. Router Switch Test-net Tester-1/ Ishapes Ishapes Ishapes RnRManager Federated Federated standalone Shared Memory Shared Memory S N O DDS E Asoap RT-Net RnR / Dur. Soap RT-Net Durability P libs T W Perf1 Perf2 Perf3 Switch Perf-net
  53. 53. Teaser: ‘Streams’☐ Bundled Example Copyright  2013,  PrismTech  –    All  Rights  Reserved.☐ Streaming 10 mln. Samples☐ Throughput: up to 29 mln. Samples/s ! (shmem/i7)
  54. 54. Configuration Copyright  2013,  PrismTech  –    All  Rights  Reserved.☐ Perf1 ☐ Ospl_shmem_native_rnr.xml☐ Perf2 ☐ Ospl_shmem_native_statistics.xml☐ Perf3 ☐ Ospl_sp_native_resource_limits.xml
  55. 55. QoS conflict detectionCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  56. 56. QoS TuningCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  57. 57. Statistics MonitoringCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  58. 58. Data-CaptureCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  59. 59. Scripted RnRCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  60. 60. Graphical RnRCopyright  2013,  PrismTech  –    All  Rights  Reserved.
  61. 61. OpenSplice DDS
  62. 62. References Copyright  2011,  PrismTech  –    All  Rights  Reserved. OpenSplice | DDS Escalier ¥ #1 OMG DDS Implementation ¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS ¥ Open Source ¥ Open Source ¥ Open Source ¥ www.opensplice.org ¥ www.scala-lang.org ¥ github.com/kydos/escalierOpenSplice DDS ¥ Simple C++ API for DDS ¥ DDS-PSM-Java for OpenSplice DDS ¥ DDS-based Advanced Distributed ¥ Open Source ¥ Open Source Algorithms Toolkit ¥ github.com/kydos/simd-cxx ¥ github.com/kydos/simd-java ¥ Open Source ¥ github.com/kydos/dada
  63. 63. :: Connect with Us :: ¥opensplice.com ¥forums.opensplice.org ¥@acorsaro ¥opensplice.org ¥opensplicedds@prismtech.com ¥@prismtechOpenSplice DDS ¥ crc@prismtech.com ¥sales@prismtech.com ¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro

×