SWIM - Accelerated Data Movement

1. 1
Copyright Solace
Confidential
SWIM
Accelerated Data Movement
Wayne Osse
Chief Architect
Global Aviation & Transport

2. 2
Copyright Solace
Confidential
o Accelerating Data Movement
o Global Finance versus SWIM
o Lessons Learned with ANSPs
o Data Movement Deployments for SWIM and Global SWIM
Agenda

3. 3
Copyright Solace
Confidential
Services
Accelerating Data Movement – The Message Bus
Enterprise Service Bus
Microservices
Message Bus
o Increase Reliability & Security – speed without safety is meaningless
o Acceleration with High Reliability
‐ High Persistent Message Rates - Rapid Migration with Open Standards
‐ High Persistent Message Fan Out - Accelerate Federation with Bridging
‐ WAN Optimization - Granular access control

4. 4
Copyright Solace
Confidential
o Stage I
Point to Point
o Stage II
Publish-Subscribe
o Stage III
Publish-Subscribe with NextGen Messaging
Greatly increased Reliability and Performance
Open Standards
Multi-Cloud
Global Financial Systems

5. 5
Copyright Solace
Confidential
o Stage I
Point to Point
o Stage II
Publish-Subscribe – why stop here?
OR
Publish-Subscribe with NextGen Messaging
Greatly increased Reliability and Performance
Open Standards
Multi-Cloud
SWIM

6. 6
Copyright Solace
Confidential
‐ Reliability, Security, Manageable, Event Driven
‐ Open APIs: AMQP 1.0 (Qpid), JMS, REST, MQTT (Paho), OpenMAMA
‐ Native APIs: C, .NET, Java, JNI, JavaScript/WebSockets, …
‐ Built-in Federation, Bridging
‐ Cross-API Operation
‐ Disaster Recovery
‐ In Service Upgrades
‐ Performance: throughput, latency, predictability/determinism
‐ Resilience: spikes, slow consumers, fan out
‐ Version Compatibility
‐ On-Premise, Hybrid, Multi-Cloud
‐ World Class Support
Data Movement in Financial Systems

7. 7
Copyright Solace
Confidential
‐ Reliability, Security, Manageable, Event Driven
‐ Open APIs: AMQP 1.0 (Qpid), JMS, REST, [MQTT (Paho)]
‐ [Native APIs: C, .NET, Java, JNI, JavaScript/WebSockets, …]
‐ Built-in Federation, Bridging
‐ Cross-API Operation
‐ Disaster Recovery
‐ In Service Upgrades
‐ Performance: throughput, latency, predictability/determinism
‐ Resilience: spikes, slow consumers, fan out
‐ Version Compatibility
‐ On-Premise, Hybrid[, Multi-Cloud]
‐ World Class Support
Data Movement in SWIM

8. 8
Copyright Solace
Confidential
o Aviation data rates continue to grow rapidly. Data movement technologies must be resilient to
increasing fan out and spikes in data.
o Ecosystem participants in data movement may not be able to upgrade in a timely fashion,
greatly increasing the importance of backward and forward compatibility of message brokers
and APIs.
o Ecosystem participants can behave in slow and unpredictable ways. This cannot affect well-
behaving participants.
o Data Distribution has increased dramatically as data becomes a high value currency. New
technologies require fanning out copies of all messages to more and more participants for audit,
risk, big data, real-time analytics, and machine learning systems.
o Secure WAN traffic optimization is key to aviation ecosystems, especially with external
consumers and publishers of data.
o Cloud technologies are challenging aviation participants. Choices on one platform can restrict
choices on other platforms. As public cloud outages have shown, being multi-platform and
multi-cloud increases redundancy and reliability.
Lessons Learned

9. 9
Copyright Solace
Confidential
o Open standards are critical to provide freedom. Supporting AMQP 1.0, REST and JMS is
important in North American ANSPs, while the Yellow Profile requires AMQP 1.0 and
REST.
o High availability and disaster recovery remain critical, especially as hybrid aviation
systems proliferate.
o Simplified monitoring and management can make aviation ecosystems more agile. Deep
transparency into operational aspects of data movement are critical to projecting
capacity and being empowered to pro-actively address issues before they become
problems.
o Mature data movement technologies provide features that ATM systems can leverage to
reduce bandwidth use (via compression, eliding, filtering and routing), absorb shocks to
the system and provide a highly secure system with granular authorization control.
o Finally, but certainly extremely important, is how a vendor supports its customers. World
class support is absolutely critical for aviation systems that passengers depend on.
Lessons Learned (continued)

10. 10
Copyright Solace
Confidential
SWIM - Node
JMS AMQP 1.0 REST
Microservices
Services
Message Bus

11. 11
Copyright Solace
Confidential
SWIM – Multi-Node (ANSP)
On-Premise
JMS AMQP 1.0 REST
On-Premise
JMS AMQP 1.0 REST
On-Premise
JMS AMQP 1.0 REST
On-Premise
JMS AMQP 1.0 REST
Message Bus
Message Bus Message Bus Message Bus

12. 12
Copyright Solace
Confidential
SWIM – Hybrid and Multi-Cloud
JMS AMQP 1.0 REST
JMS AMQP 1.0 REST
Private Cloud
On-Premise
JMS AMQP 1.0 REST
Public Cloud

13. 13
Copyright Solace
Confidential
Global SWIM
JMS AMQP 1.0 REST
JMS AMQP 1.0 REST
JMS AMQP 1.0 REST
JMS AMQP 1.0 REST
ANSP ANSP
ANSP ANSP

14. 14
Copyright Solace
Confidential
New York
London
Singapore
Tokyo
Chicago
Shanghai
Real-Time Data Sharing Over Long-Distance Networks

15. 15
Copyright Solace
Confidential
Inter-ANSP Federation
Canadian
ANSP
FDPS
AIM_FN
S
STDDS
US
ANSP
FDPS
AIM_FN
S
STDDS
Other
ANSP
FDPS
AIM_FN
S
STDDS
Optimized Communications

16. 16
Copyright Solace
Confidential
Questions

17. 17
Copyright Solace
Confidential
o Federate to simplify the architecture and smarten data as early as possible to enable the ability
to greatly improve efficiency. This also places lower stress on more nodes while empowering
data for optimization as close to the events as possible, so the edge estate can be smaller.
o Separate the message bus from the ESB-like functions in order to harden the highest layer
possible.
o Mediation and transformation are clearly required. But these can be applied at the edge
preferably. This allows for filtering out data close to the source to prevent the waste involved in
sending data that is not needed, consuming valuable resources along the way.
o Data is valuable. It is made more valuable when it can be relied on.
Architectures

18. 18
Copyright Solace
Confidential
Robust: Guaranteed Messaging; How it Works
Publisher Subscriber
If subscriber available,
message is delivered
immediately3b
4
If subscriber is slow
or disconnected, their
backlog is spooled to disk,
delivered as soon as
client is able to receive
Redundant
Mate
• Connected to primary
via two 10 Gbps links
Disk Array
Receipt acknowledged
since message is
guaranteed
3a
Message persisted
in on-board RAM
1
Message and state
replicated to mate,
which confirms receipt
2

19. 19
Copyright Solace
Confidential
M
10M
20M
30M
40M
50M
60M
0K
50K
100K
150K
200K
250K
300K
350K
400K
450K
2.0 4.0 6.0 8.0 11.0 13.0 15.0 17.0 20.0 22.0
Messages
perSecond
Throughput
Messages
inSpool
Message
Spool
Ingress Rate
MinutesElapsed
Spooling, 1/3 Subs Disconnected Recovery Upon Reconnection
Egress Rate
(varies by subscriber
availability)
o Publisher rates not affected by
slow/offline consumers
o Fast consumers not affected in rate or
latency by slow/offline consumers
o Re-connected subscribers “catch up”
without impacting other clients
o Patented technology
Robust: Unmatched in Slow Consumer Handling
20 20 21 20
27 28
30
27
0
10
20
30
40
Pre-Failure Spooling Catchup/Recovery Post-Recovery
Micro-
seconds
of
Latency
(Cut-Through)
Avg 99.9th

20. 20
Copyright Solace
Confidential
SAN
Robust: Fault Tolerant with built-in HA + DR
HA Failover in seconds
‐ regardless of how much data is
queued
‐ Zero message loss
o Appliances deployed as fault
tolerant pairs
o All messages, state and config
replicated to warm standby via high-
speed mate link

21. 21
Copyright Solace
Confidential
SAN
SAN
WAN
Datacenter 1
Datacenter 2
Robust: Fault Tolerant with built-in HA + DR
o HA Failover in seconds
‐ regardless of how much data is queued
‐ Zero message loss
o Fast, Robust Built-in DR
‐ Auto-replicate messages, state, queues and configs
‐ Failover without involving SAN or Network teams
‐ Replicates synchronous and asynchronous flows
‐ Optimized for applications
‐ Doesn’t use Storage replication

22. 22
Copyright Solace
Confidential
Hybrid Cloud Data Movement
Datacenter
Private
Cloud
Legacy
Public
Clouds
JMS MQ