Speed Dialing the Enterprise

1. BPM –Speed Dialing the Enterprise
October 9th, 2014
BPM Forum

2. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling

3. ETL Tool
Ability to load large amounts of data from XML, flatfiles, CSV
Ability to transform large amounts of data in-memory
Ability to write transformation logic in the tool
Ability to add out-of-box data validity checks and profiling
Easy administration

5. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling

6. Big XML Processing
Large XML (> 1 GB) can be parsed in- memory by caching portions on Ehcache
Multiple files can be processed simultaneously
Ability to optionally use BigMemory to keep parsed portions on disk instead of Heap
Simultaneously process multiple large XML documents
Ehcache
BigMemory
Integration Server
Large XML files

7. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling

8. Big Memory
Capabilities
BigMemory
Terabytes of data in memory, 1000x faster than disk
Scale up on a single machine and out across your machines
Full fault-tolerance –99.999% availability
Benefits
BigMemory
In-memory real-time access to information
Predictable latency at extreme scale
Simple to use on commodity hardware

9. Simple and Predictable Scaling
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Ability to scale to TB of memory with
improved fault tolerance
Support for extended and hybrid storage
Leverage SSD & Flash technologies
Operational flexibility

10. Universal Access
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Broadening the applicability of Terracotta platform
Support for other application development environments/languages
Support .NET/C# and C++ clients
BigMemorySQL
Support integration via lingua franca of data access world -SQL

11. ContinousUptime
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Providing continuous availability of data with zero downtime across different deployment topologies
Purpose-built support for data replication and recovery across WAN
Ensuring Business Continuity across (multi) data centers

12. Core Platform Enhancements
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Platform improvements to support increased performance and operational efficiency
Accessed through APIs
Search Performance – Optimized Pagination
Full Data lifecycle support when no clients connected to BigMemoryServer

13. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling

14. Fast / Bulk writes for large sets
Batch adapters
Transaction management
Connection pooling
Database procedure calls
Table listeners

15. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling

16. Monitoring –Process Visibilty

17. Triggering and scheduling
On Demand
Scheduled
Event driven
• HTTP(S)
• FTP(S)
• SOAP
• JMS
• Scheduler
• SMTP
• BPM Event
• Java Client
• C/C++ Client
• .NET Client
• Adapter
• FLOW

18. Performance
Read file from disk
2 ms
Iterate through 2 MB XML file
1 ms
Iterate through 96 MB flat file
2 ms
Iterate through 1-2 MB flat file
1 ms
Transformations
less than 1ms -2ms
Acces to Big Memory
1 ms
Batch inserts in 5 tables, 2000 rows in total
600 -800 ms
Batch insert in 2 tables, 6000 rows in total
150 ms
Process 1 XML for for AT end-to-end
10 -25 sec
Process 240 XML files for AT end-to-end on 1 server
2-4 min
Process 90 MB flatfile, 270.000 lines for DE end-to-end on 1 server, 540.000 rows
10 min
Process splitted 90 MB flatfile (200 files), 270.000 lines for DE end-to-end on 1 server, 540.000 rows
3 -4 min