2. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling
3. ETL Tool
Ability to load large amounts of data from XML, flatfiles, CSV
Ability to transform large amounts of data in-memory
Ability to write transformation logic in the tool
Ability to add out-of-box data validity checks and profiling
Easy administration
5. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling
6. Big XML Processing
Large XML (> 1 GB) can be parsed in- memory by caching portions on Ehcache
Multiple files can be processed simultaneously
Ability to optionally use BigMemory to keep parsed portions on disk instead of Heap
Simultaneously process multiple large XML documents
Ehcache
BigMemory
Integration Server
Large XML files
7. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling
8. Big Memory
Capabilities
BigMemory
Terabytes of data in memory, 1000x faster than disk
Scale up on a single machine and out across your machines
Full fault-tolerance –99.999% availability
Benefits
BigMemory
In-memory real-time access to information
Predictable latency at extreme scale
Simple to use on commodity hardware
9. Simple and Predictable Scaling
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Ability to scale to TB of memory with
improved fault tolerance
Support for extended and hybrid storage
Leverage SSD & Flash technologies
Operational flexibility
10. Universal Access
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Broadening the applicability of Terracotta platform
Support for other application development environments/languages
Support .NET/C# and C++ clients
BigMemorySQL
Support integration via lingua franca of data access world -SQL
11. ContinousUptime
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Providing continuous availability of data with zero downtime across different deployment topologies
Purpose-built support for data replication and recovery across WAN
Ensuring Business Continuity across (multi) data centers
12. Core Platform Enhancements
Simple & predictable scaling
Universal Access
Continuous Uptime
Core Platform Enhancements
BigMemory
Platform improvements to support increased performance and operational efficiency
Accessed through APIs
Search Performance – Optimized Pagination
Full Data lifecycle support when no clients connected to BigMemoryServer
13. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling
14. Fast / Bulk writes for large sets
Batch adapters
Transaction management
Connection pooling
Database procedure calls
Table listeners
15. Agenda
Business Case Scenario
Extraction
Enhanced XML
Transformation
Big Memory
Loading
Batching
Monitoring
Performance
Profiling
18. Performance
Read file from disk
2 ms
Iterate through 2 MB XML file
1 ms
Iterate through 96 MB flat file
2 ms
Iterate through 1-2 MB flat file
1 ms
Transformations
less than 1ms -2ms
Acces to Big Memory
1 ms
Batch inserts in 5 tables, 2000 rows in total
600 -800 ms
Batch insert in 2 tables, 6000 rows in total
150 ms
Process 1 XML for for AT end-to-end
10 -25 sec
Process 240 XML files for AT end-to-end on 1 server
2-4 min
Process 90 MB flatfile, 270.000 lines for DE end-to-end on 1 server, 540.000 rows
10 min
Process splitted 90 MB flatfile (200 files), 270.000 lines for DE end-to-end on 1 server, 540.000 rows
3 -4 min