Be the first to like this
Investment banks rely extensively on grids to dramatically increase throughput for their calculations for analytics (especially risk). The traditional design pattern involves executing compute intensive workflows where jobs require movement of large data files to the compute nodes, calculation results creating files which then are again consumed by the next job in the flow. Increasingly, the pattern is shifting to running short lived tasks where the bottleneck is data i.e. the time spent to move data back and forth between compute nodes can be overwhelming - turning a compute bound job to be a IO bound one. For instance, real time pricing for financial derivative instruments could just take a few milliseconds, but, the time required for the data transfer could be hundreds of milliseconds.
The talk focuses on one architectural pattern gaining popularity - move the compute to the data. The data is partitioned in grid memory across many nodes and the compute task is routed to the node with the right data set provisioned based on the data hints it provides during launch.
We discuss the features of the main-memory based data grid solution that uses different data partitioning policies such as hashing or data relationship based to manage data across a large cluster of nodes. We also discuss techniques for rebalancing data and behavior across the Grid nodes to achieve the best throughput and lowest latency.