Monte Carlo on GPUs


Published on

A presentation about NAG\'s work for Monte Carlo simulations on GPUs systems. For an audience in the Finance industry.

Published in: Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Monte Carlo on GPUs

  1. 1. Monte Carlo Simulation and its Efficient Implementation Robert Tong <ul><li>28 January 2010 </li></ul>
  2. 2. Outline <ul><li>Why use Monte Carlo simulation? </li></ul><ul><li>Higher order methods and convergence </li></ul><ul><li>GPU acceleration </li></ul><ul><li>The need for numerical libraries </li></ul>
  3. 3. Why use Monte Carlo methods? <ul><li>Essential for high dimensional problems – many degrees of freedom </li></ul><ul><li>For applications with uncertainty in inputs </li></ul><ul><li>In finance </li></ul><ul><ul><li>Important in risk modelling </li></ul></ul><ul><ul><li>Pricing/hedging derivatives </li></ul></ul>
  4. 4. The elements of Monte Carlo simulation <ul><li>Derivative pricing </li></ul><ul><ul><li>Simulate a path of asset values </li></ul></ul><ul><ul><li>Compute payoff from path </li></ul></ul><ul><ul><li>Compute option value </li></ul></ul><ul><li>Numerical components </li></ul><ul><ul><li>Pseudo-random number generator </li></ul></ul><ul><ul><li>Discretization scheme </li></ul></ul>
  5. 5. <ul><li>In the past </li></ul><ul><ul><li>Faster solution has been provided by increasing processor speeds </li></ul></ul><ul><ul><li>Want a quicker solution? Buy a new processor </li></ul></ul><ul><li>Present </li></ul><ul><ul><li>Multi-core/Many-core architectures, without increased processor clock speeds </li></ul></ul><ul><ul><li>A major challenge for existing numerical algorithms </li></ul></ul><ul><li>The escalator has stopped... or gone into reverse! </li></ul><ul><ul><li>Existing codes may well run slower on multi-core </li></ul></ul>The demand for ever increasing performance
  6. 6. Ways to improve performance in Monte Carlo simulation <ul><li>Use higher order discretization </li></ul><ul><li>Keep low order (Euler) discretization – </li></ul><ul><ul><li> make use of multi-core potential </li></ul></ul><ul><ul><li>e.g. GPU (Graphics Processing Unit) </li></ul></ul><ul><li>Use high order discretization on GPU </li></ul><ul><li>Use quasi-random sequence (Sobol’, …) and Brownian Bridge </li></ul><ul><li>Implement Sobol’ sequence and Brownian Bridge on GPU </li></ul>
  7. 7. Higher order methods – 1 (work by Kai Zhang, University of Warwick, UK)
  8. 8. Higher order methods – 2
  9. 9. Numerical example – 1
  10. 10. Numerical example – 1a
  11. 11. Numerical example – 1b
  12. 12. Numerical example – 2a
  13. 13. Numerical example – 2b
  14. 14. GPU acceleration <ul><li>Retain low order Euler discretization </li></ul><ul><li>Use multi-core GPU architecture to achieve speed-up </li></ul>
  15. 15. The Emergence of GPGPU Computing <ul><li>Initially – computation carried out by CPU (scalar, serial execution) </li></ul><ul><li>CPU </li></ul><ul><ul><li>evolves to add cache, SSE instructions, ... </li></ul></ul><ul><li>GPU </li></ul><ul><ul><li>added to speed graphics display – driven by gaming needs </li></ul></ul><ul><ul><li>multi-core, SIMT, limited flexibility </li></ul></ul><ul><li>CPU and GPU move closer </li></ul><ul><ul><li>CPU becomes multi-core </li></ul></ul><ul><ul><li>GPU becomes General Purpose (GPGPU) – fully programmable </li></ul></ul>
  16. 16. Current GPU architecture – e.g. NVIDIA Tesla
  17. 17. Tesla – processing power <ul><li>SM – Streaming Multiprocessor </li></ul><ul><ul><li>8 X SP - Streaming Processor core </li></ul></ul><ul><ul><li>2 X Special Function Unit </li></ul></ul><ul><ul><li>MT – Multithreaded instruction fetch and issue unit </li></ul></ul><ul><ul><li>Instruction cache </li></ul></ul><ul><ul><li>Constant cache (read only) </li></ul></ul><ul><ul><li>Shared memory (16 Kb, read/write) </li></ul></ul><ul><li>C1060 – adds double precision </li></ul><ul><ul><li>30 double precision cores </li></ul></ul><ul><ul><li>240 single precision cores </li></ul></ul>
  18. 18. Tesla C1060 memory (from: M A Martinez-del-Amor et al. (2008) based on E Lindholm et al. (2008))
  19. 19. Programming GPUs – CUDA and OpenCL <ul><li>CUDA (Compute Unified Device Architecture, developed by NVIDIA) </li></ul><ul><ul><li>Extension of C to enable programming of GPU devices </li></ul></ul><ul><ul><li>Allows easy management of parallel threads executing on GPU </li></ul></ul><ul><ul><li>Handles communication with ‘host’ CPU </li></ul></ul><ul><li>OpenCL </li></ul><ul><ul><li>Standard language for multi-device programming </li></ul></ul><ul><ul><li>Not tied to a particular company </li></ul></ul><ul><ul><li>Will open up GPU computing </li></ul></ul><ul><ul><li>Incorporates elements of CUDA </li></ul></ul>
  20. 20. First step – obtaining and installing CUDA <ul><li>FREE download from </li></ul><ul><li>See: Quickstart Guide </li></ul><ul><li>Require: </li></ul><ul><ul><li>CUDA capable GPU – GeForce 8, 9, 200, Tesla, many Quadro </li></ul></ul><ul><ul><li>Recent version of NVIDIA driver </li></ul></ul><ul><ul><li>CUDA Toolkit – essential components to compile and build applications </li></ul></ul><ul><ul><li>CUDA SDK – example projects </li></ul></ul><ul><li>Update environment variables (Linux default shown) </li></ul><ul><ul><li>PATH /usr/local/cuda/bin </li></ul></ul><ul><ul><li>LD_LIBRARY_PATH /usr/local/cuda/lib </li></ul></ul><ul><li>CUDA compiler nvcc works with gcc (Linux) MS VC++ (Windows) </li></ul>
  21. 21. Host (CPU) – Device (GPU) Relationship <ul><li>Application program initiated on Host (CPU) </li></ul><ul><li>Device ‘kernels’ execute on GPU in SIMT (Single Instruction Multiple Thread) manner </li></ul><ul><li>Host program </li></ul><ul><ul><li>Transfers data from Host memory to Device (GPU) memory </li></ul></ul><ul><ul><li>Specifies number and organisation of threads on Device </li></ul></ul><ul><ul><li>Calls Device ‘kernel’ as a C function, passing parameters </li></ul></ul><ul><ul><li>Copies output from Device back to Host memory </li></ul></ul>
  22. 22. Organisation of threads on GPU <ul><li>SM (Streaming Multiprocessor) manages up to 1024 threads </li></ul><ul><li>Each thread is identified by an index </li></ul><ul><li>Threads execute as Warps of 32 threads </li></ul><ul><li>Threads are grouped in blocks (user specifies number of threads per block) </li></ul><ul><li>Blocks make up a grid </li></ul>
  23. 23. Memory hierarchy <ul><li>On device can </li></ul><ul><ul><li>Read/write per-thread </li></ul></ul><ul><ul><ul><li>Registers </li></ul></ul></ul><ul><ul><ul><li>Local memory </li></ul></ul></ul><ul><ul><li>Read/write per-block shared memory </li></ul></ul><ul><ul><li>Read/write per-grid global memory </li></ul></ul><ul><ul><li>Read only per-grid constant memory </li></ul></ul><ul><li>On host (CPU) can </li></ul><ul><ul><li>Read/write per-grid </li></ul></ul><ul><ul><ul><li>Global memory </li></ul></ul></ul><ul><ul><ul><li>Constant memory </li></ul></ul></ul>
  24. 24. CUDA terminology <ul><li>‘ kernel’ – C function executing on the GPU </li></ul><ul><ul><li>__global__ declares function as a kernel </li></ul></ul><ul><ul><ul><li>Executed on the Device </li></ul></ul></ul><ul><ul><ul><li>Callable only from the Host </li></ul></ul></ul><ul><ul><ul><li>void return type </li></ul></ul></ul><ul><ul><li>__device__ declares function that is </li></ul></ul><ul><ul><ul><li>Executed on the Device </li></ul></ul></ul><ul><ul><ul><li>Callable only from the Device </li></ul></ul></ul>
  25. 25. Application to Monte Carlo simulation <ul><li>Monte Carlo paths lead to highly parallel algorithms </li></ul><ul><li>Applications in finance e.g. simulation based on SDE (return on asset) </li></ul><ul><li>drift + Brownian motion </li></ul><ul><li>Requires fast pseudorandom or </li></ul><ul><li>Quasi-random number generator </li></ul><ul><li>Additional techniques improve efficiency: Brownian Bridge, stratified sampling, … </li></ul>
  26. 26. Random Number Generators: choice of algorithm <ul><li>Must be highly parallel </li></ul><ul><li>Implementation must satisfy statistical tests of randomness </li></ul><ul><li>Some common generators do not guarantee randomness properties when split into parallel streams </li></ul><ul><li>A suitable choice: MRG32k3a (L’Ecuyer) </li></ul>
  27. 27. MRG32k3a: skip ahead <ul><li>Generator combines 2 recurrences: </li></ul><ul><li>Each recurrence of form (M Giles, note on implementation) </li></ul><ul><li>Precompute in operations on CPU, </li></ul>
  28. 28. MRG32k3a: modulus <ul><li>Combined and individual recurrences </li></ul><ul><li>Can compute using double precision divide – slow </li></ul><ul><li>Use 64 bit integers (supported on GPU) – avoid divide </li></ul><ul><li>Bit shift – faster (used in CPU implementations) </li></ul><ul><li>Note: speed of different possibilities subject to change as NVIDIA updates floating point capability </li></ul>
  29. 29. MRG32k3a: memory coalescence <ul><li>GPU performance limited by memory access </li></ul><ul><li>Require memory coalescence for fast transfer of data </li></ul><ul><li>Order RNG output to retain consecutive memory accesses </li></ul><ul><li>is stored at </li></ul><ul><li>sequential ordering </li></ul><ul><li>(Implementation by M Giles) </li></ul>
  30. 30. MRG32k3a: single – double precision <ul><li>L’Ecuyer’s example implementation in double precision floating point </li></ul><ul><li>Double precision on high end GPUs – but arithmetic operations much slower in execution than single precision </li></ul><ul><li>GPU implementation in integers – final output cast to double </li></ul><ul><li>Note: output to single precision gives sequence that does not pass randomness tests </li></ul>
  31. 31. MRG32k3a: GPU benchmarking – double precision <ul><li>GPU – NVIDIA Tesla C1060 </li></ul><ul><li>CPU – serial version of integer implementation running on single core of quad-core Xeon </li></ul><ul><li>VSL – Intel Library MRG32k3a </li></ul><ul><li>ICC – Intel C/C++ compiler </li></ul><ul><li>VC++ – Microsoft Visual C++ </li></ul>GPU CPU-ICC CPU-VC++ VSL-ICC VSL-VC++ Samples/ sec 3.00E+09 3.46E+07 4.77E+07 9.35E+07 9.32E+07
  32. 32. MRG32k3a: GPU benchmarking – single precision <ul><li>Note: for double precision all sequences were identical </li></ul><ul><li>For single precision GPU and CPU identical </li></ul><ul><li>GPU and VSL differ </li></ul><ul><li>max abs err 5.96E-008 </li></ul><ul><li>Which output preferred? </li></ul><ul><li>use statistical tests of randomness </li></ul>GPU CPU-ICC CPU-VC++ VSL-ICC VSL-VC++ Samples/ sec 3.49E+09 3.58E+07 5.24E+07 1.02E+08 9.75E+07
  33. 33. LIBOR Market Model on GPU <ul><li>Equally weighted portfolio of 15 swaptions each with </li></ul><ul><li>same maturity, but different lengths and </li></ul><ul><li>different strikes </li></ul>
  34. 34. Numerical Libraries for GPUs <ul><li>The problem </li></ul><ul><ul><li>The time-consuming work of writing basic numerical components should not be repeated </li></ul></ul><ul><ul><li>The general user should not need to spend many days writing each application </li></ul></ul><ul><li>The solution </li></ul><ul><ul><li>Standard numerical components should be available as libraries for GPUs </li></ul></ul>
  35. 35. NAG Routines for GPUs
  36. 36. nag_gpu_mrg32k3a_uniform
  37. 37. nag_gpu_mrg32k3a_uniform
  38. 38. nag_gpu_mrg32k3a_uniform
  39. 39. Example program: generate random numbers on GPU <ul><li>... // Allocate memory on Host host_array = (double *)calloc(N,sizeof(double)); // Allocate memory on GPU cudaMalloc((void **)&device_array, sizeof(double)*N); // Call GPU functions // Initialise random number generator nag_gpu_mrg32k3a_init(V1, V2, offset); </li></ul><ul><li>// Generate random numbers nag_gpu_mrg32k3a_uniform(nb, nt, np, device_array); // Read back GPU results to host cudaMemcpy(host_array,gpu_array,sizeof(double)*N,cudaMemcpyDeviceToHost); ... </li></ul>
  40. 40. NAG Routines for GPUs
  41. 41. nag_gpu_mrg32k3a_next_uniform
  42. 42. nag_gpu_mrg32k3a_next_uniform
  43. 43. Example program – kernel function <ul><li>__global__ void mrg32k3a_kernel(int np, FP *d_P){ </li></ul><ul><li>unsigned int v1[3], v2[3]; </li></ul><ul><li>int n, i0; </li></ul><ul><li>FP x, x2 = nanf(&quot;&quot;); </li></ul><ul><li>// initialisation for first point </li></ul><ul><li>nag_gpu_mrg32k3a_stream_init(v1, v2, np); </li></ul><ul><li>// now do points </li></ul><ul><li>i0 = threadIdx.x + np*blockDim.x*blockIdx.x; </li></ul><ul><li>for (n=0; n<np; n++) { </li></ul><ul><li>nag_gpu_mrg32k3a_next_uniform(v1, v2, x); </li></ul><ul><li>} </li></ul><ul><li>d_P[i0] = x; </li></ul><ul><li>i0 += blockDim.x; </li></ul><ul><li>} </li></ul>
  44. 44. Library issues: Auto-tuning <ul><li>Performance affected by mapping of algorithm to GPU via threads, blocks and warps </li></ul><ul><li>Implement a code generator to produce variants using the relevant parameters </li></ul><ul><li>Determine optimal performance </li></ul><ul><li>Li, Dongarra & Tomov (2009) </li></ul>
  45. 45. <ul><li>Working with Fixed Income Research & Strategies Team (FIRST) </li></ul><ul><ul><li>NAG mrg32k3a works well in BNP Paribas CUDA “Local Vol Monte-Carlo” </li></ul></ul><ul><ul><li>Passes rigorous statistical tests for randomness properties (Diehard, Dieharder,TestU01) </li></ul></ul><ul><ul><li>Performance good </li></ul></ul><ul><ul><li>Being able to match the GPU random numbers with the CPU version of mrg32k3a has been very valuable for establishing validity of output </li></ul></ul>Early Success with BNP Paribas
  46. 46. BNP Paribas Results – local vol example
  47. 47. <ul><li>“ The NAG GPU libraries are helping us enormously by providing us with fast, good quality algorithms. This has let us concentrate on our models and deliver GPGPU based pricing much more quickly.” </li></ul>And with Bank of America Merrill Lynch
  48. 48. <ul><li>“ Thank you for the GPU code, we have achieved speed ups of x120” </li></ul><ul><li>In a simple uncorrelated loss simulation: </li></ul><ul><ul><li>Number of simulations 50,000 </li></ul></ul><ul><ul><li>Time taken in seconds 2.373606 </li></ul></ul><ul><ul><li>Simulations per second 21065 </li></ul></ul><ul><ul><li>Simulated default rate 311.8472 </li></ul></ul><ul><ul><li>Theoretical default rate 311.9125 </li></ul></ul><ul><li>24 trillion numbers in 6 hours </li></ul>“ A N Other Tier 1” Risk Group
  49. 49. NAG routines for GPUs – 1 <ul><li>Currently available </li></ul><ul><ul><li>Random Number Generator (L’Ecuyer mrg32k3a) </li></ul></ul><ul><ul><ul><li>Uniform distribution </li></ul></ul></ul><ul><ul><ul><li>Normal distribution </li></ul></ul></ul><ul><ul><ul><li>Exponential distribution </li></ul></ul></ul><ul><ul><ul><li>Gamma distribution </li></ul></ul></ul><ul><ul><li>Sobol sequence for Quasi-Monte Carlo (to 19,000 dimensions) </li></ul></ul><ul><ul><li>Brownian Bridge </li></ul></ul>
  50. 50. NAG routines for GPUs – 2 <ul><li>Future plans </li></ul><ul><ul><li>Random Number Generator – Mersenne Twister </li></ul></ul><ul><ul><li>Linear algebra components for PDE option pricing methods </li></ul></ul><ul><ul><li>Time series analysis – wavelets ... </li></ul></ul>
  51. 51. Summary <ul><li>GPUs offer high performance computing for specific massively parallel algorithms such as Monte Carlo simulations </li></ul><ul><li>GPUs are lower cost and require less power than corresponding CPU configurations </li></ul><ul><li>Numerical libraries for GPUs will make these an important computing resource </li></ul><ul><li>Higher order methods for GPUs being considered </li></ul>
  52. 52. Acknowledgments <ul><li>Mike Giles (Mathematical Institute, University of Oxford) – algorithmic input </li></ul><ul><li>Technology Strategy Board through Knowledge Transfer Partnership with Smith Institute </li></ul><ul><li>NVIDIA for technical support and supply of Tesla C1060 and Quadro FX 5800 </li></ul><ul><li>See </li></ul><ul><li> </li></ul><ul><li>Contact: </li></ul><ul><li>[email_address] </li></ul>