Introducing Scalable Quantum Approaches in Language Representation
Upcoming SlideShare
Loading in...5
×
 

Introducing Scalable Quantum Approaches in Language Representation

on

  • 441 views

High-performance computational resources and distributed systems are crucial for the success of real-world language technology applications. ...

High-performance computational resources and distributed systems are crucial for the success of real-world language technology applications.
The novel paradigm of general-purpose computing on graphics processors (GPGPU) offers a feasible and economical alternative: it has already become
a common phenomenon in scientific computation, with many algorithms adapted to the new paradigm. However, applications in language technology do not readily adapt to this approach. Recent advances show the applicability of quantum metaphors in language representation, and many algorithms in quantum mechanics have already been adapted to GPGPU computing. SQUALAR aims to match quantum algorithms with heterogeneous computing to develop new formalisms of information representation for natural language processing in quantum

Statistics

Views

Total Views
441
Views on SlideShare
441
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Introducing Scalable Quantum Approaches in Language Representation Introducing Scalable Quantum Approaches in Language Representation Presentation Transcript

  • Introducing Scalable Quantum Approaches in Language Representation Introducing Scalable Quantum Approaches in Language Representation ´ ´ Peter Wittek Sandor Daranyi Swedish School of Library and Information Science University of Boras˚ 27/06/11
  • Introducing Scalable Quantum Approaches in Language RepresentationOutline 1 SQUALAR 2 Quantum Approaches 3 Scalability 4 A Case Study: BEAGLE 5 Conclusions
  • Introducing Scalable Quantum Approaches in Language Representation SQUALARFundamental Ideas Language Technology Quantum High-performance Theory Computing
  • Introducing Scalable Quantum Approaches in Language Representation SQUALARWhy Do We Need It? Language independence Linguistic processing is typically a multi-level, bottom-up approach The processing starts at word level and then attempts to identify higher level grammatical or semantic structures Scalability Many algorithms do not readily lend themselves to parallelization. Stanford shallow parser processes about eight sentences per second running on a single core.
  • Introducing Scalable Quantum Approaches in Language Representation Quantum ApproachesExamples Focus: vector space based models (semantic spaces). Quantum phenomena in languages may be present at different levels: Sub-word level, terms, combination of concepts, and documents. Encoding term order, operator algebra.
  • Introducing Scalable Quantum Approaches in Language Representation ScalabilityHeterogeneous Computing Focus: distributed computing with graphics hardware. A heterogeneous node: multicore CPUs and GPU accelerator cores. CPU: task parallel GPU: data parallel GPU instances are now available by several cloud providers
  • Introducing Scalable Quantum Approaches in Language Representation ScalabilityThe Case for GPUs Figure: Floating-Point Operations per Second for the CPU and GPU
  • Introducing Scalable Quantum Approaches in Language Representation ScalabilityWhy Is It Hard to Program GPUs? Control ALU ALU ALU ALU Cache DR AM DR AM CP U GPU Streaming hardware Explicit memory management
  • Introducing Scalable Quantum Approaches in Language Representation ScalabilityThe Case for Distributed GPU systems GPU programming enforces minimal communication. This is in sharp contrast with traditional HPC approaches such as MPI. On the other, MapReduce also discourages internode communication. Memory limitations of the GPU beg for distributed systems.
  • Introducing Scalable Quantum Approaches in Language Representation ScalabilityTools That Help GPU programming Low level: OpenCL, CUDA Standard libraries: CuFFT, CuBLAS, CuSparse Experimental frameworks: MapReduce on GPUs (MARS) Distributed computing MPI... MapReduce framework: Hadoop Experimental: Heterogeneous systems: GPMR (distributed MapReduce on GPUs using MPI)
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEBEAGLE Simultaneously encodes word order and context information within semantic vectors. The outer product of two vectors compressed (Holographic Reduced Representation) n Circular convolution: zi = j=0 xj(modn+1) y(i−j)(modn+1) Complexity: O(n2 ), or O(n log n) with Fourier transformation. Two different vectors for each word w: An environmental vector A memory vector
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEComputational steps Random indexing Convolution FFT Pointwise multiplication IFFT
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEExperiment details Corpus Approx. 18,000 documents and 340,000 terms Hardware CPU: Xeon E5620, 2.4GHz GPU: Tesla C2050, 448 Cuda cores, 1.15GHz Software Lucene with StandardAnalyzer SemanticVectors (SVN)
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEResults: Random Indexing 14.5x slow-down Figure: Time taken in ms for random indexing based on a Lucene index
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEResults: Random Indexing I/O-bound Graphics hardware underutilized Is it a lost cause? MARS and GPMR, tfidf indexing already adapted to MapReduce Even though I/O-bound, word counting can be accelerated 11x speed-up on a single GPU
  • Introducing Scalable Quantum Approaches in Language Representation A Case Study: BEAGLEResults: Convolution Number of convolutions: 12,380,531 CPU: 1,281s GPU: 408s Na¨ve implementation without optimization ı Theoretical maximum is around 20x for vectors of 200 dimensions over a single-threaded CPU implemantation 10x over a four-thread CPU implementation
  • Introducing Scalable Quantum Approaches in Language Representation ConclusionsSummary http://www.squalar.org/publications.html Overall acceleration for BEAGLE with a non-optimized implementation: 2.78x. All operations should be performed on the GPU. Distributed systems are a small step forward from single GPU systems. Hoping to attract interest.