TRB: A platform for TDC and digital
readout

Outline

   •   Motivation for TRB
   •   Implementation of the idea
   •   Measurements and results
   •   New developments
   •   Local computing capability: my point of view
   •   Summary




 2010-02-05                 Michael Traxler, GSI      1
Motivation for TRB

Given Task: TDCs + fast DAQ for 2244 channels tRPC

   • Big project, many people involved => huge effort
   • TDCs are used everywhere in nuclear physics
     experiments
   • A general solution is wanted, which can be reused
     for other detectors and different experiments
      – Not only for TDCs, but also for other tasks
   • HADES: around 80k channels, 100kHz event rate,
     250MBytes/s sustained data rate for Au+Au
   • Other issue: price and development time

 2010-02-05                  Michael Traxler, GSI        2
Concept for HADES DAQ: TRB

 • One platform for all tasks
 • Directly mounted on the detector
   – no long cables
 • Integrated DAQ
 • Includes local power-supplies          Publication:
 • Modular design                         „A General Purpose Trigger and
                                          Readout Board for HADES and
                                          FAIR-Experiments“
   – Pluggable AddOns to TRB              I. Fröhlich et al., Nuclear Science,
                                          IEEE Transactions on,
 • High granularity (~70 TRBs)            Volume 55, Issue 1, Feb. 2008
                                          Page(s):59 - 66
 • Dedicated network protocol (TRBnet)
 • Reduces development effort/time and simplifies the
   debugging process

2010-02-05                      Michael Traxler, GSI                             3
The TRBv2

                                                                
                                                                 4 TDCs – 128
                                    TDC 2, 3                      channels
                                                                
                                                                 FPGA – Virtex4LX40
                                                 Optical link
                                                                
                                                                 4x512Mb SDRAM
                                   SDRAM
      TDC                                                       
                                                                 ETRAX FS – 4
      0, 1
                                                                  processors, Linux
                         FPGA
                         Virtex4
                                                                
                                                                 100Mb/s,TCP/IP
                                         DSP                    
                                                                 2,5 Gb/s optical link
                                                                
                                                                 DSP TigerSharc
                                                                
                                                                 AddOn connector
                ETRAX                        DC/DC
                                                                
                                                                 48V isolated DC/DC
                        SDRAM
                                                                  converters
Ethernet



   2010-02-05                           Michael Traxler, GSI                         4
Results of TRB development
Time resolution:
   • 128 channels: ~40ps RMS
   • 32 channels: ~16ps RMS
Field of Usage:
  • Successfully used in many production beam times in HADES
  • Platform for: TDC (RPC + discriminator and charge measurement for
    PMTs), ADC and pure digital readout (everything)
  • Used not only by HADES:
    – PANDA - DIRC detector (in beam in 2009), PANDA - MDC readout
    – CBM – MAPS detector development
    – PET- scanner prototype in Coimbra
    – KVI - development of FPGA algorithms
    – HPLUS - in China, Lanzhou Institute
    – And many more planned applications

   2010-02-05                  Michael Traxler, GSI                     5
New developments / future plans

  • Better time resolution
     – Replace HPTDCs
  • Reduce costs
     – More cost sensitive FPGA
     – Remove DSP
  • Needed tasks:
     – Replace the obsolete components
     – 1 GbE Ethernet with Linux-CPU




2010-02-05                Michael Traxler, GSI   6
TDC implemented in FPGA

  • A Tapped Delay Line (carry chain) TDC has been
    implemented in a FPGA (Virtex 4) (asynchronous
    design)
     – Time resolution: <10ps
     – 32 channels in one FPGA
     – Very promising results!
  • To Do:
     – Implement all features of HPTDC in the FPGA (e.g.
       window matching)
     – Implementation of design in cost sensitive FPGAs
       (Lattice ECP2M, Altera Arria GX, etc.) and evaluate
       performance

2010-02-05                 Michael Traxler, GSI              7
TDC in FPGA: results




2010-02-05       Michael Traxler, GSI   8
Local Computing
 • Potential is very high
   – FPGA + DSP
 • Realization is really hard
   – DSP has been abandoned (no manpower)
   – FPGA does data transfer/sorting/zero
     suppression/networking/switching (RTL)
   – KVI: Peak detection with baseline restoration (RTL)
   – All: Several man year projects
 • Going beyond the mentioned is very ambitious
   – Runge-Kutta for tracking + other complex algorithm
   – Special hardware algorithms double the work; should be
     very similar for off- and online analysis
 • Parallel calculation on GPUs seems to me the way to go
   – Very promising results for Runge-Kutta
2010-02-05                 Michael Traxler, GSI               9
Local Computing II

  • Concept in many experiments
     –   Digitize at the detector, the closer the better
     –   Apply simple algorithms to reduce data amount
     –   Transport the data (data transport is relatively cheap)
     –   Local computing is expensive and is producing heat!
     –   Commercial general purpose computing (e.g. GPUs) is
         not beatable, except for special applications




2010-02-05                   Michael Traxler, GSI                  10
Summary

  • A very successful platform for many channels TDC +
    DAQ has been built, useful for many applications
  • In the future we can adapt much better to the users
    need by using FPGAs as TDCs: the compromise out
    of channels (price) and time resolution can be
    changed by programming
  • Local computing resources (FPGA) are available
  • Costs will be reduced




2010-02-05              Michael Traxler, GSI              11
Involved People in TRB design

        E. Bayer1, M. Böhmer5, I. Fröhlich4, J. Michel4,
     M. Kajetanowicz3, K. Korcyl2, G. Korcyl2, M. Palka1,2 ,
P. Salabura2, P. Skott1, M. Traxler1, R. Trebacz2, S. Yurevich1


                          1
                              GSI, Darmstadt, Germany,
                 2
                      Jagiellonian University, Krakow, Poland,
                   3
                     Nowoczesna Elektronika, Krakow, Poland,
              4
                J.-W. Goethe-Universitaet, Frankfurt, Germany,
                 5
                    Technische Universität, München, Germany




 2010-02-05                        Michael Traxler, GSI           12
TRB




             Thank you for your attention!




2010-02-05             Michael Traxler, GSI   13
System Overview
                                      RPC                                             VME CPU
                                                                                        MU
To the Front End Electronics




                                      MDC
                                                                                        CTS


                                      TOF

                                                                                                  VULOM3

                                    Shower




                                     RICH             ...



                                     F. Wall
                                                                      Ethernet      Parallel
                                                                                 Event Building
                                                                                  (computers)
                                   Start, Veto

                               2010-02-05               Michael Traxler, GSI                               14

M Traxler TRB and Trasgo

  • 1.
    TRB: A platformfor TDC and digital readout Outline • Motivation for TRB • Implementation of the idea • Measurements and results • New developments • Local computing capability: my point of view • Summary 2010-02-05 Michael Traxler, GSI 1
  • 2.
    Motivation for TRB GivenTask: TDCs + fast DAQ for 2244 channels tRPC • Big project, many people involved => huge effort • TDCs are used everywhere in nuclear physics experiments • A general solution is wanted, which can be reused for other detectors and different experiments – Not only for TDCs, but also for other tasks • HADES: around 80k channels, 100kHz event rate, 250MBytes/s sustained data rate for Au+Au • Other issue: price and development time 2010-02-05 Michael Traxler, GSI 2
  • 3.
    Concept for HADESDAQ: TRB • One platform for all tasks • Directly mounted on the detector – no long cables • Integrated DAQ • Includes local power-supplies Publication: • Modular design „A General Purpose Trigger and Readout Board for HADES and FAIR-Experiments“ – Pluggable AddOns to TRB I. Fröhlich et al., Nuclear Science, IEEE Transactions on, • High granularity (~70 TRBs) Volume 55, Issue 1, Feb. 2008 Page(s):59 - 66 • Dedicated network protocol (TRBnet) • Reduces development effort/time and simplifies the debugging process 2010-02-05 Michael Traxler, GSI 3
  • 4.
    The TRBv2  4 TDCs – 128 TDC 2, 3 channels  FPGA – Virtex4LX40 Optical link  4x512Mb SDRAM SDRAM TDC  ETRAX FS – 4 0, 1 processors, Linux FPGA Virtex4  100Mb/s,TCP/IP DSP  2,5 Gb/s optical link  DSP TigerSharc  AddOn connector ETRAX DC/DC  48V isolated DC/DC SDRAM converters Ethernet 2010-02-05 Michael Traxler, GSI 4
  • 5.
    Results of TRBdevelopment Time resolution: • 128 channels: ~40ps RMS • 32 channels: ~16ps RMS Field of Usage: • Successfully used in many production beam times in HADES • Platform for: TDC (RPC + discriminator and charge measurement for PMTs), ADC and pure digital readout (everything) • Used not only by HADES: – PANDA - DIRC detector (in beam in 2009), PANDA - MDC readout – CBM – MAPS detector development – PET- scanner prototype in Coimbra – KVI - development of FPGA algorithms – HPLUS - in China, Lanzhou Institute – And many more planned applications 2010-02-05 Michael Traxler, GSI 5
  • 6.
    New developments /future plans • Better time resolution – Replace HPTDCs • Reduce costs – More cost sensitive FPGA – Remove DSP • Needed tasks: – Replace the obsolete components – 1 GbE Ethernet with Linux-CPU 2010-02-05 Michael Traxler, GSI 6
  • 7.
    TDC implemented inFPGA • A Tapped Delay Line (carry chain) TDC has been implemented in a FPGA (Virtex 4) (asynchronous design) – Time resolution: <10ps – 32 channels in one FPGA – Very promising results! • To Do: – Implement all features of HPTDC in the FPGA (e.g. window matching) – Implementation of design in cost sensitive FPGAs (Lattice ECP2M, Altera Arria GX, etc.) and evaluate performance 2010-02-05 Michael Traxler, GSI 7
  • 8.
    TDC in FPGA:results 2010-02-05 Michael Traxler, GSI 8
  • 9.
    Local Computing •Potential is very high – FPGA + DSP • Realization is really hard – DSP has been abandoned (no manpower) – FPGA does data transfer/sorting/zero suppression/networking/switching (RTL) – KVI: Peak detection with baseline restoration (RTL) – All: Several man year projects • Going beyond the mentioned is very ambitious – Runge-Kutta for tracking + other complex algorithm – Special hardware algorithms double the work; should be very similar for off- and online analysis • Parallel calculation on GPUs seems to me the way to go – Very promising results for Runge-Kutta 2010-02-05 Michael Traxler, GSI 9
  • 10.
    Local Computing II • Concept in many experiments – Digitize at the detector, the closer the better – Apply simple algorithms to reduce data amount – Transport the data (data transport is relatively cheap) – Local computing is expensive and is producing heat! – Commercial general purpose computing (e.g. GPUs) is not beatable, except for special applications 2010-02-05 Michael Traxler, GSI 10
  • 11.
    Summary •A very successful platform for many channels TDC + DAQ has been built, useful for many applications • In the future we can adapt much better to the users need by using FPGAs as TDCs: the compromise out of channels (price) and time resolution can be changed by programming • Local computing resources (FPGA) are available • Costs will be reduced 2010-02-05 Michael Traxler, GSI 11
  • 12.
    Involved People inTRB design E. Bayer1, M. Böhmer5, I. Fröhlich4, J. Michel4, M. Kajetanowicz3, K. Korcyl2, G. Korcyl2, M. Palka1,2 , P. Salabura2, P. Skott1, M. Traxler1, R. Trebacz2, S. Yurevich1 1 GSI, Darmstadt, Germany, 2 Jagiellonian University, Krakow, Poland, 3 Nowoczesna Elektronika, Krakow, Poland, 4 J.-W. Goethe-Universitaet, Frankfurt, Germany, 5 Technische Universität, München, Germany 2010-02-05 Michael Traxler, GSI 12
  • 13.
    TRB Thank you for your attention! 2010-02-05 Michael Traxler, GSI 13
  • 14.
    System Overview RPC VME CPU MU To the Front End Electronics MDC CTS TOF VULOM3 Shower RICH ... F. Wall Ethernet Parallel Event Building (computers) Start, Veto 2010-02-05 Michael Traxler, GSI 14