The document discusses the SERENA data processing pipelines. It describes the raw data formats for the SERENA instruments STROFIO, MIPA, and PICAM. It also outlines the processing steps to convert telemetry data to calibrated science data and generate quick-look products like histograms, images, and time series from the raw SERENA data. This includes software to convert telemetry to raw data, raw to calibrated data, and raw to partially processed data which can then be converted to calibrated data.

1. SERENA Data ProcessingSERENA Data Processing
1
SERENA BC SGS meeting 21-22/05/2015
ESAC, Madrid
Francesco Lazzarotto INAF IAPS Rome 21/05/15
francesco.lazzarotto@iaps.inaf.it

2. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 2
SERENA Processing overviewSERENA Processing overview
The proposal of the SERENA PI is to build a system
where the SERENA SGS applications run both in
the ESAC based system and in a system based at
IAPS.
The aim of this concerns:
reliability, allowing hot redundancy for the
SERENA data processing services
supervision of the whole data production process by
the SERENA PI
establishing a common I/F with ESA.

3. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 3
SERENA processing servicesSERENA processing services
The SERENA experiment processing services have
the aim of automatically process the raw data
packets stream incoming from the BC spacecraft
into human readable data files.
It shall be possible run all applications in batch
mode operated by a scheduler such as cron
Single tasks may then also be wrapped in GUI based
applications

4. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 4
SERENA pipelinesSERENA pipelines
Science Data processing software
[TMdata]-->TM2Raw-->[PDS4RawData]
[PDS4RawData]-->Raw2Cal-->[PDS4CalData]
[PDS4RawData]-->Raw2PP-->[PDS4PPData]-->PP2Cal-->[PDS4CalData]
Science Quick Look
[any]-->QLA-->[StatsData] and/or [PlotData]
Legenda
PP = Partially Processed
Stats = data containing statistics about the analysed data
Plot = images or other formats able to be plotted by standard software
TM2Raw = telemetry data to Raw data software
Raw2Cal = PDS4 Raw data to Calibrated data software
Raw2PP = PDS4 Raw data to Partially Processed data software
PP2Cal = Partially Processed data to Calibrated data software
QLA = Quick Look Analysis
[xx] = dataset of type xx

5. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 5
SERENA (PDS4) Raw DataSERENA (PDS4) Raw Data
from https://pds.nasa.gov/policy/PolicyOnProcessingLevels03112013.pdf
Original data from an instrument. If compression,
reformatting, packetization, or other translation has
been applied to facilitate data transmission or
storage, those processes will be reversed so that the
archived data are in a PDS4 approved archive
format.
we'll refer to this data format as PDS4 raw, to disambiguate from the
instrument native data format, as a matter of fact SERENA instruments can
produce on-board accumulated data.

6. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 7
Strofio PDS4 Raw DataStrofio PDS4 Raw Data
Uncompressed, unformatted, un-packed raw data
The STROFIO raw data products include
1) Basic rates: Event counters per second, accumulated for a period of time (counters
include start/stop pulses, position and time measurements and coincident
position/time measurements; for the left and right anodes),
2) High-quality ToF spectra (rebinned or integrated into low-time resolution spectra),
3) Low-quality ToF spectra (rebinned or integrated into low-time resolution spectra),
and
4) Raw events: Raw events saved in the order in which they were received and filtered
depending on the selected option (no filtering, high and/or quality).

7. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 8
MIPA PDS4 Raw DataMIPA PDS4 Raw Data
The MIPA raw data products consist of a matrix of
accumulated counts as a function of position, energy
and mass, with up to 24 pixels, 96 energy levels and
N mass bins. Number of mass bins TBC.

8. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 9
PICAM PDS4 Raw DataPICAM PDS4 Raw Data
The PICAM raw data products consist of a matrix of
accumulated counts from all detector pixels (1-60
pixels) for up to 32 energy levels and N mass bins.
Number of mass bins TBC.

9. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 10
SERENA pipelinesSERENA pipelines
Data processing software
[TMdata]-->TM2Raw-->[PDS4RawData]
[PDS4RawData]-->Raw2Cal-->[PDS4CalData]
[PDS4RawData]-->Raw2PP-->[PDS4PPData]-->PP2Cal-->[PDS4CalData]
Science Quick Look
[any]-->QLA-->[StatsData] and/or [PlotData]
Legenda
PP=Partially Processed
Stats=data containing statistics about the analysed data
Plot=images or other formats able to be plotted by standard software
TM2Raw=telemetry data to Raw data software
Raw2Cal=PDS4 Raw data to Calibrated data software
Raw2PP=PDS4 Raw data to Partially Processed data software
PP2Cal=Partially Processed data to Calibrated data software
QLA=Quick Look Analysis
[xx]=dataset of type xx

10. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 11
Event ListsEvent Lists
Time Tag <Position> Energy <attrn> ... ... <attrm>
... ... ... ... ...
Basic representation of measurement data
outcaming from a particle detector.
Each interaction event between the particle and the detector is
saved with the event describing the attributes. Attributes may
also be structured.
Histogramming on time attribute we accumulate time series, on
position we accumulate images and on energy we accumulate
spectra.
Conditions on the event attributes can help to discriminate
events.

11. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 12
Example: ELENAExample: ELENA
ELENA instrument may produce the following data
products:
1)Event Lists (the most important raw format from
which all the other data can be reproduced)
2)Monodimensional images and/or 1D histograms
3)2D images (pixel on y and time or space on x)
4)Time Series (time vs counts/flux)

12. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 13
ELENA Event ListELENA Event List

13. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 14
ELENA Sectors histogramELENA Sectors histogram
Sectors [1-32] (Pixel)
c
o
u
n
t
s

14. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 15
ELENA Histogram (2)ELENA Histogram (2)

15. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 16
ELENA Time SeriesELENA Time Series

16. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 17
ELENA 2DImagesELENA 2DImages

17. 21/05/15 Francesco Lazzarotto SERENA SGS meeting 18
ELENA pipeline logfilesELENA pipeline logfiles
2015-05-07T11:56:47.497237660|nibbio|gesh|INFO|1 param: ok
2015-05-07T11:56:47.530848635|nibbio|gesh|INFO|creating gnuplot temp script in
2015-05-07T11:56:47.536302681|nibbio|gesh|INFO|n. of pars = 1
2015-05-07T11:56:47.541704765|nibbio|gesh|INFO|genehist: creating 4 histograms file(s)
2015-05-07T11:56:47.558989083|nibbio|gesh|INFO|elena histograms generated (histos on rows) on file /tmp/tmp.EouvruJ7J8_on_rows.dat
2015-05-07T11:56:47.570270686|nibbio|gesh|INFO|creating reversed histo data file (histogram on columns) on file /tmp/tmp.ezfzsm94go_on_cols.dat
2015-05-07T11:56:47.581347232|nibbio|gesh|INFO|created reversed histo data file (histogram on columns) in /tmp/tmp.ezfzsm94go_on_cols.dat
2015-05-07T11:56:47.587414204|nibbio|gesh|INFO|removing histogram on rows data file in /tmp/tmp.EouvruJ7J8_on_rows.dat
2015-05-07T11:56:47.605369559|nibbio|gesh|INFO|./gehs.sh: creating temp gnuplot script on file /tmp/tmp.UAGXN3MYWI.gp
2015-05-07T11:56:47.614962678|nibbio|gesh|INFO|awk output is 4
2015-05-07T11:56:47.622007987|nibbio|gesh|INFO|gnuplot script generated
2015-05-07T11:56:47.629641639|nibbio|gesh|INFO|created gnuplot temp script in /tmp/tmp.UAGXN3MYWI.gp
2015-05-07T11:56:47.635893271|nibbio|gesh|INFO|running gnuplot
2015-05-07T11:56:53.961194671|nibbio|gesh|INFO|removing gnuplot temp script in /tmp/tmp.UAGXN3MYWI.gp