This document discusses challenges in system monitoring and data acquisition. It describes the requirements for a modular distributed monitoring system including time synchronization, embedded processing, notifications, and data storage. A typical system uses NI CompactRIO platforms with FPGA and real-time processors to acquire signals, perform processing, and assure reliable operation. Data is stored locally and on network attached storage using various file formats like binary, XML and databases that involve tradeoffs between size, read/write speeds, and organization.

1. Big Data Europe
System Monitoring: Data acquisition challenge
BigDataEurope 2nd SC3 Workshop, Brussels 4/10/164-oct.-16
D. Foussekis, F. Mouzakis (CRES)

2. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.2
General requirements in System monitoring
 Modular distributed system using standard Ethernet network.
 Specs compliant with International Standards.
 Robust and reliable for 24/7 standalone non-stop operation.
 Scalable and reconfigurable.
 Time Synchronization across all modules (GPS or Master module).
 Embedded processing capabilities.
 Send notifications and alarms.
 Data storage
www.big-data-europe.eu

3. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.3
Typical DAQ system core component (1/2)
 NI-CompactRIO platform http://www.ni.com/compactrio/
 Designed for harsh environments
-20º to -55ºC temperature, 5grms vibration, 30g shock
 Low power requirements
~10W at 9-30V for battery powered standalone operation
 Data acquisition based on FPGA hardware offering fast I/O response times
and increased reliability
www.big-data-europe.eu
Source: NI.com

4. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.4
Typical DAQ system core component (2/2)
 Various modules for Analogue I/O and Digital I/O
High precision (24-bit delta-sigma) A/D converters (s.g. modules)
16-bit 100kHz Simultaneous Sampling AI modules.
 XML - Configuration file
 Raw (high speed) data sent over the Ethernet and stored on a NAS.
 All data packets are time-stamped with common GPS or local time.
 Backup storage on each module USB port, in case of network loss.
www.big-data-europe.eu
Source: NI.com

5. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.5
Typical CMS configuration
www.big-data-europe.eu
. . .
Gigabit Local Area Network (LAN)
Network Attached Storage (NAS)System Monitoring PC

6. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.6
Distributed system
www.big-data-europe.eu
Photo: Courtesy of Vestas Wind Systems A/S
Tower (bending)Tower (torsion)
Electrical generator Gearbox
Blades

7. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.7
Architecture
www.big-data-europe.eu
Real-Time microprocessor:
• Runs a real-time OS (NI Linux Real-Time)
• Communicates and uploads data
• Performs statistical analysis
• Stores and compresses raw data
FPGA microprocessor:
• Acquires signals (analogue & digital I/O)
• Runs without OS at high clock rates and no
jitter
• Executes different tasks with true parallelism
• Performs low-level signal processing
(i.e.: filtering, control)
• Assures non-stop operation
Source: NI.com

8. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.8
Field-Programmable Gate Array (FPGA)
www.big-data-europe.eu
- Reconfigurable Hardware chip.
- Analogous to a printed circuit board with unconnected components on it.
- Connections in an FPGA circuit are dynamically defined in software.

9. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.9
Example of FPGA implementation
www.big-data-europe.eu
Boolean operations (green lines) based on
digital I/O (A,B,C,D,F) execute in parallel with
math operations (orange lines) based on
analogue I/O (W, X, Y, Z).
The compiler produces an optimized bitstream
file, with all the routing paths, between millions
of logic gates, memory blocks and I/O channels.

10. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.10
Raw Data storage – Existing File Formats
www.big-data-europe.eu
ASCII files
Characteristics
• Human-readable
• Easily portable to other applications such as
Microsoft Excel
Considerations
• Significantly larger disk footprint
• Slow read and write

11. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.11
Raw Data storage – Existing File Formats
www.big-data-europe.eu
Binary files
Characteristics
• Compact file size
• Fast streaming (read & write)
Considerations
• Not human-readable
• Not easily exchangeable

12. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.12
Raw Data storage – Existing File Formats
www.big-data-europe.eu
XML files
Characteristics
• Stores complex data structures
• Shows display in a Web browser or in a text
editor
Considerations
• Even larger disk footprint
• Front-end schema design
• Does not stream

13. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.13
Raw Data storage – Existing File Formats
www.big-data-europe.eu
Database files
Characteristics
• Store data centrally
• Organize and query test results with MySQL
or Access
Considerations
• Time intensive startup effort
• Requires maintenance
• Potentially high cost

14. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.14
Raw Data storage – Existing File Formats
www.big-data-europe.eu
Binary Structured (TDMS – NI Technical Data Management Technology)
Characteristics
• Single streaming binary file
• Three levels of hierarchy for better organization
File, groups, and channels
• Customizable, descriptive properties at each level
• User-defined Meta data for campaign properties
Considerations
• Third party format

15. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.15
File Format Comparison
www.big-data-europe.eu

16. BigDataEurope 2nd SC3 Workshop, Brussels 4/10/16
System monitoring p.16
(Instead of) Conclusion
www.big-data-europe.eu
Ron Musick
Lead Researcher at Lawrence Livermore
National Laboratory