Talk by Eduard Güldner

1. Train IT Platform Monitoring
Icinga2 @ Siemens Mobility Customer Services
siemens.com/mobility-servicesUnrestricted © Siemens AG 2017

2. Unrestricted © Siemens AG 2017
March 2017Page 2 Siemens Mobility Services
About me
Eduard Güldner
• @Home in Nuremberg
• Bachelor Thesis about using Icinga2 to monitor cars
• Support Engineer (Siemens MO CS STC SC-SO)

3. Unrestricted © Siemens AG 2017
March 2017Page 3 Siemens Mobility Services
Siemens AG
Mobility is one of 9 Divisions
Siemens AG Managing Board
Lisa
Davis
Roland
Busch
Siegfried
Russwurm
Klaus
Helmrich
Janina
Kugel
Ralf P.
Thomas
Power and Gas
Mobility
Willibald Meixner (CEO)
Business Units:
• Large Gas Turbines, Generators
• Distributed Generation
• Steam Turbines
• Compressors
• Energy Solutions
• Instrumentation and Electrical
Jochen Eickholt (CEO)
Business Units:
• Mobility Management
• Turnkey Projects & Electrification
• Mainline Transport
• Urban Transport
• Customer Services
Wind Power and Renewables
Digital Factory
Anton Sebastian Huber (CEO)
Business Units:
• Factory Automation
• Motion Control
• Control Products
• Product Lifecycle Management
• eCar Powertrain Systems
• Customer Services DF&PD
Power Generation Services
Process Industries and Drives
Randy Zwirn (CEO)
Business Units:
• Power and Gas
• Distributed Generation and Compressors
• Wind Power and Renewables
Jürgen Brandes (CEO)
Business Units:
• Large Drives
• Process Automation
• Oil & Gas and Marine
• Mechanical Drives
Energy Management
Healthcare
Ralf Christian (CEO), Jan Mrosik (CEO)
Business Units:
• Medium Voltage & Systems
• Low Voltage & Products
• Transformers
• High Voltage Products
• Transmission Solutions
• Energy Automation
• Smart Grid Solutions & Services
Bernd Montag (CEO)
Business Units:
• Imaging & Therapy Systems
• Clinical Products
• Diagnostics
• Customer Solutions
• Audiology
Building Technologies
Financial Services
Matthias Rebellius (CEO)
Business Units:
• Control Products & Systems
Roland Chalons-Browne (CEO)
Business Units:
• Commercial Finance
• Project & Structured Finance Power, Oil & Gas
• Project & Structured Finance Energy
Management, Mobility and Industries
• Project & Structured Finance Healthcare and
Leveraged Finance
• Insurance
• Treasury
• Financing & Investment Mgmt.
Markus Tacke (CEO)
• Onshore
• Offshore
Joe Kaeser
President and Chief Executive Officer
Divisions

4. Unrestricted © Siemens AG 2017
March 2017Page 4 Siemens Mobility Services
Urban Transport Turnkey Projects &
Electrification
Mobility Division/MO CC
Mobility Division
Customer Services as a service provider for all mobility solutions
Michael Peter (CEO)
Pierre Bauer (CFO)
Sabrina Soussan (CEO)
Marko Feulner (CFO)
Johannes Emmelheinz (CEO)
Klaus Rathke (CFO)
Sandra Gott-Karlbauer (CEO)
Franz Gredler (CFO)
Products, solutions and turnkey
systems for rail
and road automation and
optimization
Short-distance, regional and
long-distance rolling stock, and
product and system solutions for
passenger and freight transport
Railbound urban public
transport vehicles, eBuses and
passenger coaches
Complete rail and road solutions
and rail electrification solutions
Services and tools for road and
rail infrastructure and rolling
stock
Jochen Eickholt
(CEO)
Karl Blaim
(CFO)
Business Units
Mainline Transport Customer ServicesMobility Management
Ralph Hasselbacher (CEO)
Carsten Schladitz (CFO)

5. Unrestricted © Siemens AG 2017
March 2017Page 5 Siemens Mobility Services
Mobility Division
A broad portfolio including services for all mobility systems
Rail automation for passenger and
freight services
Automated payment systems
Mass transit and mainline
rail vehicles
Regional and high speed trains
Services for rail and road transport
Power supply solutions for rail and
road vehicles
Turnkey rail systems
Road traffic management
Integrated, intermodal
mobility solutions
Locomotives

6. Unrestricted © Siemens AG 2017
March 2017Page 6 Siemens Mobility Services
Siemens Mobility Services
A long tradition for a successful future
BANGKOK
1999
Maintenance for
Full Turnkey Metro
system
BERLIN
1881
Operation and
Maintenance of
first electric tram-
system
LONDON
1997
From now on each
delivered train from
Siemens to the UK
will be serviced by
us
BERLIN
1924
Spare parts for first
automatic traffic
signal system
WORLD
Today
75 major projects
around the globe

7. Unrestricted © Siemens AG 2017
March 2017Page 7 Siemens Mobility Services
Germany
TransRegio – Desiro ML (2008-2023)
Rhein-Ruhr-Express –
Desiro HC (2018-2050)
boxXpress – Vectron (2013-2026)
Potsdam – Combino (1999-2019)
Vattenfall Europe Mining (1996-2024)
Duesseldorf – Skytrain (2002-2019)
France
RATP-OVS (Orly) – VAL (2001-2018)
Aérosat (Roissy) – VAL (2012-2017)
Keolis Lille – VAL (2011-2018)
Tisséo (Toulouse) – VAL (2011-2020)
Keolis Rennes – VAL (2006-2018)
Siemens Mobility Services
Excerpt of our maintenance references around the globe
China
HXD1 Loco MS (2013-2018)
HXD1B Loco MS (2013-2016)
Hong Kong – West Rail Line (2008-2020)
Beijing – Jing Jin DPL Line (2011-2016)
Shenzhen – Line 4 (2013-2016)
Australia
Melbourne – Metro (2000-2018)
Chile
Santiago – Metro Communication
System (2008-2017)
Santiago – Metro (2005-2017)
Valparaiso – Metro (2014-2019)
Dominican Republic
OPRET (Santo Domingo) – Metro
(2009-2017)
India
Central Railway Mumbai –
EMU (2015-2017)
Indian Railway – WDx4 (2014-2017)
Delhi Rapid Metro Gurgaon (2013-2023)
Qatar
Education City LRT O&M
(2016-2019)
Lithuania
Freight Locos (2013-2017)
Turkey
TCDD – Velaro (2015-2024)
Netherlands
ProRail BV - Simis W/C (2014-2024)
HSL Zuid (2006-2031)
Czech Republic
Prag – Metro (2005-2019)
Poland
PKP IC – U4-D (2010-2024)
PKP Cargo – Vectron (2016-2024)
MRCE – Vectron (2014-2020)
Poznan – Combino (2007-2019)
Thailand
MRTA Bangkok – Metro (2004-2024)
BTS Bangkok – Skytrain (2000-2029)
Denmark
Freight Locos (2009-2020)
Banedanmark (2007-2041)
Asia, AustraliaEuropeAmerica
Spain
Nertus2) – Velaro E (2007-2023)
Adif Madrid-Sevilla-Málaga (1992-2025)
Adif Barcelona-Figueras (2012-2017)
Metro de Barcelona (TMB) (2009-2018)
Metro Madrid on board units
(2011-2023)
Adif – Spain high speed network
(2005-2025)
Argentina
Metrovias (Buenos Aires) – Metro
(1997-2016)
Austria
Wiener Lokalbahnen Cargo –
U4 (2011-2039)
STB – U4 (2015-2020)
Italy
FUC – U4 (2012-2017)
InRail – U4 (2012-2017)
CFI – Vectron (2015-2017)
GTT – Turin VAL (2008-2017)
Russia
RZD – Desiro RUS (2011-2063)
RZD – Velaro RUS (2007-2045)
RZD – Luzhskaya Cargo (2014-2026)
Saudi Arabia
Mekka-Medina – High speed
(2017-2029)
USA
Amtrak – ACS-64 (2014-2029)
All Aboard Florida (2014-2030)
Chicago Department of Aviration – VAL
(1994-2017)
Portugal
SIMEF1) – Loco MS (2009-2022)
Refer – Signaling (2014-2018)
Refer – Substations (2011-2016)
United Kingdom
South West Trains –
Desiro (2002-2025)
Heathrow Express – Desiro (1997-2023)
TransPennine Express –
Desiro (2005-2016)
London Midland – Desiro (2005-2025)
ScotRail – Desiro (2008-2020)
Northern Rail – Class 333 (1998-2016)
GAF – Desiro Class 360 (2004-2016)
Thameslink – Desiro City (2015-2045)
EIL – Velaro Eurostar (2015-2030)
London Crossrail (2017-2032)
Edinburgh Tram Network (2014-2024)
LocomotivesHigh speed and Intercity trains Turnkey solutionsUrban transport Rail electrificationRail automationCommuter and regional trains
1) Joint venture of Siemens and EMEF 2) Joint venture of Siemens and Renfe
Bulgaria
BDZ – Desiro (2005-2020)
Algeria
Algier – Metro (2009-2020)
Africa
Switzerland
Bernmobil – Combino (2016-2029)

8. Unrestricted © Siemens AG 2017
March 2017Page 8 Siemens Mobility Services
Siemens Mobility Services for Velaro E
1.1 million kilometers without maintenance-related delays
Fleet: Velaro E
• 26 trains in daily service:
Madrid–Barcelona and Madrid–Malaga
• Complete maintenance for 26 eight-car trains by
Siemens Mobility Services for a period of 14 years
• 24-hour service by 160 Siemens employees to ensure
the availability and reliability of the trains every day
• Flexible maintenance intervals according to the
"performance-based maintenance" concept
Full reimbursement of the ticket price to passengers in
case of delays >15 minutes
Only one out of 2,300 trips is noticeably delayed

9. Unrestricted © Siemens AG 2017
March 2017Page 9 Siemens Mobility Services
Monitoring Train IT

10. Unrestricted © Siemens AG 2017
March 2017Page 10 Siemens Mobility Services
Monitored Software

11. Unrestricted © Siemens AG 2017
March 2017Page 11 Siemens Mobility Services
Numbers
~1000
Services
50 Hosts
Checked in
4.6 sec
15 Mio. Log
Events / 24h

12. Unrestricted © Siemens AG 2017
March 2017Page 12 Siemens Mobility Services
(Very) High Level View of Infrastructure

13. Unrestricted © Siemens AG 2017
March 2017Page 13 Siemens Mobility Services
(Very) High Level View of Infrastructure

14. Unrestricted © Siemens AG 2017
March 2017Page 14 Siemens Mobility Services
Example for Train Data
• ÖBB (Austria):
• Transmission, axle bearing and motor temperatures, door
power and air conditioning system data are transmitted
continuously

15. Unrestricted © Siemens AG 2017
March 2017Page 15 Siemens Mobility Services
Some facts about SMDA (Smart Monitoring and Data Analysis)
• Most components are written in Java
• Hosted on-premise
• aka “almost legacy“
• aka “one of the reasons I‘m allowed to talk about this“
• don‘t worry, cloud based solution is already in the pipeline J
• Many dependencies (you are not alone in such a big company)

16. Unrestricted © Siemens AG 2017
March 2017Page 16 Siemens Mobility Services
Approach / Breakdown
• Question: How to monitor a big heterogeneous platform?
or: “How to monitor, the monitoring system?” J
• Two Monitoring Dimensions / Factors to consider:
• Data Flow
• Data Hierarchy

17. Unrestricted © Siemens AG 2017
March 2017Page 17 Siemens Mobility Services
Data Hierarchy
Application
Application
Container
OS

18. Unrestricted © Siemens AG 2017
March 2017Page 18 Siemens Mobility Services
Data Flow
Transmission Processing Visualization

19. Unrestricted © Siemens AG 2017
March 2017Page 19 Siemens Mobility Services
Data Flow
Transmission Processing Visualization

20. Unrestricted © Siemens AG 2017
March 2017Page 20 Siemens Mobility Services
Transmission
• Use Icinga to query ELK

21. Unrestricted © Siemens AG 2017
March 2017Page 21 Siemens Mobility Services
Transmission
• Use custom Icinga Plugins to monitor things like:
• File count
• File Age
• Folder Timestamp

22. Unrestricted © Siemens AG 2017
March 2017Page 22 Siemens Mobility Services
Transmission – Example: File count – Service Definiton
Top Tipp:
Use the mighty Icinga
Language J

23. Unrestricted © Siemens AG 2017
March 2017Page 23 Siemens Mobility Services
Transmission – Example: File count – Service in Host
Top Tipp:
“Balance” your
service configuration
between Host &
Service Definitions
Notifications for this folder are automatically
sent to Group „XYZ“, remember? J

24. Unrestricted © Siemens AG 2017
March 2017Page 24 Siemens Mobility Services
Transmission – Example: File count
Top Tipp:
Always try to output
performance data in
your custom plugins.
Your colleagues will
thank you J
• Very useful to see trends

25. Unrestricted © Siemens AG 2017
March 2017Page 25 Siemens Mobility Services
Data Flow
Transmission Processing Visualization

26. Unrestricted © Siemens AG 2017
March 2017Page 26 Siemens Mobility Services
Processing & Visualization
• Use check_mssql_health or check_oradb_health to monitor data
processing & display
• Many thanks to Gerhard Lausser (GitHub @lausser) for these
wonderful plugins!

27. Unrestricted © Siemens AG 2017
March 2017Page 27 Siemens Mobility Services
Processing & Visualization
• Do SQL Queries to check file transfer status. Service definition:
Top Tipp:
You can even do
“housekeeping
queries” in Icinga2 J

28. Unrestricted © Siemens AG 2017
March 2017Page 28 Siemens Mobility Services
Processing & Visualization
• Host definiton

29. Unrestricted © Siemens AG 2017
March 2017Page 29 Siemens Mobility Services
Processing & Visualization

30. Unrestricted © Siemens AG 2017
March 2017Page 30 Siemens Mobility Services
Data Hierarchy
Application
Application
Container
OS

31. Unrestricted © Siemens AG 2017
March 2017Page 31 Siemens Mobility Services
Data Hierarchy
Application
Application
Container
OS

32. Unrestricted © Siemens AG 2017
March 2017Page 32 Siemens Mobility Services
OS
• mostly covered by icinga2 standard plugins:
• Linux: disk, load, swap, …
• Windows: disk-windows, load-windows, …
• or by monitoring plugins project (https://www.monitoring-plugins.org/)

33. Unrestricted © Siemens AG 2017
March 2017Page 33 Siemens Mobility Services
Data Hierarchy
Application
Application
Container
OS

34. Unrestricted © Siemens AG 2017
March 2017Page 34 Siemens Mobility Services
Application Container
• Apache Tomcat
• Deploy Jolokia as webapp
• Write Icinga2 Plugin in Java to query metrics
• Two different approaches:
• check logic is inside plugin
• get „raw data“ via performance data output & use dummy checks
to evaluate results (Icinga2 Language)

35. Unrestricted © Siemens AG 2017
March 2017Page 35 Siemens Mobility Services
Application Container
•Tomcat Examples:
• Heap Memory Usage (max/used ratio)
• Connector Stats or App Context
• Detect Deadlocked Threads

36. Unrestricted © Siemens AG 2017
March 2017Page 36 Siemens Mobility Services
Heap Memory Usage (https://jolokia.org/tutorial.html)

37. Unrestricted © Siemens AG 2017
March 2017Page 37 Siemens Mobility Services
Heap Memory Usage – IcingaWeb2

38. Unrestricted © Siemens AG 2017
March 2017Page 38 Siemens Mobility Services
Check Logic should be dynamic
1. Do active checks via custom plugin (import as performance data
into Icinga2)
2. Use dummy service checks to evaluate performance data

39. Unrestricted © Siemens AG 2017
March 2017Page 39 Siemens Mobility Services
Active Checks
https://nagios-plugins.org/doc/guidelines.html#AEN200

40. Unrestricted © Siemens AG 2017
March 2017Page 40 Siemens Mobility Services
Dummy Service Example

41. Unrestricted © Siemens AG 2017
March 2017Page 41 Siemens Mobility Services
Dummy Service Example

42. Unrestricted © Siemens AG 2017
March 2017Page 42 Siemens Mobility Services
Dummy Service Example
•Dummy Services can also output performance data J

43. Unrestricted © Siemens AG 2017
March 2017Page 43 Siemens Mobility Services
Icinga2 Language
• Very(!) powerful
• Lambda functions, runtime object access, loops, conditions, etc.
• Check out the blog post by M. Friedrich „Icinga 2: Host state
calculation from all services” as an example.
(https://www.icinga.com/2016/06/09/icinga-2-host-state-calculation-
from-all-services/)

44. Unrestricted © Siemens AG 2017
March 2017Page 44 Siemens Mobility Services
Data Hierarchy
Application
Application
Container
OS

45. Unrestricted © Siemens AG 2017
March 2017Page 45 Siemens Mobility Services
Application
• Same procedure:
• One active check gathers all important data into Icinga2 via
performance data output
• Dummy Service Checks evaluate the data & display the results

46. Unrestricted © Siemens AG 2017
March 2017Page 46 Siemens Mobility Services
Application: Example ActiveMQ
• ActiveMQ:
• „Apache ActiveMQ™ is the most popular and powerful open source
messaging and Integration Patterns server.”
• Is used to queue/dequeue train messages
• Uses jetty as a webserver or Servlet/JSP-Container
• Monitoring via JMX

47. Unrestricted © Siemens AG 2017
March 2017Page 47 Siemens Mobility Services
Application: Example ActiveMQ
• Use jolokias’ bulk request feature for querying multiple requests:

48. Unrestricted © Siemens AG 2017
March 2017Page 48 Siemens Mobility Services
Application: Example ActiveMQ

49. Unrestricted © Siemens AG 2017
March 2017Page 49 Siemens Mobility Services
Application: Example ActiveMQ

50. Unrestricted © Siemens AG 2017
March 2017Page 50 Siemens Mobility Services
Acknowledgments
•Entire Icinga Community – for making and sharing such great
open-source software
• Netways – for having me as a speaker
• YOU – for listening to my talk
THANK YOU!
J

51. Unrestricted © Siemens AG 2017
March 2017Page 51 Siemens Mobility Services
Icinga2 @ Siemens Mobility Customer Services - Monitoring Train IT
Siemens AG
Mobility Division
Customer Services
Support Center Security & Operation
Nuernberger Str. 74
D-91052 Erlangen
Railcover, 100% Railability, SITRAIL, CORMAP, Sinalytics, Easy Spares, Easy SparovationPart, SIMOS Smart
Monitoring are registered trademarks of Siemens.
The information in this document contains only general descriptions and features that, in a specific application case, do
not always apply in the form described or may change as a result of further development of the products. Features are
only binding if they have been expressly agreed at the time the contract is signed.
siemens.com/mobility-services