This document discusses integrating IBM Z (mainframe) systems into ServiceNow and Splunk using Ironstream. It describes how Ironstream can provide 360-degree visibility across IT infrastructure, including mainframes, by enabling discovery and event management in ServiceNow and real-time log collection and normalization for analytics in Splunk. Use cases discussed include monitoring operational status, reducing downtime, meeting SLAs, detecting security threats, and achieving compliance. Customer stories demonstrate how Ironstream helped organizations achieve a single source of truth across all systems in ServiceNow and support optimal service delivery and PCI compliance by including mainframe data in Splunk.
My INSURER PTE LTD - Insurtech Innovation Award 2024
Integrating IBM Z into ServiceNow & Splunk
1. Downtime
is Not an Option
Integrating IBM Z into ServiceNow & Splunk
Ian Hartley I Product Management Director
® ®
2. Downtime is Not an Option
• : Outages happen
• : Failures & missed SLAs are costly
• : Reputations are damaged
Online outages, delays continue to impact Costco shoppers
The issues were a major problem on Thanksgiving and again on Black Friday
*https://www.atlassian.com/incident-management/kpis/cost-of-downtime
Average estimate…
$9K per minute!*
5. Wait a minute…
We are talking about MAINFRAMES
They are rock solid & secure…
…right!?
6. Maybe…
• Mainframe is not a static environment
• System & application changes take place
• Most system outages are due to human error
…and mainframe is no exception
• Mainframe remains a mission-critical platform
• Supports vital services connected to the rest of the
enterprise stack
• When a service fails… the broader organization needs
visibility into the mainframe to trouble-shoot
…and resolve FAST
• Better still? Get visibility of issues BEFORE they occur
7. Today’s IT is Complex – Need 360° Visibility
7 Mainframe
8. Legacy IBM systems are left out of today’s
leading IT analytics & operations platforms
Distributed and
Cloud environments
Mainframe and
IBM i Systems
IBM Z
Mainframe
IBM i
System
8
9. Big Iron to
Big Data
Analytics
Challenges
Systems Management Facility
(SMF), Syslog, Log4j web and
application logs, RMF, RACF,
USS files and standard datasets
Complex data structures
(SMF) with headers, product
sections, data sections,
variable length and self-
describing
• EBCDIC not recognized
outside of the mainframe
world
• Binary flags and fields
9
Millions of log records
generated daily – 9.7TB
average daily mainframe log
data …and growing
Not real-time, typically have to
wait overnight for an offload
Typical daily FTP
upload/downloads can’t get
granular
10. Ironstream Removes the Barrier
Enables 360° visibility
IBM Z
Mainframe
IBM i
System
360°
view across
the enterprise
10
11. Use Cases
• Monitor operational status of enterprise IT infrastructure
• Make better decisions to take control of the IT infrastructure
• Monitor resource utilization & availability
• Problem detection & isolation
• Reduce MTTI, MTTR
• Meet SLAs
• System health, KPI monitoring with Splunk IT Service Intelligence
• Detect & mitigate security threats
• Privileged activity, anomalies, data movement
• Achieve compliance
• Pass audits
• Comprehensive surveillance with Splunk Enterprise Security
14. Ironstream Integration with ServiceNow Discovery
• Rapidly configure and launch secure discovery of IBM i
and mainframe resources and their relationships
• Auto-populate and maintain the ServiceNow
Configuration Management Database (CMDB)
• Automatically map dependencies & assign relationships
• Get a single view of entire infrastructure to enable
smarter IT decisions
• Reduce decision times and errors, increase productivity
with intelligent automation
14
15. MID Server
Agent
Mainframe or IBM i LPAR
Discovery Agent
MID Server
IMSMQ
Db2 CICS
Network
Host
Resources
Probes sent to run
discovery scripts
Scripts execute
commands
Client runs command
against agent
Agent executes
commands on LPAR
Agent sends output
back to client
1
2
3
4
5
6Sensor parses output
and creates CIs
Resources and Subsystems
Discovery Workflow
REXX
VTAM
MQ Commands
MVS
Db2 Queries
Ironstream
MCS
16. Ironstream for
ServiceNow
Discovery
• DB2 - DDF, DSG, databases,
table spaces and deep
configuration data
• Completed jobs
• DASD storage
• Storage groups
• CICS - regions, transactions,
programs
• IMS - regions, databases,
transactions, programs
• MQ - managers, channels,
queues
• Memory
• LPAR
• CPU
• Network connectivity
• Installed IBM/non-IBM software
• Local Storage with ASPs
• Memory
• LPAR
• CPU
• Network connectivity
• Installed software
• Selected system values
• Subsystems
• Active jobs
• Job queues
• Output queues
• Libraries
• Program objects
17. Ironstream Integration with ServiceNow
Event Management
• Extends cross-platform capabilities of
ServiceNow ITOM to include mission-critical
IBM mainframe & IBM i environments
• Significantly reduces event noise and floods
generated by third-party monitoring tools,
• Monitor service health
• Prevent outages
• Easily take action
• Sophisticated z/OS and IBM i event status
management of any messages which go
through the console to establish proactive
enterprise systems management
MAINFRAME 1
MAINFRAME 2
18. MID Server
Agent
Mainframe or IBM i LPAR
Ironstream
MCS
Event Mngmnt
Agent
MID Server
JES,
SDSF
MQ
Db2 CICS
Network
Host
Resources
Integrates with
Event Management
Passes to MID Server
MCS filters & formats
messages/information
Agent detects
messages, runs scripts,
commands etc. on LPAR
Agent sends output
to client
5
4
3
1
2
6
Leverages workflow to
process & automate
Resources and Subsystems
Event Management Workflow
REXX
VTAM
NetView
MVS
TCP/IP
19. Ironstream for
ServiceNow
Event
Management
• 110+ Event Rules including:
• System Console/Syslog Events
• CICS Transient Data Queue
Events
• IMS Master Terminal Operator
Events
• Interval Monitoring
• SMS Group Threshold
Monitoring
• MQ Manager, Channels,
Queues
• Active Jobs
• RMF based Performance
Monitoring
• Custom Message Interface
• Message Automation
• Reliable TCP Communication
• Heartbeat and positive message
acknowledgement
• Message Buffering
• Command Console
• 170+ Rules for IBM i, including:
• AS/400 State
• ASP State
• Audit Journal Alerts
• Job Queue State
• Job/Subsystem State
• Memory Pool State
• Threshold Monitoring
• I/O per second
• TIMW Status
• Message Queue Alerts
• MQ Series States
• TCP Connection Status
• CMTW Status
• MTXW Status
• Output Queue State
• Services State
• Agent Connection
• Wait Status Monitoring
24. Splunk: Industry-Leading Platform For Machine Data
Online
Services
Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Application
s
Custom
Apps
Messagin
g
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
On-Premises
Private
Cloud
Public
Cloud
Enterprise Scalability
Universal Indexing
Developer
Platform
Report &
analyze
Custom
dashboards
Monitor
& alert
Ad hoc
search
!
24
25. Comprehensive Security & Operational Metrics
Disk Information
• Reads/Writes
• Disk Capacity
• Disk Space Availability
• Disk Busy
• Disk Response Times
Job Information
• CPU used
• Socket sends/receives
• Stream file, directory & Symlink reads
• Stream file writes
• Seize/Wait time
• Communication Puts/Gets
CPU information Per Virtual CPU
• Time used
• Number of CPUs active
TCP communications
• Detailed stats at Datagram
• Fragmentation information
Physical Processor information per CPU
• Time used
• Owning Partition
Virtual Processor information per Virtual CPU
• Status, Time active, Time used.
• Configured/Uncapped available time
• Instruction count
Memory pool information per Pool
• Database faults
• Non-database faults
• Job transitions Size
• Disk I/O stats
• Pages aged and stolen
Job summary information
• CPU used
• Disk I/O detail
• Database/Non-database
• Page faults
• I/O Pending faults
Security Information
• User Profiles
• System Values
• Object attributes & authorities
• Authorization Lists, Job Descriptions
• Commands
• Active Jobs, Spool Files
• Changes to values, authorities, profiles, auth. lists
• Access attempts (authentication or object access)
• Sensitive object access
26. Four Key Use Cases addressed with Ironstream
and Splunk
• Response times/SLAs
• Latencies
• Exceptions
• Resource utilization
• Sensitive data access
& movement (PII/PHI)
• Configuration settings
(e.g. FISMA)
• IRS Pub 1075
• PCI DSS
• Incident triage
• Anomalous behavior
detection
• Glass table view of
entire service process
• Predictive analytics
• User Authentications
• Account & login activity
• FTP sessions & file
activity
Security
Operational
Intelligence
IT
Monitoring
Compliance
26
27. Precisely
Ironstream
for Splunk
360ᵒ View:
• High performance, real-time
collection of IBM mainframe
information
• Normalizes the z/OS data so it can
be used by Splunk
• Same Splunk dashboards, bigger,
more complete data sets; free apps
• Network managers, security
analysts, application analysts,
enterprise architects can use
without requiring mainframe
access or expertise
27
28. What does Ironstream provide for Splunk?
28
• High performance, cost-effective platform for collecting critical
log, machine, and event data
• Normalization of mainframe and IBM i data for off-platform
analytics & operations engines, including cloud
• Completes the enterprise-wide picture of IT infrastructure
• Better visibility
• Better agility
• Better control
• Addresses the SME challenge: Used by network managers,
security analysts, application analysts, enterprise architects
without requiring detailed mainframe or IBM i access or
expertise
29. 29
Example Dashboards powered by Ironstream
Security
• Authorization Failures
• Change Profile Events
• System Value Changes
• User Activities
Operations
• Capacity Monitoring
• CPU Utilization
• Create/Delete objects
• Disk Performance
• Job Durations
• LPAR Performance
• Message Queue Events
• System Performance
Application Data
• Employee Database Use Case
Splunk Dashboards
30. Ironstream for Splunk works with Mainframe Data
Precisely
Ironstream
Data
Forwarder
TCP/IP
Ironstream
Desktop
DCE IDT
Data Collection
Extension
Real-time Collection
Assembler C,
COBOL,
REXX
!
Data Sources
HTTP(S)
SMF RMF
File
Load
Log4j IMSSYSLOG
SYSLOGD
System
State
SYSOUT
Live SPOOL
Db2 USS
Alerts
Network
Components
Forwarder
API
31. Ironstream for
Splunk
Splunk
Enterprise
IT Service
Intelligence
Powerful insights of your enterprise
for IT Operations and Security with
Ironstream for Splunk
Precisely Ironstream integrates with
the Splunk ITSI premium app to
predict and prevent service
degradation with a unified
monitoring experience
Enterprise
Security
Data Model
for Mainframe
Precisely Ironstream integrates with
the Splunk Enterprise Security
premium app to provide enterprise-
wide view of security across all
platforms
The Precisely Ironstream Data
Model provides a structured and
logical view of mainframe log data
elements in Splunk for faster
searching, analysis and Splunk
development
34. Achieving a Single Source of Truth
with Ironstream for ServiceNow
Need for visibility across all IT
infrastructure in ServiceNow –
including mainframe
• Rely on ServiceNow CMDB as
“single source of truth”
• No comprehensive coverage
for mainframe
100’s business application touch
mainframe
• Mainframe key resource
• No insight – constant demand
on mainframe team
• Manual integration
• Too costly
• Impossible to maintain
• Out-of-date before complete
Ironstream for ServiceNow
• Certified ServiceNow solution
• Simple install & configuration
• Seamless integration with
ServiceNow Discovery
• Auto-populates & maintains
CMDB
• Auto-maps dependencies &
relationships
• Improved IT visibility
• Complete, accurate, up-to-date
CMDB
• Visibility of all IT infrastructure
• Visibility of relationships,
connections & dependencies
• Better agility
• Improved service availability
• Better change management
• Significant reductions
• People-hours
• Related costs
Challenge Solution Results
35. Supporting
optimal service
delivery at U.S.
based Loan
Service Provider
with Ironstream
for Splunk
O B J E C T I V E
• To monitor mainframe IT operations to
track health of service delivery for Loan
Service Providers
• Capture mainframe business data in
support of system and application
monitoring in Splunk
C H A L L E N G E
• Required several data feeds including SMF,
SYSLOG and SYSOUT for batch job
monitoring
• Filtering the log data to selected jobs
• Loading business data from sequential files
S O L U T I O N
• Precisely Ironstream forwarding required
log data and filter it to specific messages
and jobs
• Splunk for IT operations analytics
B E N E F I T
• Increased visibility to support optimal
service delivery for loan service providers
• Fast time to value, with ease of installation
and configuration, in contrast to
competitive solutions
36. European bank
tackles PCI-DSS
compliance with
Ironstream for
Splunk
C H A L L E N G E
• Working against a tight deadline to
build a solution with Splunk to
continuously monitor all relevant
PCI DSS requirements.
• Needed a proven, easy-to-use
solution to include their busy,
complex mainframe environment,
which included 6 mainframes and
900+ production CICS regions
S O L U T I O N
• Ironstream for Splunk:
• Seamless integration to include
mainframe log data into Splunk
• Proven to be easier to install,
configure and use vs. competition
B E N E F I T
• Compliance with PCI DSS mandates
• Single, enterprise-wide monitoring
solution for all systems, including
mainframe
There’s the added dimension that outages and service degradations are more public than ever before. There are monitoring agencies and web sites dedicated to tracking downtime…
And, beyond that – not only is everybody watching, but as soon as something goes wrong, everyone is sharing it. Here we have social media posts about some recent, high-profile outages at Costco and Ticketmaster.
Costco had a real hard time over Black Friday 2019 – one of the most important days of the year for a retailer – where their online shopping went down. It was down for 16 hours! It costs them $11 million dollars – and that’s just from lost revenue. On top of that, their failings were shared across social media. And all of this because of an internal server error – if you can’t read the message, it says “The server encountered an internal error or misconfiguration and was unable to complete your request.” I wouldn’t want to be the IT manager responsible for that server, would you?
The more your business relies on digital, and the higher your customer expectations, the worse an outage can be. For example, Ticketmaster users were extremely animated voicing their displeasure online when trying to buy tickets to a sporting event and there was a system error.
So – in short – our customers need to have the power to prevent outages before they happen – and when that’s not possible, they need to identify and fix them as soon as possible, to limit the negative impact to the business.
So it is vital that organizations get a good grip on what is occurring across their I.T. landscape.
However, IT today is extremely complex, with systems that are both interconnected to deliver services, yet silo’d from a management, security and tools perspective. This stands in the way of the awareness, agility and availability that’s required. And those silo’d tools are very much the problem when it comes to mainframes and IBM i systems because platforms like Splunk don’t natively integrate with them – but they are critically important.
This slide here actually shows you part of a real network – and it came from one of our customers. There are a lot of moving parts in here. It is complex. It has to be available. It has to be reliable. To ensure this, the customer needs visibility into what's going on. So what is happening in that big black box at the bottom, that which happens to be the mainframe – but could have also been IBM i.
You can see that it’s a major component in some critical systems. And they need to see what is going on inside that box. Without a tool like Ironstream, they're really going to struggle to tap into that and see what's going on in there – in their mainframe or IBM I -- alongside all the other moving parts and pieces across the infrastructure.
And that’s the visibility that Ironstream provides.
Today’s leading modern platforms do an excellent job for distributed and cloud environments
Assure Monitoring and Reporting
At ServiceNow, our core principle is straight forward. ServiceNow makes work, work better for people. Transform old, manual ways of working into modern digital workflows, so employees and customers get what they need, when they need it—fast, simple, easy.
A key component is delivering high-performance business services with visibility and AIOps. While I just covered some of the major challenges, ServiceNow helps establish visibility, deliver healthy services, and optimize spend.
For visibility, the key is establishing a complete, current, and accurate view of resources and assets across your entire operations estate and provide service-aware context for your most important apps and services.
For health, delivering high-performance business services requires a complete understanding across the organization by leveraging AIOps to help identify, isolate, and resolve business-impacting issues.
For optimization, eliminating manual processes with automation empowers IT teams and organizations to get a handle on the exploding growth of cloud and software spend and reduce the impact of planned and unplanned audits.
ServiceNow helps achieve these goals with our IT Operations Management and Software Asset Management capabilities.
The combination of Ironstream and ServiceNow enables enterprises to auto-populate and maintain the Configuration Management Database (CMDB) with all major IBM mainframe and IBM i Configuration Items (CIs) and their relationships.
Alors, comment fait Ironstream ?
Un agent sur le mainframe ou l’IBMi utilise des sondes et des capteurs pour découvrir les composants.
Ceux-ci sont communiqués au ServiceNow MID server, qui à son tour envoie les informations à ServiceNow fonctionnant dans le Cloud où les items de configuration du mainframe et IBMi apparaissent dans les écrans et les tableaux de bord de ServiceNow.
Tout cela peut fonctionner en mains libres, de manière automatisée et programmée. Vous obtenez ainsi une vue précise et actualisée de votre mainframe et de votre paysage IBMI.
Très simple et efficace.
Turning to Event Management…
High performance, low-cost, platform for collecting critical system information in real-time
Normalization of the z/OS and IBM i data so it can be used by off platform analytics engines
Full analytics, visualization, and customization with no limitations on what can be viewed
Ability to easily combine information from different data sources and systems
Address the SME challenge: use by network managers, security analysts, application analysts, enterprise architects without requiring mainframe access or expertise
The SMF and log data on your mainframe holds the key to true insights about your complete enterprise, but if this machine data stays silo’d within your mainframe team, you’re essentially flying half-blind. Include ALL the relevant data for Splunk to correlate, and for you to analyze, in your Splunk Enterprise, Splunk Enterprise Security and/or Splunk IT Service Intelligence.
A key player in the U.S. secondary mortgage market where they buy loans from approved lenders. They had been using Splunk to monitor the health of their critical applications but their IBM mainframe data was not being included, creating a blind spot in tracking the health of their service delivery for their Loan Service Provider customers. After implementing Ironstream, they have integrated the mainframe data with the other important system data in Splunk and now have a comprehensive, end-to-end view of their application and system health.