Data Virtualization Manager for z/OS

IBM z Analytics Data Virtualization Manager
IBM z Analytics
IBM Data Virtualization Manager for z/OS
Leverage data virtualization synergy
with API economy to evolve the information architecture on IBM Z
Jørn Thyssen | jthyssen@rocketsoftware.com

Agenda
IBM Data Virtualization Manager V1.1
• Big Data vs. Dark Data
• Traditional Data Integration
• Mainframe Data in the API Economy
• Demo
Db2 Tools Update
• Continuous delivery
• REST enablement of tools

Speed of business forcing need for
real-time data
There is simply too much data for older
architectures
Data resides in silos and unfamiliar formats
Need for more data agility to
support cloud, mobile…

Dark data has to be made accessible and
transformed to a useable format before it can be
turned into business insight

Traditional data integration inadequate
6
Becoming no longer viable to move data
(using extract, transform, load ETL)
• Risk to data security
• Data inconsistency
• Rigid, limits business agility
• High cost and latency

A New Approach - Data Virtualization
Data virtualization enables data structures
that were designed independently to be
leveraged together, in real-time, without
the necessity of physical data movement.

How Customers use Data Virtualization
8
Difficult access to data
(SMF, non-relational, MQ)
Need immediate insight into your
customer or business
Real-time Analytics
Real-time Z Data is a needed
via multiple APIs
Faster, easier delivery of digital
systems of engagement
Modernization
Improve efficiency/performance for
any application needing Z data
Too costly and time-consuming for
applications to get Z Data
Optimization

Data Virtualization Technology Accelerates Real-time Analytics
9
Data Virtualization
Manager for z/OS
DB2 Analytics
Accelerator Loader
QMF for z/OS
Machine Learning
for z/OS
Open Data Analytics
for z/OS
Predictive Analytics
Discovering patterns/meaning in data
Query Acceleration
High-speed processing for
complex Db2 queries
BI Solution
with Access, Virtualization
and Visualization
and Data Preparation
Common metadata maps to share
and reuse
Universal Data Access
Data Virtualization for all enterprise data –
on and off mainframe

QMF - Data Virtualization Expands Insight into VSAM, IMS,
Adabas
10
DB2
VSAM
IMS
Adabas
QMF
In-Memory
Data
Virtualization
• Easy mapping wizard. Setup once, then data can be used by any QMF user
• Federation is virtualized ”in-memory” in z System
• Exploits specific z Systems hardware and zIIP specialty engine
Relational DBs
DB2 LUW
MS SQL
Oracle
Teradata
Sybase
PostgreSQL
Non Traditional
Excel
Flat File
TPF
Hadoop
Spark
Website
data

Applying Data-in-place Analytics within Enterprise:
IBM Open Data Analytics for z/OS
11

Some of the concepts explained…
• Virtual tables to access data sources
• Hardware dependant optimized code base
• Architectural rules
• Parallelism / Map Reduce
• No Staging
• Health-level monitoring
• Virtual Parallel Data
• Performance

13

Enabling APIs for Data on IBM Z
CICS
IMS
WAS
MQ
DB2
 Agile web, mobile, cloud access via REST to
mainframe applications such as CICS, IMS, WAS…
CICS
IMS
WAS
MQ
DB2
z/OS Connect
Enterprise Edition
Trusted zone
Loadbalancer
IntelligentContentrouting
Gateway/DataPower
Securitylayer
DMZ Inner APIsOuter APIs
Discovery Services
REST Services
REST API
Consumers

Enabling APIs for Data on IBM Z
CICS
IMS
WAS
MQ
DB2
Data
Virtualization
Manager for z/OS
Adabas
IMS DB
IDMS
SMF
VSAM
 Agile web, mobile, cloud access via REST to
mainframe data such as Adabas, IMS, VSAM…
CICS
IMS
WAS
MQ
DB2
z/OS Connect
Enterprise Edition
Trusted zone
Loadbalancer
IntelligentContentrouting
Gateway/DataPower
Securitylayer
DMZ Inner APIsOuter APIs
Discovery Services
REST Services
REST API
Consumers
DVM Non z/OS

The Data Catalog Perspective

Map a Data Source to a create a Virtual Table

Automated generation of code to access data from Spark &
Python
➢ Reduce coding of data access via automated generation of Scala code
(Python in progress)
➢ Automatically optimized based on available parallelism to generate
best performance results without manual coding

Adabas Db2 for z/OS
IMS DB IDMS
SEQ files VSAM
SMF records Log Streams
Virtual Tape zFS
Db2 Warehouse
Db2 Warehouse
on Cloud
Db2 Informix
MAINFRAME
z/OS Systems
Distributed
More than just data access …
JDBC/ODBC
REST AP/SOAPI
NoSQL
Access to all data from all applications
without
need to know which format the data it is in
using the API of your choice
IBM Data Virtualization
Manager for z/OS
IBM z/OS Connect EE

Selective direct Distributed Relational Database Architecture support
Manager for z/OS
Oracle
Data
Provider for DRDA
(Linux)
Oracle
SQL
Server
Microsoft
Service for DRDA
Apache
Derby
DRDA

MongoDB API for Mainframe Data
• Enables cloud/mobile developers to use
mainframe data as if it was MongoDB
MongoDB API
Socket Layer
Query Translator
Message Processor
EXECSQL
Mongo BSON
Wire Protocol
• MongoDB API for
accessing:
– Adabas, Db2, IMS, IDMS,
VSAM, Physical Sequential,
IBM MQ….
• Accelerates time to value
for mainframe participation
in digital initiatives

Cobol
DSClient API for z/OS
programs, e.g., Cobol or
assembler
Use case example:
• Extend existing z/OS
programs with access to
new data sources

MQ Queues and Messages
• Eliminate the need to
extract data from MQ
• Perform SQL-based
analytics directly
against MQ data
• Join any DVM data
sources with MQ
messages
Queue Manager
Non-relational Data
Adabas IDMS IMS VSAM
MQGET MQPUT
SMF
SQL JSON Services
z/OS
Coming
Soon
PI92252

Where are opportunities for Data Virtualization?
• Digital applications needing access to Z data…
• z/OS Connect and APIs for web, mobile, cloud
• Any application that requires access to Z data
• Federation Server, DataStage, Optim, QualityStage,
Cognos…
• Hadoop Data Lakes, ETL, MongoDB…
• Heritage Z data customers wanting to exploit
their data in:
• VSAM, IDMS, IMS, ADABAS, Physical Sequential, SMF

Faster time to insight
Enabling portfolio managers to provide timely
investment advice
17 million VSAM records
Huge data volumes – 15 VSAM files concatenated
together brought back 17 million records
Real-time information
For business analysts who no longer waited for data to
be loaded
Customer Use Case: Self-service Analytics for Investment Advice
• Prior to doing analytics, analysts had to enlist DBAs to create reports from VSAM, a complicated, multi-step extraction process
• IBM Data Virtualization Manager enabled real-time access to IMS DB and VSAM data from the online dashboard of the business
intelligence application
• Analysts can respond faster to business enabling portfolio managers to make more relevant, timely investment suggestions
Global Financial Services Firm

Improved operational efficiency
Overcame time delays associated with inefficient
batch processes
From days to milliseconds
Online account origination went from 3 days to
200 milliseconds,
API-enabled IBM Z apps/data
Enhanced developer productivity with APIs to policy
and actuarial data in IMS DB
Customer Use Case: Modernization to Enhance Online Service
• New online customers at major insurance company had to wait days for confirmation of coverage when adding a new insurance product
• Actuarial data in IMS DB was API-enabled using IBM Data Virtualization Manager, allowed developers to incorporated risk calculation
and cost estimates into new online service.
• Online policy origination went from 3 days to 200 milliseconds, and registered 400+ new policies in the first 2 weeks of going live.
North American Insurance Firm

Faster Time to Insight
accessing more than 7 million records went from 12
hours to less than 13 minutes
Mountains of data to process
Poor data quality, complicated by millions of records
to process took 12 hours to load
Improved Operational Efficiency
Eliminating large batch processes provided faster time
to insight and reduced costs, with 93% of processing
running on zIIP engine
Customer Use Case: Optimized Access to Z Data for Loan Processing
• Processing student loan requests took too long creating regulatory compliance issues.
• With IBM Data Virtualization Manager for z/OS accessing more than 7 million IMS records, the cycle went from 12 hours (via ETL) to
less than 13 minutes.
• Complex joins were performed in-memory on the mainframe, with 93% of related processing running on the System z Integrated
Information Processor (zIIP).
Govt. Student Loan Processing

Simple
Get transactional access, no data movement
Open to all Apps
Modern APIs enable access
Secure
Avoid risk by reducing moving data off Z
Systems
Fast
Exploits Z architecture, including parallelism and
in-memory processing
Cost Effective
Keeps Z costs down with up to 99% zIIP offload
Non z/OS data
Manager for z/OS
Optimizing access to all Enterprise Data

Db2 Tools Update
30

31
Continuous Delivery – IBM Db2 Tools

IBM z Analytics Data Virtualization ManagerIBM z Analytics
32
IBM Db2 Tools
• Many new function APARs for
– Db2 Administration Tool & Db2 Object Comparison Tool
– Db2 Query Monitor
– Db2 Recovery Expert
– Db2 Cloning Tool
– Db2 Automation Tool
– Db2 SQL Performance Analyzer
– Omegamon XE for Db2 Performance Expert
– …
• Tools support (toleration / exploitation) Db2 12 for z/OS support
– http://www-01.ibm.com/support/docview.wss?uid=swg21972885
• Contact your local friendly IBM’er for further details
Db2 for z/OS Db2 Tools

33
Quotes from digital transformation front lines…
• “the key drivers of digital transformation are profitability, customer satisfaction, and increased speed-to-
market.”
• “Lean product management practices help teams ship features that customers actually want, more
frequently. This faster delivery cycle lets teams experiment, creating a feedback loop with customers.
• “Automation is a huge boon to organizations. With more work automated, high performers free their
technical staff to do innovative work that adds real value to organizations”
• “Implementing technologies and processes that enable developer productivity, reducing code
deployment lead times and supporting more reliable infrastructures.”
• “Supporting team experimentation and innovation, to create and implement better products faster.”
• “Innovation in Business: New digital products and/or digitizing existing business models that goes
beyond existing business needs and foster new innovative products and service and cater to
changing business needs.

34
Self-services – (not exactly a novel concept)
• 1880 – Vending Machines invented – first time a customer could purchase without assistance of a vendor.
• 1916 – Piggly Wiggly (Clarence Saunders) introduces “self-service grocery stores” (go get the stuff yourself)
• 1947 – Self-service gas pumps
• 1969 – Chemical Bank installed the first ATM
• 1994 – Amazon and Ebay online shopping
• 2006 – Amazon web services officially relaunched – S3 cloud storage, Simple Queue Service, EC2
– “enables developers to focus on innovating with data, rather than figuring out how to store
it.”
• 2018 – Db2 for z/OS provisioning and self-service portals???

35
Self-service – for developers
• Automation
– No ISPF
• Will be used for non-z
developers
– Again: No ISPF
• Should be able to imbed into portals and deployment tools like IBM
UrbanCode Deploy
– REST API preferable

z/OSMF
REST Server Workflow
Manages z/OSMF Workflows
Access to z/OS Resources
(SDSF, ISPF, etc.)
Numerous z/OSMF REST
Interfaces (see table)
REST interfaces for
Workflows
Workflow Definition File
(XML)
Steps
Workflow variable input file
Templates
STCs: IZUSVR1
IZUANG1
▪ When using FTP to place the files in USS HFS
▪ Use binary transfer
▪ Browse or Edit can be used to view the files with the Data
Encoding: ASCII option
IBM z Analytics
z/OS Mgmt Facility for REST

37
REST interfaces for IBM Db2 for z/OS & IBM Db2 tools
• Available today through z/OS Management Facility
– IBM Db2 Administration Tool (z/OS MF artifacts distributed in SADBSAMP library)
– IBM Db2 Cloning Tool (obtain z/OS MF artifacts through your local friendly IBM’er)
– IBM Db2 Utilities (no z/OS MF artifacts officially available – I have examples)
– … anything running as JCL, REXX, USS, …
• Available today through native REST API in Db2 for z/OS
– Db2 supplied stored procedures, like ADMIN_COMMAND_DSN, ADMIN_COMMAND_DB2,
ADMIN_INFO_SYSPARM, etc.
• Future plans
– Native REST interfaces for IBM Db2 Tools
– Much more functionality to come (autonomics, recovery, monitoring, …)
Contact me for more
details!

38
Admin Tool and Cloning Tool
• Provisioning & Snapshot/Refresh
REST interfaces
Db2
Provisioning
Request
AOC REST Calls execGEN
runDDL
CKZ REST Calls CKZcloneS
CKZcloneT
createWF
createWF
Db2
Provisioning
Complete
Db2
Data Refresh
Request
CKZ REST Calls CKZcloneS
CKZcloneT
createWF
Db2
Data Refresh
Complete
REST interfaces
Provisioning option Data refresh/Create Snapshot

Demo of REST calls (if time allows)

40

Data Virtualization Manager for z/OS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Virtualization Manager for z/OS

Similar to Data Virtualization Manager for z/OS (20)

Recently uploaded

Recently uploaded (20)

Data Virtualization Manager for z/OS