Powerup performance of Informatica Environment Nov 5, 2015

Power up performance of your
Informatica Environment
Wise Men Confidential
www.wisemen.com | info@wisemen.com | +1 281-953-4500

Disclaimer Notice
Wise Men Consultants, Inc. / Wise Men Solutions LLC (“Wise Men”) specifically disclaims any
and all responsibility for the content or any material submitted during this presentation. All
information hereunder is disclosed “as is” without warranty or guarantee of any kind as to its
accuracy, completeness, operability, fitness for any particular purpose, merchantability, or any
warranty express, implied or statutory. Wise Men shall not be liable to any one for any
damage, expenses, or claim of loss of any kind or character (direct, indirect, or consequential)
arising from the use of or reliance on the information shared herein. You may not copy, use,
reproduce, republish, transmit or exploit any information provider herein without the written
consent of Wise Men authorized representative. Any and all information provided in this
presentation is proprietary and all rights including but not limited to intellectual property
rights in these belong to Wise Men. By attending this presentation any further you hereby
agree to abide by any and all intellectual property notices and restrictions contained herein.
2

Speaker Profile
Shadab Alam , Technical Lead
 As a technical lead of Integration and Information Services Practice,
 Shadab is responsible for development of best practices,
methodologies, solution frameworks & providing technical
assessment and consultancy to new and existing project
engagements at Wise Men.
 Shadab has over 10 years of IT experience in multiple industries
including banking and finance, Pharmaceutical, Hospitality etc. He
has strong knowledge of various datawarehouse & BI products such
as Informatica PowerCenter, Microsoft BI tool stack, SAP Business
Objects, TIBCO Spotfire, Oracle PL/SQL, Teradata and utilities.
3

 Introductions
 Wise Men Overview
 Data Integration Performance Challenges
 Effect Of Poor Performance On The Organization
 What are the Sources of The Problem ?
 Various Best Practices
 Case Studies
 Summary
 Q & A
Agenda

Wise Men Confidential 5
Established in 1997 and
headquartered in Houston, Wise
Men is a global technology
services and solutions provider in
the integration and ERP space.
We are a Women and Minority
owned company with over 550
employees and contractors
worldwide.
In 2000, we started our
Middleware operations with
SeeBeyond’s DataGate and
eGate. We helped our customers
in the seamless technology
integration in the Egate, JavaCAPS
and Fusion space. We pride
ourselves as being integration
experts in platforms like
Informatica, TIBCO, Fusion, Web
Methods, SAP PI, GIS and few
others
USA, India (Hyderabad and
Pune) and Canada
About us History
Global
Presence
Philosophy: Integrate, Analyze, and Prosper
Our Motto: High customer satisfaction – We never quit!
Wise Men Corporate Overview

6
• Integration and Information Strategy and blue prints
• Business Process Automation
• SOA / EAI / B2B / EDI / DI / ETL / DQ / MDM
• TIBCO, Informatica, Oracle Fusion, SAP PI, GIS
Integration
platform Services
• SAP & Oracle
• Custom application development
• Modernization, Testing & Legacy Migration
• Application Management Services
Application
Services
• Infrastructure / Middleware Management Services
• Helpdesk, Incident Management, Problem management
• Middleware Migration Services
• 24/7 Database, Linux services
• Solution manager based SAP help desk
Managed
Services
18+
Years in the
industry
8+
550+
Vertical specializations
5Locations across
the globe
Employees globally
Offerings

Wise Men Confidential 7
•Integration Architecture Assessment and Strategy Consulting
•Integration Initiatives Business Value Assessment
•Application and Enterprise Integration Architecture Consulting
•MDM and BI Architecture and Strategy Consulting
Architecture
•Development services for EAI, SOA, ETL and B2B Initiatives
•Infrastructure & capacity planning, go-live/cut-over planning and
support
•End to End MDM implementation
•Data warehouse, Data Quality and Business Intelligence
Development
•Infrastructure setup and maintenance
•Automated deployment and promotion framework
•Disaster Recovery setup and assessment
Administration
Services
•Application Wise Desk
•24 X 7 Operations support, levels 1, 2 and 3 using onsite, onshore
and offshore development centers
•Migration & Upgrades, Training
•MicroSourcingTM
Managed Services
and Sourcing
WiseMenBTAFramework
Integration Practice Service Offerings

Integration Platform Services
8
EAI B2B MDM SOA BRM ETL
Deep Experience In Variety of Projects Migrations
Product Migration
Vendor or
Platform Migration
Wise Men BTA Framework
Reference Architectures,
Patterns & Decision Trees
Integration Standard
Documents
Project Plan templates
Platform Migration &
Upgrade Best Practices
In-House Technical Experts on Multiple Integration Vendor Technologies
Expertise in Multiple Industries
Oil & Gas
Energy &
Utility
Healthcare
Higher
Education
Hospitality &
Gaming
Logistics & Wholesale
Distribution

9
Performance Improvement of Informatica Environment

Data Integration Performance Challenges
Companies are struggling to keep up the pace with changing business scenarios like
Merger &Acquisitions or Addition of new business lines.
 Increased data volume and complexity
 Multiple sources and formats of data
 Need for real time integration
10

Data Integration Performance Challenges
 Design issues
 Increase in volume of data
 Increased complexity
 Poorly configured system resources
 Lack of Installation and setup best practices
11

Impact of Poor Performance
 Timeliness : Lead to loss of timely information and potentially a affecting a business
critical decisions.
 Flexibility : In ability to accommodate the dynamic business changes
 Compliance/SLA: Missed service level agreements(SLA) or compliances.
 Productivity: Time lost in waiting for data loads to complete
 Hardware Resource: Un-even consumption of resources
12

Addressing the Challenges
 Development Best Practices
 Informatica PowerCenter features
 Administration Best Practices
13

Development Best Practices
The development best practices for performance involves optimization guidelines
majorly involves optimizing :
 Transformations
 Session, and other workflows tasks
15

Development Best Practices – Optimizing Transformation
Optimizing Source Qualifier Transformation
 Use Optimizer hints when multiple tables are used in one Source Qualifier
 Source Qualifier would pull the data faster when there is index defined on
group by/order by/where clause columns
 For flat file sources ,if possible use fixed width file format instead of
character based delimiter as Informatica Integration services spends more
time in parsing such source file formats.
16

Optimizing Expression Transformation :
 Factor Out Common Logic
 Replacing Common Expressions with Local Variables/User defined functions
 Using Operators Instead of Functions
 Simplify nested function when possible.
i.e. instead of IIF(condition1,result1,IIF(condition2,result2,IIF(…)))))
Try
DECODE(TRUE,condition1,result1, … condition n, result n)
17

Optimizing Aggregator Transformation :
 Use it as early in the mapping as possible
 Use sorted input to decrease use of aggregate cache
 Limit the number of input or output ports
 Group by on simple columns, when possible
18

19
Aggregator
Sorted input
Option

Optimizing Filter/Router transformation :
 Consider Source Qualifier with a filter to limit rows within relational sources
 Filter as close to the source as possible
 Replace multiple filters with a router
20

Optimizing Joiner Transformation :
 Designate the master to the source with fewer duplicate key values
 Designate the master source as the source with fewer rows
 Perform joins in a database when possible
 Join on sorted column when possible ( Provide sorted data into Joiner Transformation)
21

Optimizing Lookup Transformation :
 Remove the columns which are not used in target or in lookup override
 Use Equality condition first when there are multiple conditions in lookup
 Override the default order by clause .
 Use cached lookup wisely
 Use of conditional call
 Re-use the cache if lookup table is not changing frequently
22

Optimizing Sequence Generator Transformation :
 Make the Sequence Generator transformation a re-usable transformation and use it in
multiple mappings
 Enable Number of cached values property by setting values to it.
23

Optimizing SQL Transformation :
 Avoid Script mode. If possible use Query
mode instead.
 Avoid SQL Transaction statement such as
commit and rollback (if possible)
24

Development Best Practices – Optimizing Session Task
Optimizing Session Task
 Assign enough buffer memory to session’s DTM Process
 Use right cache size for all cache related transformation
 Use incremental aggregation option if the percentage of changes is less as compared
to over all volume of source data
 Configure the database connection using native driver rather than ODBC driver
 Try configuring the session to load the target database using external loader
 Configure the data movement as bulk mode when there is no constraints on target
tables
 Increase the database commit interval if there is big volume of data movement
25

 Collect Performance data : Enable the performance
counter in development/test environment to check
any further bottlenecks.
• value under readfromdisk and writetodisk , a non
zero value indicates insufficient cache settings
• a non zero count for Errorrows indicates the
need for elimination of transformation error to
improve the performance
 Session Log : Look for warning message in session log
for insufficient memory buffer
• Integration service writes a warning message in
session log when session is not able allocate
required buffer memory to DTM process.
“Message: WARNING: Insufficient number of data blocks
for adequate performance. Increase DTM buffer size of
the session. The recommended value is xxxx.”
26
Reviewing & Identifying the bottleneck

 Thread statistics :Check for
thread statistics for reader,
transformation, writer’s
busy percentage
 Higher busy percentage of
any one of the
thread(Reader, Writer,
Transformation) as
compared to other
indicates the performance
bottleneck at that point.
27

28
Informatica Product Features for performance tuning

Informatica Features
Features
 Pushdown Optimization
 Pipeline Partition
 Dynamic Partition
 Concurrent Workflows
 Grid Deployment
29

Informatica Features –Pushdown Optimization
Pushdown Optimization
 Pushes transformation logic to the source or target database to utilize the power
of database engine
 Converts transformation logic into SQL statements
 Minimizes the movement of data between servers
 Freedom from memory or disk management for all transformation needing cache
30

Options for applying Pushdown Optimization :
 Source Side:- The Integration Service pushes as much
transformation logic as possible to the source database.
 Target Side:- The Integration Service pushes as much
transformation logic as possible to the target database.
 Full( Both source and Target):- The Integration Service
attempts to push all transformation logic to the target
database. If the Integration Service cannot push all
transformation logic to the target database, it performs both
source-side and target-side pushdown optimization
31
 Using a parameter ($$PushdownConfig):- The parameter lets you run a session using the different
types of pushdown optimization. The settings in the $$PushdownConfig parameter override the
pushdown optimization settings in the session properties.

Limitations :
 The session fails if the SELECT statement in the SQL- override refers to a database sequence
 The database might not support to all the function that is used in an expression in the
transformation.
 Expression containing variable port can’t be pushed.
 Informatica can’t push the a set of parallel lookups to database
 Unconnected lookup transformation performs slower
 The session fails if the SQL override contains an ORDER BY clause and one tries to push the
Source Qualifier transformation logic to an IBM DB2, Microsoft SQL Server, Sybase ASE, or a
Teradata database.
32

33
Converting Parallel Lookup to
Pushdown optimized mapping
 A mapping containing parallel
lookups cannot be pushed to the
database
 Design the mapping and the change
the lookups into joiners

34
Converting Un Connected Lookup to
 A mapping containing
unconnected lookups might
perform slower as unconnected
lookups are converted into outer
joins
 Compensate by changing the
lookups to joiners whenever
possible.

35
Converting Expression with variable port to
 Pushdown Optimization is not possible
for this mapping because variable ports
are not supported.
 Consider replacing the variable with
actual calculation on source ports as
shown : (NET_AMOUNT = AMOUNT –
FEE, DOLLAR_AMT = NET_AMOUNT *
RATE) with (DOLLAR_AMT = (AMOUNT –
FEE) * RATE)

Informatica Features – Comparison b/w PDO and Non PDO
Properties Non-PDO PDO
Run-time Slow Quick
Development time Less More (Mapping needs to be optimized)
No of
transformations/mapping No difference Might differ
Connection Object
Different connection objects
as per normal approach
Single connection object for
source/target/lookup for full PDO
No of look-ups No Difference
No Difference (look-ups need sequential
arrangement)
Log row errors option
available (yes/no?) Need not be set Set to report error on multiple matches
Use of Informatica
functions allowed Yes
All are allowed (with Certain minute
Restrictions)
Requires centralized
database No
Yes (If tables are present in multiple
databases)

Informatica Features – Session Partitioning
Session Partitioning
 Increases the performance of Powercenter through parallel data processing
 Effectively utilize the hardware power of multiprocessor and grid based environment setup
 Splits the large data set into smaller subsets to process them in parallel
 Uses various algorithm to split the data such as database partitioning, round robin
partitioning ,hash auto key, hash user defined, key range, pass through
38

Informatica Features –Session Partition
 Database Partitioning
 Pass-through
 Round-robin
 Hash auto-keys
 Hash user keys
 Key range
39

Partition Types – Pass Through
 Data is processed without redistributing the rows among partitions
 Serves only to add a stage to the pipeline
 Use when you want an addition thread for a complex transformation but you
don't need to redistribute the data
40

Partition Type – Key range
 The DTM passes data to each partition depending on user-specified ranges
 You may use several ports to form a compound partition key
 The DTM discards rows not falling into any specified range
 If two or more ranges overlap, a row can go down more than 1 partition resulting in
duplicate data
 Use Key range partitioning when the sources or targets in the pipeline are partitioned
by key range
41

Partition Type – Round Robin
 The Integration Service distributes rows of data evenly to all partitions
 Use when there is no need to group data among partitions
 Effective when reading flat file sources of different sizes
 Use when data has been partitioned unevenly upstream and requires significantly
more processing before arriving at the target
42

Partition Type – Hash Auto Keys
 The DTM applies a hash function to a partition key to group data among
partitions
 Use hash partitioning to ensure that groups of rows are processed in the same
partition
 The DTM automatically determines the partition key based on :
• Aggregator or rank group keys
• Join keys
• Sort keys
43

Partition Type – Hash User Keys
 This is similar to hash auto keys except the user specifies which ports make up
the partition
 Alternative to hard-coded key range partition on relational target(if DB table is
partitioned)
44

Partition Type – Database Partitioning
 Only valid for DB2 and Oracle databases in a multi-node database
 Sources : Oracle and DB2
 Targets : DB2
 The number of partitions does not have to be equal to the number of database
nodes
 Performance may be better if they are equal .
45

Rules for Adding Partition Point :
 You cannot add a partition point to a sequence generator
 You cannot add a partition point to an unconnected lookup
transformation
 You cannot add a partition point on a source definition
 If a pipeline is split and then concatenated, you cannot add a partition
on any of the transformation between the split and concatenation
46

Guidelines of Adding Partitions
 Ensure you have enough free memory in order to avoid memory allocation failures
 Don’t add partition points if the session is already source or target constrained
 Tune the source or target to eliminate the bottleneck
 Don’t add partition points if the CPUs are already maxed out (%idle < ~5%)
 Eliminate unnecessary processing and/or buy more CPUs
 Take advantage of database partitioning
47

Informatica Features – Dynamic Session Partition
Dynamic Session Partitioning :
 PowerCenter dynamically decide the
degree of parallelism based on :
• Number of source database partitions
• Number of CPUs on the node
• Number of nodes in the grid
• Based on user defined number
of partitions.
 Is useful when the data volume increases
or the number of CPU’s available changes
49

Informatica Features – Dynamic Session Partition
Rules and Guidelines for Dynamic Partitioning
 Dynamic partitioning uses the same connection for each partition.
 You cannot use dynamic partitioning with XML sources and targets.
 Sessions that use SFTP fail if you enable dynamic partitioning.
 When you set dynamic partitioning and you manually partition the session on
the Mapping tab, you invalidate the session.
50

Informatica Features
51
Concurrent Workflows

Informatica Features – Concurrent Workflow
Concurrent Workflow features
 Configure a workflow to run its multiple instances
 Multiple instances can have same instance name or can be configured to have a unique
names
52

Informatica Features – Grid Deployment
Grid Deployment
 Provides load balancing and scalability by switching the nodes in grid
 Integration Service runs a service process on each available node of the grid
 Session threads are distributed to multiple DTM process in the grid
54

Informatica Features – Grid Deployment
Grid Deployment helps to achieve
 Load Balancing : Nodes are chosen dynamically based on process size , CPU
utilization , memory requirement
 High Availability : Facilitates switchover from master to backup node in case
failure
 Dynamic Partition :Facilitates dynamic Sessions partitions where in integration
service make use of multiple nodes in the grid.
55

56
Administration Best Practices

Administration best practices for performance tuning
Administration best practices involves :
 Setting up Grid configuration
• Workflow on Grid
• Session on Grid
 Configuring High availability
 Identifying and tuning up Informatica server resources such as CPU, Disk,
Memory etc.
57

Administration best practices for performance tuning
Benefits of Informatica Grid Configuration :
 Workflows on a grid
The Integration Service distributes workflows across the nodes in a grid. It
also distributes the Session, Command, and predefined Event-Wait tasks within
workflows across the nodes in a grid.
 Sessions a grid
The Integration Service distributes session threads across nodes in a grid.
59

Administration best practices –Grid Configuration
60
Setting workflow on Grid Setting Session on Grid

Administration best practices
Optimizing Grid Deployments
 Add compatible nodes to the grid
 Increase storage capacity and bandwidth
 Use shared file systems
 Use a high-throughput network when accessing source/target over the network
61

Administration best practices –High Availability configuration
Key underlying components to achieve a Powercenter HA solution are :
 Highly available database
 Redundant Network
 Highly Available Shared file system
63
General Recommendations
 Network >1 GB
 HA CFS w/ heartbeat & failover
 Redundant network

Administration Best Practices – HA Best Practices
High availability setup best practices – Informatica components
 Configure IS AND RS to run on opposite primary nodes
 Configure gateway operation and application to run on separate nodes
 Configure the IS AND RS to run on multiple worker nodes
 Use highly available database for the repository and domain
64

65
Identifying & Tuning Server & Resources

Administration Best Practices
Tuning Informatica Server/Resources
 Improve network speed -Decrease the number of network hops
 Use Multiple CPUs – Run multiple session on multiple CPUs
 Reduce Paging – Configure physical memory for integration service
process
 Use processor binding : Distribute CPU well between different server
resources.
66

Summary
 Different challenges with data integration performance
 Causes of Performance degradation
 Impact of Performance issues
 Addressing the issues
• Developer best practices
• Informatica product Features
• Administration best practices
 Wise Men has been providing Informatica based services for last 10+
years including performance improvement assessments, architecture
engagements and Informatica development & support services
67

Copyrights 2015. Wise Men. All Rights Reserved.
Downtown Office
1010 Lamar Ste 550
Houston, TX 77002
O: +1 281-953-4500
Global Delivery Center
8-2-596/C/4 & 8-2-596/C/5
Ascent Towers, Floor No. 4 & 5
Road No: 10, Banjara Hills
Hyderabad – 500034, India
O: +91 40 4036 3344
Pune Office
Sr. No. 106/ 107
Lohia Jain Business Center
Floor No. 4
Senapati Bapat Road
Pune 411016, India
Headquarters
1500 S Dairy Ashford Ste 285
Houston, TX 77077
O: +1 281-679-6740
Thank you

Powerup performance of Informatica Environment Nov 5, 2015

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Powerup performance of Informatica Environment Nov 5, 2015

Similar to Powerup performance of Informatica Environment Nov 5, 2015 (20)

More from Wise Men

More from Wise Men (20)

Recently uploaded

Recently uploaded (20)

Powerup performance of Informatica Environment Nov 5, 2015