Although there is disagreement about the exact definition of data governance, the consequences of ineffective data governance are well known: lack of control on one of your organizations most critical assets - its data – which ultimately leads to increased risk, cost inefficiencies, regulatory noncompliance, and potentially costly data breaches. An example of such is the data breach that occurred at Her Majesty's Revenue and Customs (HMRC) agency in the UK last October 2007. Two computer discs owned by HRMC containing data relating to child benefits went missing. The two discs contained the personal details of all families in the United Kingdom claiming child benefits, thought to be approximately 25 million people (nearly half of the country's population). The discs were sent by junior staff as unrecorded internal mail. After not receiving the disks at the destination - and then not being able to find them after an extensive search - they announced its loss to the public on November 20 th 2007 due to that countries disclosure laws for lost data. The personal data on the missing discs included names, addresses, and dates of birth of children, together with the National Insurance numbers and bank details of many of their parents. Unfortunately, the HMRC breach is just one of many such occurrences that have occurred all over the world. In fact, it is estimated that over 245 million customer and employee records have been leaked since 2005 in the US alone. [www.privacyrights.org] This situation is only one example, but clearly highlights the following: 1) how easily such a disaster can occur even unintentionally 2) the stark consequences of such mistakes and 3) the importance of effective data governance. References : http://www.guardian.co.uk/politics/2007/nov/20/economy.personalfinancenews http://en.wikipedia.org/wiki/2007_UK_Child_Benefit_data_scandal
Rational Data Architect is more than a data modeling tool. It is also a: -documentation tool. It helps you to create diagrams of existing database structures -Information Integration tool. Helps to define federation concepts -XML mapping tool. Map database schemas to SOA structures -Code Development tool. Create valid DB2 SQL code. IBM Data Studio is the product that does all this outside of RDA. -Traceability tool. Know why, what and when for every change. New release features integrations with IBM Rational Software Architect, Eclipse 3.2 and IBM Information Server; additional mappings and expanded support for XML, DB2 V9, Sybase, Informix and mySQL.
We begin with a production system or clone of production <Click> Optim extracts the desired data records, based on user specifications, and safely copies them to a compressed file <Click> IT loads the file into the target Development, Test or QA environment. <Click> After running tests, IT can compare the results against the baseline data to validate results and identify any errors. They can refresh the database simply by re-inserting the extract file, thereby ensuring consistency. Manage test data across enterprise : all related applications, databases, and platforms Extract referentially intact subsets of data (including complete business objects) (ex. payments, vouchers, paychecks) Dynamically create destination environment; insert or load data to target De-identify or mask data in non-production environments Edit test data to create error and boundary conditions Compare baseline data against successive test run results to identify errors that would have otherwise gone undetected
Here is an example of model-driven governance based on a fictitious example. An e-retailer, JK Country wants to expand its channel to other online sellers. Critical to the development process is compliance to PCI standards, so real account numbers and credit cards cannot be used in development and testing. In the data model, our data architect now has the ability to define privacy rules consistently for the enterprise. He can define once data definition for credit card numbers, account numbers, contact information, etc. and the rules for masking them, These can then be reused as models and model changes are defined. Now our data architect models additions and changes to the business objects reflecting the new partner channel. Then associates the changes with the appropriate privacy models, generates the physical data model and database objects. When ready to test the application, the architect simply publishes all the definitions for test data creation to the Optim Test Data Management and Privacy Solution including the business object definitions (the physical model with both the explicit and implicit relationships), the sensitive fields and their data masking rules, and subset definitions to constrain the test dataset sizes. Now the tester can use test data that directly reflects production characteristics , but safeguards customer privacy and complies with PCI standards.
Our Guardium solution addresses the full life cycle of database security, risk management and compliance.
Deploy – Install, configure, change, and promote applications, databases and services into production Overall, the change management process has been prone to error, tedious and slow. For example, DBAs must often spend hours analyzing objects and dependencies before deciding on a course of action. And after all of the ramifications of the changes are known, several more manual steps are needed to preserve database objects such as tables, data, application bindings and permissions. Optim Change Manager akes it fast and easy for DBAs to model the target database, compare two sets of objects to see where they differ, identify dependent objects, migrate a set of objects to the target, or redefine the target objects to be like the source. Changes automatically roll through all related objects. Over time, the DBA's ability to control database performance has eroded, or at least become much harder, as additional layers emerge in the application stack. SQL is generated by frameworks not programmers, database connections are managed by systems administrators not DBAs, and dynamic SQL complicates security management. DBAs like the added control they can gain from using static SQL, and now it is possible to gain that control easily over existing Java applications by using our new client optimization technology delivered in Data Studio Developer and Data Studio pureQuery Runtime 1.2. This is a new approach to performance optimization that focuses on how to optimize database access from the database client rather than only looking within the database engine. Client optimization captures SQL from Java applications and enables administrators to bind the SQL to DB2 for static execution without changing a single line of application code. All of the gain of static SQL - making response time stable, reducing security risks, increasing throughput – and none of the pain. Future enhancements include plans to give DBAs control over performance knobs in the application server and to make client configuration manageable – finally.
Differentiated features: We can customize changes and migrate in a single change We have the most flexible data preservation options We can incorporate IDA physical data models
4-TuneSQL10 The same as that on the picture
Integrated Data Management: Product Strategy and Vision.ppt
Integrated Data Management Product Strategy and Vision Curt Cotner IBM Fellow Vice President and CTO for IBM Database Servers [email_address]
What do Businesses Have? A Collection of Disparate, Single-Purpose Products CA ERwin IBM InfoSphere Data Architect Embarcadero ER/Studio Sybase PowerDesigner Design IBM DB2 tools BMC Patrol Quest Central Oracle Diagnostic Pack Operate Oracle Tuning Pack Solix EDMS IBM Optim Data Growth Solution Optimize Quest Spotlight Quest TOAD IBM Data Studio Developer Oracle JDeveloper Develop Embarcadero Rapid SQL IBM Comparison Tool for DB2 z/OS Embarcadero Change Manager Data Studio Administrator Deploy Oracle Change Management Pack Quest InTrust Guardium IBM Optim Govern Oracle Vault
The gaps create risk … <ul><li>Loss of customers </li></ul><ul><ul><li>Average customer churn rate up 2.5% after a breach </li></ul></ul><ul><li>Loss of revenue </li></ul><ul><ul><li>$197 USD per customer record leaked </li></ul></ul><ul><ul><li>Average cost was ~ $6.3 million / breach in this study </li></ul></ul><ul><ul><li>Average cost for financial services organizations was 17% higher than average </li></ul></ul><ul><li>Fines, penalties or inability to conduct business based on non-compliance </li></ul><ul><ul><li>PCI </li></ul></ul><ul><ul><li>Sarbanes-Oxley (SOX) </li></ul></ul><ul><ul><li>HIPAA </li></ul></ul><ul><ul><li>Data Breach Disclosure Laws </li></ul></ul><ul><ul><li>Gramm-Leach-Bliley Act </li></ul></ul><ul><ul><li>Basel II </li></ul></ul>Source: “2007 Annual Study: Cost of a Data Breach” , The Ponemon Institute
Driven by the increasing numbers of physical systems, system management has become the main component of IT costs and is growing rapidly Many Servers, Much Capacity, Low Utilization = $140B unutilized server assets
What do Businesses Need? An integrated environment to span today’s flexible roles <ul><li>Manage data throughout its lifecycle </li></ul><ul><ul><li>From design to sunset </li></ul></ul><ul><li>Manage data across complex IT environments </li></ul><ul><ul><li>Multiple interrelated databases, applications and platforms </li></ul></ul><ul><li>Facilitate cross-functional collaboration </li></ul><ul><ul><li>Within IT </li></ul></ul><ul><ul><li>Among Line of Business, Compliance functions </li></ul></ul><ul><ul><li>Across disparate skill sets </li></ul></ul><ul><li>Optimize business value </li></ul><ul><ul><li>Respond quickly to emerging opportunities </li></ul></ul><ul><ul><li>Improve quality of service </li></ul></ul><ul><ul><li>Reduce cost of ownership </li></ul></ul><ul><ul><li>Mitigate risk </li></ul></ul>
Integrated Data Management – What’s Different? AppDev <ul><li>Produce enterprise-ready applications faster </li></ul><ul><ul><li>Improve data access, speed iterative testing and empower collaboration across the lifecycle </li></ul></ul><ul><li>Consistently achieve service level targets </li></ul><ul><ul><li>Automate and simplify operations with contextual intelligence across the solution stack </li></ul></ul><ul><li>Support business growth </li></ul><ul><ul><li>Accommodate new initiatives without expanding infrastructure </li></ul></ul><ul><ul><li>Simplify application upgrades, consolidation and retirement </li></ul></ul><ul><li>Facilitate alignment, consistency and governance </li></ul><ul><ul><li>Upfront business policies and standards; share, extend, and apply throughout the lifecycle </li></ul></ul>Data Architect DBAs Application Manager Testers
The broadest range of capabilities for managing the value of your data throughout its lifetime InfoSphere Data Architect InfoSphere Discovery Optim Development Studio Optim Test Data Management Optim Data Growth Solutions Optim Data Privacy Solutions DB2 Performance Expert and Extended Insight Feature Optim pureQuery Runtime Guardium DB2 Audit Management Expert Database Encryption Expert Optim Database Administrator DB2 Optim Query Tuner (a.k.a. Optimization Expert) Develop Design Deploy Optimize Operate Govern Policies Models Metadata
InfoSphere Data Architect <ul><li>InfoSphere Data Architect is a collaborative, data design solution to discover, model, relate, and standardize diverse data assets. </li></ul><ul><li>Key Features </li></ul><ul><li>Create logical and physical data models </li></ul><ul><li>Discover, explore, and visualize the structure of data sources </li></ul><ul><li>Discover or identify relationships between disparate data sources </li></ul><ul><li>Compare and synchronize the structure of two data sources </li></ul><ul><li>Analyze and enforce compliance to enterprise standards </li></ul><ul><li>Support across heterogeneous databases </li></ul><ul><li>Integration with the Rational Software Delivery Platform, Optim, IBM Information Server, and IBM Industry Models </li></ul>
Automate Data Design via Model-driven Transformation Rational Software Architect InfoSphere Data Architect Optim Database Administrator WebSphere Business Modeler XSD INTEGRATION DEVELOPER DATA ARCHITECT SOLUTION ARCHITECT <ul><li>Built-in transformation </li></ul><ul><li>Compare and sync facilitates merge </li></ul>DATABASE ADMINISTRATOR UML PDM
Optim Test Data Management Solution <ul><li>Accelerate time to market </li></ul><ul><ul><li>Create “right sized” test databases </li></ul></ul><ul><ul><li>Extract referentially intact subsets </li></ul></ul><ul><ul><li>Compare baseline data against test results to pinpoint and resolve application defects faster </li></ul></ul><ul><ul><li>Edit test data to create error and boundary conditions </li></ul></ul><ul><ul><li>Easily refresh, reset and maintain test environments </li></ul></ul><ul><li>Cut storage costs </li></ul><ul><ul><li>Reduce storage requirements by using smaller subsets for testing </li></ul></ul><ul><li>Enable compliance </li></ul><ul><ul><li>De-identify or mask data </li></ul></ul>Production or Clone Dev QA Test Extract Optim Streamline building test databases, improve application quality, cut IT costs and accelerate solution delivery
Enterprise Challenge: Data Privacy Optim Data Privacy Solution <ul><li>Application-aware masking capabilities ensure data is realistic but fictional </li></ul><ul><li>Prepackaged data masking routines make it easy to de-identify elements </li></ul><ul><ul><li>E.g. credit card numbers & email addresses </li></ul></ul><ul><li>Lookup values </li></ul><ul><li>Intelligence </li></ul><ul><li>Arithmetic expressions </li></ul><ul><li>Concatenated expressions </li></ul><ul><li>Date aging </li></ul><ul><li>String literal values </li></ul><ul><li>Character substrings </li></ul><ul><li>Random or sequential numbers </li></ul>A comprehensive set of data masking techniques to transform or de-identify data, including: Data is masked with realistic but fictional information
Aligning Around Data Privacy <ul><li>Define policies once and reuse </li></ul><ul><li>Flow definitions to team members </li></ul><ul><li>Drive consistent practices around privacy </li></ul>Data Architect Tester Developer Define privacy policies Analyse use of sensitive data Provision fictionalized test data
Optim Data Growth Solution <ul><li>Complete business object provides historical reference snapshot of business activity </li></ul><ul><li>Storage device independence enables storage cost optimization </li></ul><ul><li>Immutable file format enables data retention compliance </li></ul>Active Historical Retrieve Archives Reporting Data Historical Data Reference Data Archive Freed Facilitates application growth, application upgrades, application retirement, and retention management. Application Application XML ODBC / JDBC Retrieved Universal Access to Application Data
Guardium -- Non-invasive, real-time monitoring SQL Server <ul><li>Non-invasive architecture </li></ul><ul><ul><li>Outside database </li></ul></ul><ul><ul><li>Minimal performance impact </li></ul></ul><ul><ul><li>No DBMS or application changes </li></ul></ul><ul><li>Cross-DBMS solution </li></ul><ul><li>100% visibility including local DBA access </li></ul><ul><li>Enforces separation of duties </li></ul><ul><li>Does not rely on DBMS-resident logs that can be erased by attackers, rogue insiders </li></ul><ul><li>Granular, real-time policies & auditing </li></ul><ul><ul><li>Who, what, when, how </li></ul></ul><ul><li>Automated compliance reporting, sign-offs & escalations (SOX, PCI, NIST, etc.) </li></ul>DB2
Managing the entire lifecycle of security, risk, compliance
Deploy without Disruption Optim Database Administrator Deploy Install, configure, change, promote Database Administrator <ul><li>Enhance DBA productivity and accelerate complex changes while ensuring data and process integrity </li></ul><ul><ul><li>Automatically manages dependent objects </li></ul></ul><ul><ul><li>Saves and restores data for extended alters </li></ul></ul><ul><ul><li>Generates needed maintenance commands </li></ul></ul><ul><li>Reduce errors and downtime </li></ul><ul><ul><li>Provides impact analysis visualization </li></ul></ul><ul><ul><li>Factors in impacts and side effects automatically </li></ul></ul><ul><ul><li>Automatically generates commands to undo changes </li></ul></ul><ul><li>Foster teamwork and enhance auditability </li></ul><ul><ul><li>Integrated into Rational Software Delivery Platform </li></ul></ul><ul><ul><li>Document changes for collaboration and audit </li></ul></ul>Develop Design Deploy Optimize Operate Govern Models Policies Metadata
Optim Database Administrator <ul><li>Models, automates and deploys complex schema changes </li></ul><ul><li>Identifies dependencies and analyzes impact to mitigate deployment risk </li></ul><ul><li>Preserves data, dependent objects, privileges, and application binding </li></ul><ul><li>Synchronizes, copies, clones, or merges database schema definitions from the source to the target </li></ul><ul><li>Documents changes for collaboration and audit </li></ul><ul><li>Enables undo or restart -- if needed </li></ul><ul><li>Manages common database maintenance tasks </li></ul>Improves DBA productivity and reduces application outages by automating and simplifying complex DB2 structural changes including change-in-place as well as database migration scenarios.
Data & Object Movement <ul><li>Value Proposition – </li></ul><ul><ul><li>Copying database objects and data between homogeneous and heterogeneous databases within Optim Development Studio and Optim Database Administrator </li></ul></ul><ul><li>Key Features </li></ul><ul><ul><li>Copy objects at various levels – complete databases to a fixed number of rows from a single table </li></ul></ul><ul><ul><li>Action performed in Data Source Explorer – Copy/Paste and Drag/Drop </li></ul></ul><ul><ul><li>Can automatically copy rows from related tables using: </li></ul></ul><ul><ul><ul><li>RI in database </li></ul></ul></ul><ul><ul><ul><li>Data Architect model </li></ul></ul></ul><ul><ul><ul><li>Optim application models </li></ul></ul></ul><ul><ul><ul><li>Optim Discovery </li></ul></ul></ul><ul><ul><li>Can optionally annonymize the rows using Optim Test Database Manager </li></ul></ul>
Optim Performance Manager - Overview dashboard Enlarge ( Shows average and max values in the selected timeframe ) Open details ( shows timeseries in the selected timeframe ) <ul><li>Focus on specific database to see key performance indicators </li></ul>
Optim Performance Manager - Active SQL dashboard Show top running statements at a point in time according to selected metric, e.g. Elapsed time. “ Tune” launches Query Tuner “ Identify Workloads” refers to defined workloads in Extended Insight
Optim Performance Manager – Reporting <ul><li>Reporting </li></ul><ul><li>... gives you the flexibility you need to adapt OPM on your own environment </li></ul><ul><li>coming with sample reports for </li></ul><ul><li>- disk space usage </li></ul><ul><li>- system configuration </li></ul><ul><li>- top resource using SQL statements </li></ul><ul><li>- DB connections </li></ul><ul><li>easy to modify and adapt to your environment because report definition stored in XML document </li></ul><ul><li>can be saved and shared as PDF with your colleagues </li></ul><ul><li>a graphical report builder will also follow soon </li></ul>
Optim Query Tuner and Option Workload Tuner for z/OS
Optim Query Tuner Key Features at a Glance Access Path Graph Visual Plan Hint Workload Control Center Query Report Index Advisor Query Advisor Workload Statistics Advisor Workload Index Advisor Workload Query Advisor Query Tuner User Interface Eclipse Statistics Advisor Access Path Advisor Formatter Annotation Query Based Tools and Advisors Workload Advisors Query Advisors Query Tools Database Capture Query Environ. Profile Monitor Workload Report Capture Workload Environ
IBM Optim Query Tuner key functions Yes Workload Query Reports Yes Yes Query Reports Yes Profile Based Monitor Yes Workload Environment Capture Yes Workload Index Advisor Yes Workload Statistics Advisor Yes Workload Query Advisor Yes Yes Query Environment Capture Yes Yes Index Advisor Yes Yes Statistics Advisor Yes Yes Access Path Advisor Yes Yes Query Advisor Yes Yes Visual Plan Hint Yes Yes Access Plan Graph Yes Yes Query Annotation Yes Yes Query Formatter Query Workload Tuner for DB2 for z/OS Query Tuner (single Query) Functions
Understanding Query with Query Annotation Original and transformed Query Formatted, reorganized query text Annotations (catalog stats, cost estimation)
Index Advisor Index Recommendations DDL to create the new index statement
Database Support by Product (as of today) Vormetrics Vormetrics Vormetrics EE for DB2 and IMS Database Encryption Expert OPM Extended Edition Omegamon Optim Performance Manager Optim Data Growth Optim Data Privacy Optim Test Data Management DB2 Admin Tool/Object Compare Optim Database Administrator + Workload Optim Query Tuner Optim pureQuery Runtime Optim Development Studio InfoSphere Data Architect OpenAdmin Tool Data Studio Administration Console alphaworks derivative Data Studio MySQL Sybase SQL Server Oracle IDS DB2 for i DB2 for LUW DB2 for z/OS
<ul><li>IBM Data Studio </li></ul><ul><ul><li>www.ibm.com/software/data/studio </li></ul></ul><ul><ul><ul><li>FAQs / Tutorials </li></ul></ul></ul><ul><ul><ul><li>Downloads </li></ul></ul></ul><ul><ul><ul><li>Forum / Blogs </li></ul></ul></ul><ul><ul><ul><li>Join the IBM Data Studio user community </li></ul></ul></ul><ul><li>Data Studio Book </li></ul><ul><ul><li>http:// bit.ly/dstudiobook </li></ul></ul>NEW!!!
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.