JDE & Peoplesoft 2 _ Sam Sampathnathan _ Best Practices for Managing Your JD Edwards Data.pdf
Best Practices for Managing JD Edwards Data:Preparing for Upgrades, Migrations and more Sambasivam Sampathnathan – World Wide Product Manager IBM Software Group 08/16/2011
Today’s Discussion• What drives upgrades today? • Data management strategies for your upgrade project • Archiving and Upgrades – Concepts & Beneﬁts – Key Requirements • Managing test data • Client Success Story • QuesEons & Answers
What Drives Upgrades Today• Maximize value from exisEng investments in current economic condiEons: Fusion apps sEll under development • Changing business processes/requirements • ApplicaEon instance consolidaEon • Technology – Out with the old, in with the new – New features; industry compliance – Obsolete hardware, middleware & database plaSorms • Vendor-‐imposed upgrade deadlines
Common Drivers of Data Growth• OrganizaEon growth • Reduce IT Cost • Data retenEon compliance • Too much of everything – Keeping data “always available” • Data mulEplier eﬀect
How are Organizations Responding to Data Growth? • Use database Performance partitioning • Use database vendor Database Size compression • Buy more Storage & CPU hardware Hardware Capacity Forrester estimates that, on average, data repositories for large applications grow by 50% annually (structured data). *Source: Forrester Research, Database Archiving Remains An Important Part Of Enterprise DBMS Strategy, Q3 2007
What is Archiving?• Segregate historical enterprise applicaEon data by age, status, event or other criteria • Copy historical records to a secure archive • Delete transacEon details from producEon system • Retain access to informaEon for query, reporEng, customer service, audit and discovery requests Production Archive Reference Data Archive Historical Historical Data Current
What are the Benefits of Archiving?• Streamline the upgrade project – Reduce downEme during conversion by 50% or more – Improve applicaEon performance from old version to new version by reducing OLTP workload • Maintain a superior ownership experience – ongoing – Reduce backup and disaster recovery Eme by hours – Lower database maintenance Eme for tasks like reorganizaEons, refreshes, clones – Support “compliance readiness” – Ensure consistent performance to Service Level objecEves – Reduce cost of storage and overall TCO
Data Archive Strategy: Questions to Consider• What data should I be saving, for how long and for what reasons? • What data should I be deleEng? • How am I going to ﬁnd the data when I need it? • What do I do with the data when I no longer need it? • What is the cost/beneﬁt analysis to support an archiving soluEon acquisiEon? • What is the most appropriate soluEon to meet my archiving needs?
Archiving Solution: 4 Key Requirements1 ApplicaEon Intelligent • JD Edwards EnterpriseOne • Complete Business Object 2 Take the right data out • Apply Func>onal Condi>on Checks • Accommodate Unique Business Requirements 3 • Support & automate data reten>on Store it where you policies as per ILM business requirements want • Mul>ple formats – DBMS, File 4 Access archived data when & how you want • Na>ve applica>on access • Applica>on independent access
1 Application Intelligent • Complete Business Object – Includes pre-‐deﬁned related J.D. Edwards tables – TransacEon data as well as contextual and reference data • Integrity Checks • Custom columns are recognized automaEcally • AddiEonal custom tables can be added visually • SelecEon criteria or integrity checks may be applied to any column: AutomaEcally applied to all related rows.
2 Archive the Right Set of Data Extract Production Database Archive Solution 1 - 2 Years Current Data • Access Definitions • Complete Business Object • Business Rules • Validations
2 Archiving a Complete Business Object • Represents application data record – payment, invoice, customer – Referentially-intact subset of data across related tables and applications; includes metadata • Provides “historical reference snapshot” of business activity • Federated object support across enterprise data stores
2 Complete Business Object - Example F0010 Company Master F0008 F0101 F0006 F0011 Business Unit Batch Control Fiscal Data Pattern Address Book Master (closed jobs) F0012 F0018 F0911 F0911T AAI’s Tax Table General Ledger GL Tag F0025 F4008 F0902 Ledger Type Master Tax Area Account Balances F0901 F0909 Account Master Chart of Accounts Records in table removed from database during archive process • Represents application data record – payment, invoice, customer Table captured as reference table during archive process – Referentially-intact subset of data across related tables and applications; includes metadata All Records included • Provides “historical reference snapshot” of business activity • Federated object support across enterprise data stores
2 Business Rules & Validations Example• Both invoice closed date and receipt GL Date must fall within the speciﬁed period • The invoice and receipt are posted and all corresponding G/L entries are posted • All the invoices must be paid and the payment status must be equal to ‘P’ • The domesEc and foreign open amounts both must be equal to zero • Recurrent invoices (doc. type RR) are only archived if the number of installments is blank
3 Store It Where you Want Compressed Active/Historical Online 2 - 4 Years Archives XML Extract Production Archive Database Database On/Near-Line Archive 4 - 6 Years Archive Solution Non DBMS1 - 2 Years Retention Platform Current Data • Access Definitions ATA File Server • Complete Business SATA / FATA disk Object • Business Rules 6+ Years Off-Line Archive Off-line Retention Platform • Validations CD,Tape,Optical, WORM HP StorageWorks™, IBM TSM NetApp NearStore® SnapLock™, IBM Information Archive/TSM EMC Centera™.
3 Support ILM Business Requirement FuncEonal Usage / Access Requirements Over Time ExcepEon-‐based, Infrequent, ApplicaEon-‐ Frequent, Ad-‐Hoc, Independent ApplicaEon-‐ Query-‐based Access Complete DeleEon Based Access (Self-‐ (24-‐hour IT (Dictates storage Business Object Access Help) response) planning) Journals (GL) Current – 2Y Years 3 – 5 Years 6 -‐ 10 Year 11 Invoices (AP) Current – 2Y Years 3 – 5 Years 6 -‐ 10 Year 11 TransacEons (AR) Current – 2Y Years 3 – 5 Years 6 -‐ 10 Year 11 Purch. Orders (PO) Current Year Year 2 Years 3 -‐ 10 Year 11 Sales Order (SO) Current – 2Y Years 3 – 5 Years 6 -‐ 10 Year 11 Fixed Assets (FA) Current Year Year 2 Years 3 -‐ 10 Year 11
4 Access Archived Data Active/Historical Online Compressed 2 - 4 Years Archives XML U N Extract I Native Application Archive Access VProduction Database E Database Restore R On/Near-Line Archive S 4 - 6 Years1-2 Archive Solution Non DBMS A IBM MashupsYears Retention Platform LCurrent Data • Access Definitions ATA File Server • Complete Business EMC Centera™, DR550, Etc. A Object C • Business Rules C 6+ Years Off-Line Archive Off-line Retention Platform • Validations CD,Tape,Optical, WORM E Additional Options HP StorageWorks™, IBM TSM S ODBC / JDBC NetApp NearStore® SnapLock™, S XML IBM Total Storage® solutions SQL (including the DR550) EMC Centera™. Excel / Access
4 Access Archived Data Thru Reporting Tools Reporting Solution Options • Report Writers: • Cognos • Business Objects • Discoverer • Mashup Center Archive • Access • Any SQL-based tool set • Browser • Java Application • Open & independent • No training of end users or audit staff • Leverage existing tools and skills • OLTP not required
Where Does Archiving Fit?1. Archive before an Upgrade – Reduces amount of data to convert during producEon cutover – Reduces downEme during upgrade 2. Archive in Parallel with an Upgrade – Combines common technical and funcEonal tasks to save overall elapsed Eme • Technical Tasks: Validate individual steps (such as moving query tables forward), validaEng new technical environment, performance tesEng • FuncEonal/Business Tasks: Reconciling “before” & “aner” results, regression tesEng on key business processes (such as payroll processing or ﬁnancial close)
1. Archive Before Upgrade Older Version Upgrade Project Upgraded Version Application Archive Data Current Data Current Data Application Archives Application Archives Universal Access to Application Data Application ODBC / JDBC XML Report Writer
2. Archive & Upgrade Projects in Parallel Older Version Upgraded Version Upgrade Project Archive Application Data Archive Planning Current Data Application Application Archives Universal Access to Application Data Application ODBC / JDBC XML Report Writer
Data Multiplier Effect Dev Test Back-up 1 TB UA Disaster 1 TB 1 TB Recovery ERP/CRM 1 TB ApplicationDev Back-up Dev Test Back- Test up UA UA Master Data Data Disaster Disaster Warehouses Recovery Recovery The Actual Data Burden = Size of all production database + all replicated clones According to Forrester, on average, data repositories for large applications grows by 50% annually (structured data)!
Creation of Test Environment: Current Practices#1 - Clone Production #2 – Write SQL Clone Production Request for Copy Write SQL • Complex Extract • Subject to Change Wait After Changes Production Production After Extract Database Database Copy Copy Changes • RI Accuracy? Expensive, • Right Data? Dedicated Staff, Manual examination: Ongoing Right data? Responsibility What Changed? Correct results? Share test database Unintended Result? with everyone else Someone else modify?
Effective Test Data ManagementProduction or Production Clone Relational subset Extract Load / Mask Insert / Update 2TB Compare 25 GB 25 GB Development• Create targeted, right-sized test environments Unit Test• Substitute sensitive data with fictionalized yet contextually accurate data• Easily refresh, reset and maintain test environments 50 GB 100 GB• Compare data to pinpoint and resolve application defects faster Training Integration• Accelerate release schedules Test
Testing Best Practices - OracleTip #27—Test with a Representative Subset of Production Data“When performing the development upgrade, it is important toleverage a representative subset of production data insteadof an exact copy; this is because the development environmentusually has less capacity in both memory and hard drive spacethan the test and production environments. Limiting the size ofthe conversion files during the development upgrade will betterensure that the processes will complete in a timely manner.”
Data in Test & Development Environments• 62% of companies surveyed use actual customer data instead of disguised data to test applicaEons during the development process – 50% of respondents have no way of knowing if the data used in tesEng had been compromised. • 52% of respondents outsourced applicaEon tesEng – 49% shared live data Source: The Ponemon Institute. The Insecurity of Test Data: The Unseen Crisis
Mask Data in Non-Production Environments• Also known as: data de-‐idenEﬁcaEon, depersonalizaEon, desensiEzaEon, obfuscaEon, data scrubbing • Technology that helps conceal real data • Scrambles data to create new, legible data • Retains the datas properEes, such as its width, type and format • Common data masking algorithms include random, substring, concatenaEon, date aging • Used in non-‐producEon environments as a Best PracEce to protect sensiEve data
De-identify Data Without Impacting Dev & Test• Mask or de-‐idenEfy sensiEve data elements that could be used to idenEfy an individual • Ensure masked data is contextually appropriate to the data it replaced, so as not to impede tesEng • Data is realisEc but ﬁcEonal • Masked data is within permissible range of values • Support referenEal integrity of the masked data elements to prevent errors in tesEng Personal identifiable information is masked with realistic but fictional data for testing & development purposes. JASON MICHAELS ROBERT SMITH
Why Retire or Consolidate Applications • Redundant systems acquired via mergers and acquisiEons • Line of business divested • Legacy technologies not compaEble with current IT direcEon • Required technical skills or applicaEon knowledge no longer available • Budget pressures – do more with less!
Retire Unused Legacy Applications• Consolidate mulEple applicaEons into a single instance and reEre unused applicaEons – Move from home grown to packaged system – Consolidate similar systems due to mergers and acquisiEons • Consolidate an independent business process with others – Move automaEon capabiliEes into a single system and reEre independent applicaEon • Move applicaEon from an old to new architecture – Not all data is relevant for the move, but it must be retained • Shut down legacy system without a replacement In almost ALL cases, access to legacy data MUST be retained while the applica<on and database are eliminated
Retire the Application, Not the DataArchiving allows you to move the data needed whilemaintaining access to the original data in its business-object form without the original application. Beneﬁts – Reduce IT infrastructure costs (hardware, sonware, labor costs) – Reduce infrastructure complexity (eliminate confusion) – Reclaim assets
About IBM Optim• Proven leader in Integrated Data Management (IDM): – Manage and Control Data Growth – Data RetenEon, Compliance & Discovery – Speed ApplicaEon Delivery & Quality with Test Data Management – Speed ApplicaEon Upgrades & MigraEons – ApplicaEon ReErement – Improve Storage Management – ILM – Improve ApplicaEon Performance and SLAs • Solving complex data management issues since 1989 • Global company: 2500 clients; 50% of Fortune 500 • Recognized by Gartner, IDC, META as EDM industry leader with 46% market share.
IBM Optim™ Solutions • Op>m Data Growth Solu>on (Archiving) – Improve performance – Control data growth, save storage – Support retenEon compliance – Enable applicaEon reErement – Streamline upgrades • Op>m Test Data Management Solu>on – Create targeted, right sized test environments – Improve applicaEon quality – Speed iteraEve tesEng processes • Op>m Data Privacy Solu>on – Mask conﬁdenEal data – Comply with privacy policies
IBM Optim: Enterprise Architecture Discovery Test Data Management Data Privacy Data Growth Application Retirement Organization environments are diverse, yet interrelated; therefore what you use to manage the data MUST support across your environment
Success Story: Application Upgrade & Cost Challenge Business Benefits § The upgrade and data conversion was§ Leveraging their Oracle packaged- completed over a 3-day weekend, eliminating application for manufacturing, shipping and the need for downtime during business hours. finance. Since installing in 2003, the system § Archived data is still accessible to functional has grown tremendously, from 300 GB to users via an archive database, so there was more than 1 TB of information. minimal training needed.§ Calculated it would take 5 to 7 business § Archived data was stored on “tier 2” storage. days to perform the upgrade and data This provided an estimated annual savings of conversion. That amount of downtime for a $75,000-$80,000 in storage costs. business critical application would be § Archiving project was completed 3 weeks unacceptable. ahead of schedule, allowing for additional preparation time for the upgrade project itself. “Without IBM Optim software, it would have taken us nearly twice as long to upgrade our ERP Solution application, forcing us to shut down manufacturing. Instead, we completed the • IBM InfoSphere Optim Data Growth Solution process in one three-day weekend without any business interruption.”
Other ExamplesObjective: Objective: Objective: Objective:Minimize business Need to reduce Need to reduce the Minimize businessdisruption during JD projected system database size of enterprise disruption during JDEdwards upgrade and downtime for JD system, and manage the Edwards upgrade andimprove system Edwards upgrade from 8 ongoing data growth for control 25% annual dataperformance to support days to 3 days to avoid financial and supply chain growth & correspondingbusiness operations business disruption information storage costsSolution: Solution: Solution: Solution:Optim Data Growth Optim Data Growth Optim Data Growth Optim Data GrowthSolution for JD Edwards Solution for JD Edwards Solution for JD Edwards Solution for JD EdwardsEnterpriseOne EnterpriseOne EnterpriseOne EnterpriseOneReduced the volume of Completed upgrade and Reduced the database size Attained a 50% reduction inaudit trail & workflow data conversion over a 3- & implemented data growth their needed futureactivity by 63%, day weekend, eliminating strategy, creating a 50% purchases of data storage,shortening data downtime during business reduction in system streamlined the dataconversion window and hours; stored archive data maintenance and 300% migration process duringimproving system on “tier 2” storage, saving improvement in query times the upgrade project, andperformance. $80K annually in storage reduced the time needed costs for ongoing maintenance
What’s the Take-Away?• Deﬁne your InformaEon Lifecycle Management (ILM) requirements – What needs to be archived – How long you need to hold on to them • “We’re an IT shop delivering business value” • Today’s IT strategies should provide an immediate payback AND set the stage for future innovaEon – Understand your data – Manage data growth via archiving – Use subsets for test data creaEon – Mask data in non-‐producEon environments – ReEre/consolidate legacy/unused applicaEons
In Closing…..• You are most welcome to join me at the IBM Booth aner this session to discuss this presentaEon or your speciﬁc quesEons or requirements • We’d appreciate if you can complete the evaluaEon form on your seat and deposit in the box at the IBM Booth …. you’ll also have a chance to win one of 3 iPads! • All IBM Summit presentaEon sessions are noted in the ﬂyer on your seat to help plan your parEcipaEon … we’d love to see you at some more of our sessions! • Break Free at our next IBM event … see the invitaEon envelope on your seat for details.
Disclaimers IBM customers are responsible for ensuring their own compliance with legal requirements. It is the customers sole responsibility to obtain advice of competent legal counsel as to the iden>ﬁca>on and interpreta>on of any relevant laws and regulatory requirements that may aﬀect the customers business and any ac>ons the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. The informa>on contained in this documenta>on is provided for informa>onal purposes only. While eﬀorts were made to verify the completeness and accuracy of the informa>on provided, it is provided “as is” without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this documenta>on or any other documenta>on. Nothing contained in this documenta>on is intended to, nor shall have the eﬀect of, crea>ng any warran>es or representa>ons from IBM (or its suppliers or licensors), or altering the terms and condi>ons of the applicable license agreement governing the use of IBM so`ware.