Your SlideShare is downloading. ×
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Run Your Oracle BI QA Cycles More Effectively
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Run Your Oracle BI QA Cycles More Effectively

1,374

Published on

How does one QA an OBI system? Many project teams struggle to plan out the steps and types of tests they will need to efficiently drive an efficient QA cycle. Learn about the different facets of your …

How does one QA an OBI system? Many project teams struggle to plan out the steps and types of tests they will need to efficiently drive an efficient QA cycle. Learn about the different facets of your BI system and how to properly QA each layer. Special attention will be paid to Data testing and OBI Ad-hoc testing.

Speaker: Jeff McQuigg, Solutions Architect, KPI Partners

Delivered at BIWA Summit 2013

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,374
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
36
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Run Your OBI QA Cycles More Effectively BIWA Summit Jan 10, 2013 Jeff McQuigg Sr. Architect Start Here KPI Partners, Inc.© KPI Partners Inc. Contact Us 510.818.9480 | www.kpipartners.com
  • 2. Senior Architect at KPI Partners10 years OBIEE consulting experience, 20+ years overallPersonally been involved with over 45+ OBI projects in every capacity (BIArchitect, Data Modeling, RPD Metadata, Business Analyst, ReportDeveloper, ETL Architect/Developer, Project Manager, Pre-Sales)Oracle Ace thought leader for BI & OBI: • Blogging on OBI best practices since 2007 at GreatOBI.WordPress.com • Frequent Oracle Open World speakerPersonal: My 3,000+ beer bottles of beer are on display at Brewpalace.com 2
  • 3. www.kpipartners.com Transform Data Into Insight Strategic Consulting | Systems Implementation | Training Staff built from Oracle/Siebel/Hyperion engineering teams On-site, off-shore and blended shore delivery models Exclusive pre-built solutions for Oracle BI Oracle BI & E-Business Suite Hyperion Depot Repair Analytics Student Info Analytics Fixed Asset Analytics Subledger (SLA) Analytics Endeca Manufacturing Analytics and more Exalytics Salesforce.com AnalyticsThe Leader In Oracle BI & EPM 3
  • 4. 1. Why QA plans are always different2. The importance of Builds3. What resources will you need4. The benefits of automation5. How to test OBI Meta Data Goal: Build better QA plans! 4
  • 5. Break the problem down to UI Alertsits layers ReportsEach layer can be testedindependently for the most Security Modelpart• Some assumptions needed OBI Model (Ad-hoc)End-to-End and End-to-Mid Loadstesting ensures properhandoff between layers Extracts 5
  • 6. Test the OBI stack for the following:1. Data Validation – Is the data accurate?2. User Functionality – Does the UI work properly?3. Security – Are the appropriate visibility rules applied?4. Performance – Do the loads and reports run fast enough?5. Infrastructure – Is the infrastructure reliable and robust? 6
  • 7. 1. Resources always differ • Consultants, IT BI &/or DW team, source system SMEs, internal QA teams, power users2. Project execution always differs • Agile or iterative vs. traditional waterfall • Document robustness • QA team members’ involvement during project • Different corporate toll gates/methodology3. Legacy reports / Re-platforming • May or may not have something to compare to “QA Plans are like a box of chocolates” - Forrest McQuigg4. QA source application environments differ • Dedicated vs. shared, controlled vs. no control5. Technical stack differences (less so on BI Apps) • E.g.: Real-time layer, ODS, DW, DM, UI and security integrations, large user volumes No two QA Plans are ever the same! 7
  • 8. A BI system is different than compilingcode modules for a new buildIt’s about the Data Transformation:Applying code to data to makedifferent dataETL loads drive the QA plan as theytake time (12-48 hours typically)ETL has a high degree of integration The ETL Machine • Facts depend on Dimensions • Extract Load Post Load Process AggregateSeveral iterations of ETL Builds are needed to get it right • Full load for base logic • Incremental loads for subsequent loadsIn-line ETL testing during Development can lower QA risk, but Builds are still needed 8
  • 9. Putting the Build cycles together yields a staggered plan Focus on the Full Load first then on Incremental LoadsStaggering can help if two environments QA Begins More than 2 Full loads may be needed 9
  • 10. Consider two levels of QA test cases:1. Code-to-Spec • Identify transformational logic and develop test cases based on the design spec2. Spec-to-Business Objective • Ensure the spec was written correctly. Examples: All widgets should be assigned to a customer If an owner doesn’t exist in System A, then it should use the record from system B • Perhaps 2-5 goals per spec • More applicable for custom coded solutions and less so for OOB BI Apps extensions 10
  • 11. QA Script Development should occuralongside ETL development using the samespec • Keep QA and Dev resources separated Object Code Dev Design QA Spec Execution Object QA Script Dev 11
  • 12. Pipeline as much as your team and environment can handle • Offshore capability helps tremendously • Weekend builds are important Test Fix • Baby sitting loads takes time & effort + + Solve BuildMultiple DW environments are a must • QA Pass #1 in Environment #1 while Load #2 ongoing in Environment #2 • Flexibility is key; use Pre-Production server if available • Greater complexity when SIT and UAT are used in parallel • Post Release 1 Deployments are even more complicatedPlan on at least 2 QA iterations for Full Load and at least 3 or more forIncremental LoadKeep in mind any special loads like weekly or monthly jobsOBI RPD and Reports can be layered easily on any of these environments 12
  • 13. Now is not the time to be optimistic!Add sufficient buffers in the schedule • Problems are 100% guaranteedDependencies: A bug in a dimension mayrequire a full reload to retestQA Cycles typically can run for severalmonths on a complex systemPlan on your source system supportapproach for Incremental Loads early • Multiple Prod snapshots to manage – or – • Manual creation of test cases in a QA system 13
  • 14. There are a variety of tests to run for ETL • Table row counts • Allocations & Summations (Total $ by month) • Attribute ranges (Min and Max values) • Specific Transformation logic (If-Then-Else) • Slowly Changing Dimensions, Snapshot Facts • Metadata columns (minimal on BI Apps) • Aggregates sync with base tables • 10 Random Records tracing • Engineered Records (great for incrementals and special test cases) 14
  • 15. Use SQL script files to automate Status_CD Count(*) Select ApplLogic(Status_CD), Count(*) Open 4,129 from Source.Table Closed 65,536 Group by Status_CD order by Status_CD Rejected 80,085 Minus UnSpecified 1,024 Select Status, Count(*) MINUS from Target.Table Status Count(*) Group by Status order by Status; Open 4,129 Closed 65,536For Extracts, consider external file comparison tool Rejected 80,085Source system technical SMEs write source scripts UnSpecified 1,024 • Challenging to replicate business transformation logic – get your top expertsCompare source SQL with Metric totals in OBI Answers (Anchor Metrics)Perform both single-hop (Extracts vs. Loads) and multi-hop tests (Source vs.DW Target)Leverage database constraints (NOT NULL, FK) during DEV and QA to assist 15
  • 16. Ad-hoc testing of the OBI Model is skipped toofrequentlyEven if reports are accurate, what about ad- Reportshoc queries? OBI ModelReports are built on top of the ad-hoc subjectarea DatabaseOBI thinks and generates SQL – does it do socorrectly? ConfidenceThere are tradeoffs between QA effort vs.confidence • 100% confidence is not possible # Tests 16
  • 17. Ensure proper SQL generation and consistent resultsCan be done on a buggy DW (within reason)• ETL QA Team ensures raw numbers in tables are accurate• OBI Tests are relative to those numbers even if incorrectAutomation of OBI Testing: Build test reports in OBIalongside reports in DEV• Place in separate IT only dashboard• Can be run in any environment at any time• Excellent Automation technique• Great for fast diagnosis of problems – catching unintended consequences 17
  • 18. 1. Skeleton accuracy: (Tables & Joins) • Does OBI generate the proper SQL? (BI Architect) • Do the metric values remain constant for each dimension? (Tests join paths and aggregates)2. Derived metric accuracy: • Check OBI Derived metrics based on an Anchor metrics3. Dimensional attribute accuracy: • Is descriptive data coming over from the source properly?4. Data security model accuracy: • Are filters applied properly to existing queries? • Is the SQL generated still correct & results consistent?5. Drill Paths: • Confirm that dimensional drill paths are correct 18
  • 19. “Tree Top” Tests • Break an Anchor metric out by the tops of each dimension • Make sure correct SQL & correct tables are used (Architect) • Demonstrates Unit Test • Also try multiple dimensions at the same time if possible 19
  • 20. Raw OBI mappings to base Fact table fields are“Base” or “Anchor” metrics Database Table • E.g.: Count(Headcount_Ind) or Sum(Total_Amt) • There can be filtering logic in the RPD • ETL QA verifies that these are correct Especially for BI Apps projects M1 M2Derived metrics are those calculated in OBI: • Filtered metrics: Headcount vs. Employee Headcount • Time Series: Prior Year Employee Headcount M3 M4 • Rates & Ratios: # Cases per Employee • Complex Metrics: Rolling 12 Avg. HeadcountIncorrect Anchor metrics due to ETL do not matter M5 M6 • Prior Year $ should match even if the TOTAL_AMT_USD field is wrong in the Database • Data fields are variables, just like algebra: Prior Year(x) M7 20
  • 21. Hint: Capture definitions leveraging other definedfields Reuse BI definitions as opposed to always mapping to raw tables 1. Order.Status: IfNull(ORDER_TABLE.STATUS_CD, ‘Unspecified’) 2. # Orders: Count(ORDER_TABLE.ORDER_NUM) 3. # Open Orders: # Orders where Order.Status = ‘Open’Reuse business terminology as much as possibleThree benefits:1) Makes creating test cases much easier2) Communicates the definitions better to business users3) Helps developers reuse logic when building in RPD 21
  • 22. Make reports that confirm the Derived metrics usingtheir Anchor metricsUse color coding to assistUse report calculations to demonstrateCreative solutions are a must! Report Calc 22
  • 23. Try to avoid downloads to Excel if possible• Hinders automationUse two reports if needed – Be Creative!Provide some instructions;• These reports should be used for a very long time RSUM() RSUM() RCOUNT() RCOUNT() 23
  • 24. Again be creative!This test verifies Prior Year Ship $ is accurateSolution: Run the report for 2012 and 2011• Compare 2011.CY Ship $ to 2012.PY Ship $ 2011 2012 24
  • 25. Extensive OBI ad-hoc testing may take too long Leverage your reports as surrogates for much of the OBI ad-hoc tests • They will include multiple dimensions (Skeleton test) • Various versions of the metrics within the many report structures • “Hit it from multiple angles” More reports per topic greater confidence 25
  • 26. Can be done by an internal QA group not familiar with all the detailsMust have a decent report spec to use • Difficult if an iterative report design approach is taken – minimal specs to useDisplaying the proper data set - filters (Report shows Open Orders only)All columns relatively match (% Deviation = 100 * (Plan – Actual)/ Actual) • Can be done without OBI & ETL testing (Algebra again!)Drill downs and navigations work • For navigations, key test is making sure #s remain the same!Interactions with promptsUI Items: Labeling, formatting, colors, conditional formats, UI Standards,Help links, etc. 26
  • 27. 3 Aspects: • Visibility – see the correct dashboards & subject areas & folders • Capability – Answers access, create iBots, etc. • Data Access – Correct dataset - data filtering is happening ReportsSome security testing should be done during OBI tests • Are the basics working? Security • Will it work for ad-hoc? OBI ModelCreation of test accounts flows easily from securitymodel design DatabaseFinal layer is to run reports as test users and verifydata set accuracy • For this user, for this report, are the numbers what they are supposed to be? 27
  • 28. ETL QA RolesStrong SQL, source system knowledge, source Systems Analysts, Developers,system access for entering records Source SMEsOBI ModelOBI Answers, knowledge of data model, metric Systems Analysts, QA Team, OBIdefinitions guide Developers, OBI ArchitectReportsAbility to independently confirm data from the Systems Analysts, Businesssource, general analyst, business user Analysts, End Users, QA TeamSecurityGeneral Analyst or power user skillset, Answers Systems Analysts, Business Analysts, End Users, QA TeamInfrastructureDeep technical skills, typically those who set up Infrastructure Adminsthe infrastructure 28
  • 29. 1. Perform QA as early in the process as possible2. Design a QA plan with your team’s skill sets in mind3. Plan QA Cycles around the Build4. Automate as much as possible5. Don’t ignore OBI Ad-hoc tests 29
  • 30. Q&A 30
  • 31. Contact Us facebook.com/kpipartners linkedin.com/company/kpipartners twitter.com/kpipartners youtube.com/kpipartnersThe Leader In Oracle BI & EPM 31
  • 32. Contact UsEmail: info@kpipartners.comWeb: kpipartners.com/contactKPI World Headquarters North America Offices39899 Balentine Drive New York, NY Minneapolis, MNSuite #375 Chicago, IL San Diego, CA Boston, MA Greensboro, NCNewark, CA 94560Phone: (510) 818-9480 Global Offices Bangalore, India Hyderabad, IndiaThe Leader In Oracle BI & EPM 32
  • 33. www.kpipartners.com 33

×