Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this document? Why not share!

SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario: Testing Performance and Scalability of the SAP Convergent Invoicing Package with IBM Workload Optimized Solutions and IBM Easy Tier



This technical white paper describes a project to test the performance of the SAP Convergent Invoicing package in the scenario of a company having 50 million active customers. To know more about ...

This technical white paper describes a project to test the performance of the SAP Convergent Invoicing package in the scenario of a company having 50 million active customers. To know more about IBM’s portfolio of products for big data, visit http://bit.ly/T0NnOs.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario: Testing Performance and Scalability of the SAP Convergent Invoicing Package with IBM Workload Optimized Solutions and IBM Easy Tier SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario: Testing Performance and Scalability of the SAP Convergent Invoicing Package with IBM Workload Optimized Solutions and IBM Easy Tier Document Transcript

  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario Testing Performance and Scalability of the SAP® Convergent Invoicing Package with IBM Workload-Optimized Solutions and IBM Easy TierParticipating Groups• SAP Value Prototyping, Center of Excellence, SAP Germany• IBM SAP International Competence Center, IBM Germany• IBM Research and Development, IBM R & D GermanyTechnical White Paper
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioTable of Contents 5 Huge Billing Volumes: 11 Design of the SAP Landscape A Challenge to the 11 SAP System Setup Telecommunications Industry 12 DB2 Best Practices 5 Scope of the Proof of Concept 25 Scenario Description 6 System Maintenance Not in Project Scope 26 Scenario Execution 6 Component Overview of SAP 2 8 Results and Achievements Landscape Used in Project 31 Best Practices and 6 SAP Convergent Invoicing, Conclusions Version 6 48 Resources 7 IBM DB2 LUW 9.7 Optimized for SAP Software 8 IBM eX5 Enterprise Systems 9 IBM Storwize V7000 0 IBM Easy Tier 1 2
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioExecutive Summary • Billing of 2.5 million business partners, Telecommunications ScenarioThis technical white paper – jointly pro- which includes the aggregation ofduced by SAP and IBM – describes a 2.275 billion BITs The company has 50 million activeproject to test the performance of the customers, each placing 30 calls aSAP® Convergent Invoicing package. • Invoicing of 2.5 million business part- day, producing a total of 1.5 billionThis software is used in various service ners (customers) billable items (BITs) per day, whereindustries, such as telecommunications, each BIT represents one phone call,electronic toll collection, transportation, The performance project demonstrated one SMS, or one other unit of servicepostal services, and Internet-based that all KPIs could be met by using the (for example, a ringtone or musicretail business. The test scenario was following hardware: IBM System x X5 download).built around the requirements of a large server, IBM Storwize V7000, IBM DB2,telecommunications company. and SUSE Linux Enterprise Server.The requirements of the company in thescenario helped establish the followingkey performance indicators (KPIs) forthe test. The following tasks had to beaccomplished in less than 18 hours:• Upload of 1.5 billion BITs, with a mini- mum of 100,000 BITs uploaded per second 3
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioSummary of Results This technical white paper describes the:IBM Easy Tier1 was able to shorten • Data environment chosen for billingthe processing time for all three and invoicing and the scenarios testedtasks (upload, billing, and invoicing)from 23 hours to 16.5 hours, a reduc- • Underlying IT infrastructure, SAP sys-tion of over 30%. In addition, by using tem setup, and design reasonsstorage virtualization, IBM StorwizeV7000 eliminated any storage perfor- • Database approach, design, and tun-mance bottlenecks. ing recommendations as implemented on the IBM DB2 database Project Team Dilip Radhakrishnan SAP Project manager Peter Jäger SAP Project coach SAP Markus Fehling IBM Project coach IBM Storage specialist Gerrit Graefe SAP Development architect, order to cash Michael Stafenk SAP Value prototyping, DB2 expert Ingo Dahm SAP Value prototyping, Linux expert Torsten Fellhauer SAP Value prototyping, network expert Elke Hartmann-Bakan IBM DB2 specialist Holger Hellmuth IBM DB2 specialist Jörn Klauke IBM DB2 specialist Thomas Rech IBM DB2 specialist Maik Gasterstädt IBM Storage specialist1 IBM Easy Tier is a software function within the IBM storage systems, designed to increase the IOPS performance. For more information, refer to the section about IBM Easy Tier on p. 10. 4
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioHuge Billing Volumes: • Transportation companies shipping kind of data volume a large telecommu-A Challenge to the tens of thousands of containers nications company might be expectedTelecommunications Industry across the world to handle:Telecommunications companies facethe challenge of keeping detailed To process billions of records or trans- • 50 million business partners (custom-information about millions of daily calls actions per day, enterprises need a ers) in the systemand SMS messages – for both report- high-performance IT infrastructureing and billing purposes. For example, that allows batch jobs to run quickly. • 30 billable items per business partnercompanies might need to recalculate Consequently, companies are looking per day, where each BIT representsoffered and applied tariffs based on the to exploit technologies that promise to one call or one SMS messagebuying behavior and usage pattern of significantly reduce the runtime of batchconsumers. Large telecommunications 2 jobs. • 2.5 million bills per daycompanies therefore frequently have toaggregate billions of BITs every month IBM Easy Tier can play an important The project operated under thewhen they bill their customers. This high role here. The goal of this performance assumption that all business partnerstransaction volume applies not only to project was to prove that IBM Easy Tier (customers) would receive a monthlythe telecommunications industry but can help reduce the overall batch bill and that the telecommunicationsalso to others such as: runtime. company performs billing runs only on weekdays in order to keep weekend• Web shops with customers download- system load low to allow for mainte- ing millions of music titles per day Scope of the Proof of Concept nance. This assumption lead to the The main objective of the project was to project team using the following data• Postal services managing millions of prove the capability of SAP Convergent volumes: letters and packages Invoicing to handle large data volumes using an IBM enterprise-class, Intel • Upload of 1.5 billion BITs per day• Toll collection agencies tracking thou- processor–based architecture, and to sands of cars passing each day along demonstrate how much the customer • Billing of 2.5 billion BITs per day, hundreds of roads might benefit from intelligent storage distributed evenly over 2.5 million busi- system architectures, such as IBM Easy ness partners (customers) Tier. Accordingly, the project chose the • Invoicing of 2.5 million bills per day2 This happens, for example, with mobile phone customers when the first free phone calls (or SMS messages) are not managed directly by the telecommunications company’s own network. The company will receive the billable items from roaming partners with some delay. The order of incoming BITs might not be in the same order as the calls were made. In this case a recalculation is necessary to determine which call is free and which is not. Depending on the result, the price per call might need to be recalculated as well.. 5
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThese three tasks had to be processed Component Overview of SAP or charging systems – to consolidatein less than 18 hours, allowing time for Landscape Used in Project the information into a single invoice.further steps in the order-to-cash sce- This section provides a brief overview SAP Convergent Invoicing providesnario, such as payment, dunning runs, of the components used in the perfor- a single view of customer data, withor data extractions from the business mance test. historical items stored in the contractwarehouse. accounts receivable and payable func- SAP Convergent Invoicing, Version 6 tionality of the SAP software. ExamplesThe proof of concept focused on SAP Convergent Invoicing enables an of historical data include overdue openperformance tuning of the upload enterprise to pull information from sev- items, disputed charges, or paymentsand billing part, because these parts eral billing streams and individually rated made. All of this information can beconsume more than 90% of the entire service events and – from various rating included in the final invoice in an easy-batch runtime. To make this test as real- to-understand format.istic as possible, the project team tunedperformance with an SAP production Figure 1. SAP Convergent Invoicingsystem in mind. This meant that everytuning setting was verified as if it were SAP® Convergent SAP Customer Financial SAP Convergent Invoicingapplicable to a real SAP production Charging Managementenvironment. In other words, there wasno artificial benchmarking. BIT Invoices Contract Billing: accounts Billion events BIT Billable items Open Invoices receivable each day Invoice and payableSystem Maintenance Not in BIT Storage and creation processingProject Scope BIT Legacy billing systemThe project scope did not include system Legacy rating systemmaintenance, such as SAP software • Receivables Multichannelupgrades or integrated backups, and management billthe paper does not cover this aspect. In presentment • Payment processingaddition, the concepts of high availability • Credit management Customer Partner(HA) and disaster recovery (DR) are not invoicing invoicing • Partner settlementtreated. 6
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIBM DB2 LUW 9.7 Optimized for SAP solutions, improve performance, releases and forward into the future.SAP Software and help ensure a cohesive combina- The road map shows how both com-SAP applications generate a vast tion of application and database work. panies introduce new functionality in aamount of data in day-to-day opera- IBM DB2 9, optimized for SAP software, planned way with smooth migrationstions, so no infrastructure component is an example. from one version to the next.is more important than the database.IBM has pioneered the development Figure 2 illustrates the evolving road IBM DB2 optimized for SAP softwareof data management technologies that map of DB2 optimized for the SAP soft- is the only SAP-supported databasereduce the total cost of ownership for ware, looking back through four past available that operates on all SAP- supported hardware environments fromFigure 2. DB2 Optimized for the SAP Road Map Linux, Microsoft Windows, and UNIX to IBM System i and IBM System z. DB2 provides the widest choice of support SAP NetWeaver 7.0 and higher for server, storage, and virtualization Integrated near-line storage • Integration of DB2 pureScale • 2013 technology for SAP deployments. Plus, MDC advisor stage 2 • SAP NetWeaver 7.0 EHP 1 it can be shipped and integrated with 2012 Database performance warehouse • Integrated workload management • SAP applications as a single product. 2011 SAP NetWeaver 7.0 SR3 Turnkey compression and HA solution • 2010 DB2 9.8 pureScale Integrated MDC advisor • • OLTP scale out IBM and SAP experts are dedicated to Deferred table creation • • Continuous availability SAP NetWeaver 7.0 2009 • Seamless OS and hardware working closely together to help ensure Embedded database • maintenance TCO: self-tuning • Version 9.7 that IBM DB2 sets the standard for all Minimal admin • 2008 • Full 360-degree monitoring • Near-0 storage admin other databases in the SAP ecosystem.SAP NetWeaver® 2004 • Extending online operations 2007 Streamlined admin • Version 9.5 • Even deeper deep compression In addition, IBM and SAP teams are Streamlined install • • Integrated FlashCopy 2006 • Threaded architecture performing joint projects that focus on a • DPF scaling improvements Version 9.1 • Integrated and automatic HA and DR range of areas, including performance, 2005 Compression benchmarking, functionality over first • • Storage limits removed • Selected autonomic/TCO features Version 8.2.2 of a kind (the project was breaking new • Automatic storage admin • Deployment optimized for SAP MDC = Multidimensional clustering ground with these tests), best practices, TCO = Total cost of ownership DPF = Database partition feature or combinations thereof. 7
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioFor SAP and IBM customers, this tight essentially a compromise between best and offer significant new capabilitiescollaboration means real tangible value practices and performance require- and features that address the followingin terms of performance, attractive ments. To effectively handle this large key requirements for SAP virtualizationlicense and maintenance fees, easy amount of data, the team applied DB2 solutions:usability, and innovative technology that compression, which reduced the calcu-can result in real savings. Specifically, lated storage needs by approximately • Maximum memory with unique expan-IBM DB2 provides real value by: 60% – from 50 TB to 20 TB. sion capabilities• Improving the SAP system response IBM eX5 Enterprise Systems • Fast and integrated data storage time by up to 40%, protecting existing The IBM eX5 product portfolio – repre- options hardware investment3 senting the fifth generation of servers built on Enterprise X-Architecture – was • Logical partitioning of the IBM System• Reducing, via compression, SAP used as the SAP application server x server (FlexNode) data storage needs by up to 70%, in the proof of concept. IBM servers dramatically saving on energy with IBM eX5 technology are a major The ability to modify the memory costs, administration, and hardware component in ever-changing IT infra- capacity independently of the proces- investment4 structures; they offer significant new sors, and the new high-speed local capabilities and features that address storage options, mean this system• Enabling automated SAP features that the key requirements for customers with can be highly utilized, yielding the best can reduce database administration SAP solution landscapes. return on application investment. These time by 30% and free up resources to systems enable enterprises to grow better manage the business 5 The IBM System x server portfolio their processing, I/O, and memory provides an ideal platform for SAP dimensions, provision what they needThe proof of concept was meant to applications that run virtualized in a now, and expand the system to meetbe based on a combination of perfor- private cloud environment. With mul- future requirements.mance, throughput, and best practices tiple workloads running on the samewith data volumes that had never been server, performance remains important, Memory Access for eX5 (MAX5)tested before for that particular appli- but reliability and availability become MAX5 is the name of the memory scal-cation setup. Therefore, the layout and more critical than ever. Enterprise serv- ability subsystems – memory expansionsetup of the database engine took ers with IBM eX5 technology are a key that can be added to eX5 servers.those aspects into account. It was component in a dynamic infrastructure MAX5 for the rack-mounted systems3 SAP IT case study, GK12-4329-00 (12/07).4 Refer to www.ibm.com/solutions/sap/us/en/landing/J233701A22235G06.html.5 IWB case study, SPC03025-CHEN-01 (04/08). 8
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario(System x3690 X5, System x3850 X5, • Two Intel Xeon E7 2800/4800/8800 • Four Xeon E7 2800/4800/8800and System x3950 X5) is in the form series (up to 10 core) or two Intel series (6 core/8 core/10 core) or Xeonof a 1U device that attaches below the Xeon 7500 or 6500 families (up to 6500/7500 seriesserver. 8-cores) • Scalable to eight sockets by connect-IBM System x3690 X5 • Max 1 TB RAM with MAX5 technology ing two x3850 X5 servers togetherThe x3690 X5 is positioned for SAPlarge application servers and SAP IBM System x3850 X5 • Up to 3 TB RAM with MAX5 technologydistributed applications. Often, it’s not IBM System x enterprise servers arethe capacity of processors that limits the ideal platform for business-critical IBM Storwize V7000virtualized systems for SAP solutions. and complex SAP applications, such Storwize V7000 is a powerful midrangeInstead, SAP virtualization solutions as database processing, customer disk system, designed to be easy to usedepend more on the memory capac- relationship management, and enter- and to enable rapid deployment withoutity of the host systems. With MAX5 prise resource planning, as well as additional resources. Storwize V7000memory expansion the overall systems highly consolidated, virtualized server system is virtual storage that offerscan scale up without adding additional environments. greater efficiency and flexibility throughservers or licenses. built-in solid-state drive (SSD) optimiza- With multiple workloads running on tion and thin provisioning technologies.The IBM System x3690 X5 is a scalable the same server, performance remains Storwize V7000 advanced functions2U, two-socket rack-optimized server. important but reliability and availabil- also enable the nondisruptive migrationThe x3690 X5 is a system with the ity become more critical than ever. of data from existing storage, simplify-same benefits known from the flagship Servers with IBM eX5 technology ing implementation and minimizingsystem x3850 X5. are a major component in a dynamic disruption to users. Storwize V7000 also infrastructure and offer significant new enables the virtualization and reuse ofSee the following Web page: capabilities and features that address existing disk systems, supporting awww-03.ibm.com/systems/x key requirements for customers with greater potential return on investment/hardware/enterprise. SAP landscapes. (ROI).The IBM System x3690 X5 has the fol- The IBM System x3850 X5 has the fol- See the following Web page:lowing main features: lowing main features: www-03.ibm.com/systems/storage /disk/storwize_v7000/index.html. 9
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIBM Easy Tier load. Because the workload is mostlyIBM Easy Tier is software functionality random, the to-be-read data mustwithin the IBM storage systems, and is be located physically; the data is notavailable for IBM Storwize V7000 as well stored in any cache. This results in aas IBM System Storage SAN Volume lengthy response time in the case ofController and IBM System Storage HDDs. Here, SSDs have a much bet-DS8000 series. ter response time; because they do not have any mechanical parts, theEasy Tier is designed to decrease the response time is just a fraction of theI/O response time and thereby increase HDD response time, even under load.the input/output operations per second(IOPS) performance. Easy Tier deter- Because of performance aspects, themines the appropriate tier of storage, best solution might be to store thebased on data access requirements, entire SAP database on SSDs. Evenand then automatically and nondisrup- though SSDs are much more expensivetively moves data to the appropriate tier compared to HDDs, nevertheless theat the subvolume or sub-LUN (logical price per performance is cheaper forunit number) level; typically between SSDs. Easy Tier actually combines bothSSDs and hard disk drives (HDDs). This technologies, achieving low price perfeature is designed to reduce, if not capacity with HDDs and low price pereliminate, the amount of manual effort performance with SSDs.involved. Basically, Easy Tier monitors the per-The most critical workload for stor- formance requirements of a virtual diskage systems is online transaction (VDisk, LUN); it measures IOPS perprocessing (OLTP), more precisely large block.the random-read part of this work- 10
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioFigure 3. IBM Easy Tier Design of the SAP Landscape This section describes the setup of the Easy Tier managed storage pool SAP landscape used for the perfor- mance test. SSD arrays SAP System Setup “Cold” blocks “Hot” blocks The system landscape consisted of two migrate down migrate up different SAP systems, with the SAP IDs ETG and ETL. The ETG system simu- HDD arrays lated the rating engine, which usually generates the billable items (BITs), was Logical volume not part of the performance evaluation, Storwize V7000 and is not described in this paper. The ETG system was installed on four rack server blades: three blades were usedIf a high clipping level is reached, the during a given time frame (for example, because the application server, thedata blocks are marked as “hot” and 24 hours). Statistically, there are con- SAP database (DB), and SAP centralmoved from the lower, slower tier (HDD) tiguous areas accessed, and some of instance (CI) were installed on the fourthto the higher, faster tier (SSD). In addi- them will be hot. The change rate of blade.tion, after the data has been cooled the hot areas is not within minutes, but(the IOPS requirements per block have most likely will remain over a longer From ETG, the billable items were sentdecreased) and the low clipping level period of time – for example, 24 hours. to the ETL system, which was the mainhas been reached, the data is migrated Here, Easy Tier is able to move these test system (DB and CI). This setupback from SSD to HDD. hot areas from HDD to SSD, and as a guaranteed that the creation of BITs result, the SAP transaction time will be would not influence the throughput ofValue of IBM Easy Tier to SAP reduced. the upload phase in the ETL target sys-The workload of systems supporting tem. Before the test started, the teamthe SAP ERP application is defined by ensured that the injector system (ETG)OLTP. Typically, not all data in the SAP was not the limiting factor of the upload.system’s database will be accessed 11
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThe test landscape consisted of one Figure 4. Test System Architectureserver – the IBM system x3850 X5,which hosted the database and the LANSAP central instance. The IBM systemx3850 X5 (7145-AC1) consisted of: IBM x3690 X5 IBM BladeCenter H ETG system ETL App IBM Total 30,000 SAPS IBM x3690 X5 IBM x3850 X5 Storwize V7000• Four 8-core Xeon 7560 series 2.26 ETL App Blade 1 CI and DB IBM x3690 X5 GHz processors ETL App ETL DB and CI 168 HDDs Blade 2 App 50,000 SAPS 32 SSDs Blade 3 App Total 100,000• 148 GB memory Blade 4 App SAPS SAN• Two HBA Emulex 4 Gbps SAP ETG system SAP ETL system SAPS = SAP® Application Performance Standard• Four internal 146 GB SAS HDDsThree IBM system x3690 X5 (7148-AC1) database server and CI are in a high- DB2 provides an automatic storagesystems were used as application serv- availability cluster while the application feature, allowing automatic growth iners, each with: servers are not. the size of the database across disk and file systems. DB2 also offers a high-• Two 8-core Xeon 7560 series 2.26 DB2 Best Practices availability and disaster-recovery feature GHz processors During an installation of SAP software, (HADR) together with Tivoli System implementation teams install and Automation for Multiplatforms. Other• 128 GB memory parameterize DB2 in an optimal way to settings, such as the DB2 aggregated guarantee high performance and ease registry variable DB2_WORKLOAD, self-• Two NICs (network interface cards) of use. However, for the critical parame- tuning memory management (STMM), ters, customers can change the settings and automatic and real-time statistics• Four internal 146 GB SAS HDDs to values that best fit their needs. There (RTS), are parameterized during SAP is a set of features and functionalities installation to values proven to be optimalThe team chose this architecture that can be activated or adapted during for most of the SAP workload; thesebecause it is standard in many cus- installation, such as DB2 compression, values can be changed later to bettertomer environments where the instance memory setting, or deferred suit specific workload requirements. table creation. 12
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThis proof-of-concept project worked tables based on those patterns to average, one for each available CPU inmainly with the default values rec- achieve the best compression results. the system.ommended by SAP, but specific The team also used row and indexparameters were modified to improve compression, including temporary DB2 Parallel I/Operformance for the current SAP work- table compression. For more informa- In addition, the team wanted to increaseload and are described in this section. tion about how this was done, see the query performance and optimize the following article on the SAP Developer I/O resource utilization by applying theDB2 Version Network site: Best Practice Using DB2_PARALLEL_IO registry variable.The proof of concept used DB2 9.7 DB2 Compression Feature in SAP By default, a DB2 database systemFixpack 3 throughout the project. Environment, available online at places only one prefetch request at a www.sdn.sap.com/irj/scn/index?rid= time to a table space container. ThisLinux ext3 file System /library/uuid/a02d282d-9074-2d10 is done with the understanding thatThe recommendation was to use a file -5496-ec2c65028a83. multiple requests to a single device aresystem type ext3 under Linux. serialized anyway. If a container resides SAPDATA Layout on an array of disks, there is an oppor-IBM DB2 Storage Optimization Feature The database was expected to grow tunity to start multiple prefetch requests(DB2 Compression) up to 50 TB. As a result, the data simultaneously, without serialization.With massive amounts of data from SAP placement needed to be consideredConvergent Invoicing, the IBM DB2 stor- very carefully in order to provide opti- The parameter DB2_PARALLEL_IOage optimization feature contributed mal read-write performance from the enables the DB2 system to start prefetchto massive storage savings. The SAP database. Today’s storage systems requests in parallel for a single con-team calculated database size savings are built on multilayer abstraction tainer, which might help to increase I/Oof approximately 60% with DB2 com- levels and the relationship between throughput. In this proof-of-conceptpression – from about 50 TB down to file systems and disks – as seen by project, the parallelism parameter was20 TB. the operating system – often does set to the value of 2, doubling the num- not reflect the physical conditions. ber of I/O servers from 32 to 64. TheThe largest tables (/1FE/0LTxxxIT, Therefore, a one-to-one relationship DB2_PARALLEL_IO registry variable/1FE/0LTxxxIT00, BALDAT) comprise between SAPDATA file systems and also computes the prefetch size foraround 97% (19 TB) of the total data- Linux logical volume was chosen, each table space if the PREFETCHSIZEbase size (19.6 TB). It was therefore without using the Linux logical volume option is set to AUTOMATIC, the defaultimportant to build an optimal com- manager. The team configured a for SAP systems.pression dictionary and compress the total of 32 SAPDATA file systems – on 13
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioFigure 5. DB2 Prefetch Configuration * Formula – how does DB2 determine the number of physical disks per container? Optimal prefetch size for improved query performance 64 prefetcher – DB2_PARALLEL_IO not specified then # of physical disks per container defaults to 1 Container 1 Container 2 Container 32 Table A Table A Table A – DB2_PARALLEL_IO=* # of physical disks per 1 2 3 4 5 6 7 8 125 126 127 128 container defaults to 6. – DB2_PARALLEL_IO=*:2 setting used in this PoC Disk group SAPDATA (MDG) was built up with 128 HDDs Extensize * (number of containers) * (physical disks*) = Prefetch size 2 * 32 * 2 = 128 -> with these settings a full table scan keeps the 128 disks busyBased on the formula shown in Figure 5, db2set DB_PARALLEL_IO=’*’:2 * = for all tablespacesa prefetch size of 128 pages was cal- 2 = level of I/O parallelism is twoculated, based on the settings usedin the proof of concept. In that way, for Old Valueexample, it was very likely that all 128 Number of I/O servers (NUM_IOSERVERS) = AUTOMATIC (32) DB2_PARALLEL_IO - was not setphysical disks would be used during atable scan, resulting in an optimized I/O New Valueresource utilization. Number of I/O servers (NUM_IOSERVERS) = AUTOMATIC (64) [i] DB2_PARALLEL_IO=*:2Key SAP Tables and IndexesThe definitions and storage of the keytables in SAP Convergent Invoicing weremodified to offer optimal insert, update,and delete performance and better man-ageability of large tables. The tables weresplit and the “append on” mode set. Formore information about the “append on”table definition, see p. 16.Table Space DefinitionsFor maintenance reasons, the team Tables: Tablespace name = ETL#XBD Indexes: Tablespace name = ETL#XBImoved the billing tables and theirindexes to separate table spaces. 14
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioCreating table spaces The table spaces were created by the following commands: CREATE LARGE TABLESPACE “ETL#XBD” IN DATABASE PARTITION GROUP SAPNODEGRP_ETL PAGESIZE 16384 MANAGED BY AUTOMATIC STORAGE AUTORESIZE YES INITIALSIZE 320 M MAXSIZE NONE EXTENTSIZE 64 PREFETCHSIZE AUTOMATIC BUFFERPOOL IBMDEFAULTBP OVERHEAD 7.500000 TRANSFERRATE 0.060000 NO FILE SYSTEM CACHING DROPPED TABLE RECOVERY OFF; CREATE LARGE TABLESPACE “ETL#XBI” IN DATABASE PARTITION GROUP SAPNODEGRP_ETL PAGESIZE 16384 MANAGED BY AUTOMATIC STORAGE AUTORESIZE YES INITIALSIZE 320 M MAXSIZE NONE EXTENTSIZE 64 INCREASESIZE 128 M PREFETCHSIZE AUTOMATIC BUFFERPOOL IBMDEFAULTBP OVERHEAD 7.500000 TRANSFERRATE 0.060000 NO FILE SYSTEM CACHING DROPPED TABLE RECOVERY OFF; The settings for the following table space parameters were modified against the SAP default table space definitions: • INITIALSIZE 320 MB – the initial size of the table space. Since 32 SAPDATA directories were used, each container had an initial size of 10 MB. • EXTENTSIZE 64 – every extent contains 64 pages. This avoids the effort of allocating the pages to tables more often and was useful in this scenario. • INCREASESIZE 128 MB – decreases the effort of allocating space from the file system to the table space more often and thus improves performance and avoids file system fragmentation. 15
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioMoving Tables and Indexes to After table space creation, the tables were moved to the new table spaces with theNew Table Spaces help of the online table move tool: sysproc.admin_move_table available since DB2 9.7: call sysproc.admin_move_table(‘SAPSR3’,’/1FE/0LT023IT’, ‘ETL#XBD’, ‘ETL#XBI’, ‘ETL#XBD’, ‘’, ‘’, ‘’, ‘’, ‘’, ‘MOVE’);Optimizing Compression Ratio, During the table move above, DB2 triggered the automatic dictionary creation (ADC)Performing Table Maintenance after a certain amount of data (approximately 20 MB) had been inserted into the new table. In this project, the compression rate of the dictionary created by ADC was low, because of the low filling level of the used tables. To gain a much higher compres- sion rate, the team executed a billing run to fill up the tables, and then reorganized the tables, including the re-creation of the dictionary, resetdictionary. Afterward a manual runstats was performed for each of the tables: reorg table SAPSR3.”/1FE/0LT000IT” resetdictionary; runstats on table SAPSR3.”/1FE/0LT000IT”;Defining Profile Statistics To keep the performance impact low during the automatic data collection of the large tables with the runstats utility, a statistics profile was registered for each of these tables. As result, only 1% of the data was read by the autorunstats command. runstats on table sapsr3.”/1FE/0LT000IT” tablesample system(1) set profile only;Table Definition “append on” To provide better insert performance, the team modified table definitions with the “append on” option. When this option is used, DB2 simply sticks the new row at the end of the table, makes no attempt to search for available space, and makes no effort to preserve any kind of clustering order. Note that reuse of space made avail- able by delete or update activity, which changes row size, does not occur until the table is reorganized. alter table ‘SAPSR3’,’/1FE/0LT000IT’ append on 16
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioDB2 Log Files and Log Buffer (DB The DB2 online log files were placed on a separate fast SSD disk to provide highParameters) write performance. Based on the high insert, update, and delete rates, the number and the size of the log files were increased to the following values: Old values: Log file size (4KB) (LOGFILSIZ) = 16380 Number of primary log files (LOGPRIMARY) = 20 Number of secondary log files (LOGSECOND) = 80 Log buffer size (4KB) (LOGBUFSZ) = 1024 New values: Log file size (4KB) (LOGFILSIZ) = 128000 Number of primary log files (LOGPRIMARY) = 150 Number of secondary log files (LOGSECOND) = 50 Log buffer size (4KB) (LOGBUFSZ) = 16384 The team increased the log buffer size because they had seen log buffer overflows within a database uptime of less than 24 hours, resulting from the large amount of data manipulating language (DML) (insert, update, and delete) statements per transaction. This implies that DB2 had to do multiple physical write I/Os to facilitate commit processing of a single transaction, which resulted in a performance degra- dation of response time and throughput. Because performance degradation can occur when secondary logs are used, the recommendation – based on the project results – was to set the number of secondary log files to 0. There was overhead in allocating and formatting the sec- ondary log files. For best performance, primary log space had to be allocated in sufficient quantity, such that the allocation of secondary logs was unnecessary.Proactive Page Cleaning This alternate method differs from the default behavior in that page cleaners behave more proactively in choosing which dirty pages get written out at any given point in time. This method doesn’t respect the database parameter chngpgs_thresh. 17
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioSo this technique is more aggressive cleaning, the DB2 registry variable www.sdn.sap.com/irj/sdn/go/portal /prtroot/docs/library/uuidand spreads out the page cleaning DB2_USE_ALTERNATE_PAGE_ /b0aabcc9-afc1-2a10-5091-b5cd-work by writing more frequently, but to CLEANING had to be set to on. a33036b0.fewer pages at a time. It also improvesthe way DB2 agents find free pages in STMM and Instance Memory In this proof on concept, the team usedthe buffer pool. This allows the page In order to set selected memory areas the standard settings for STMM, withcleaners to use less disk I/O bandwidth in DB2 to a specific size, it’s possible to the instance memory set to a fixedover a longer time. switch off STMM or to use STMM. This value (default for SAP installations) and can be an option in environments where let STMM adapt the remaining memory.Test runs with the billing workload the memory requirements are knowndemonstrated that as well as a run- and the workload rarely changes. For The memory allocated by DB2 istime improvement of up to 6%, Disk more information, refer to the following controlled by a single parameter,Write KB/Sec was reduced by over article on the SAP Developer Network INSTANCE_MEMORY, while two other60% (see Figure 6), whereas the Disk (SDN) site: The Evolution of the Memory important parameters, DATABASE_Read KB/Sec and the IO/sec slightly Model in IBM DB2 for Linux, UNIX, and MEMORY and APPL_MEMORY, controlincreased. To activate alternate page Windows. You can find the SDN site and the allocation of database-level memory the article online at: and application-level memory (within the limits provided by INSTANCE_MEMORY). Nearly all other memory configuration Figure 6. Summary Disk Throughput Without and With Alternate Page Cleaning parameters for the different memory heaps used by DB2 now support an Disk throughput – average Billing runs AUTOMATIC setting. Disk read KB/s Disk write KB/s I/O per sec 140 160,000 Thousands Without STMM, extensive monitoring 140,000 and adjustments would have had to 120 120,000 be performed for the different memory 100 areas (at each stage during the scal- . 10,0000 80 I/O per sec ing) to achieve optimal performance for 8,000KB/sec 60 each of the various workload profiles. 6,000 40 4,000 20 2,000 0 0 Without alternate page cleaning With alternate page cleaning 18
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioDB2 Instance Memory The database server had a total of 148 GB of main memory, with the SAP two-tier(DBM Parameter) architecture database and central instance residing on one server. The DB2 instance was set to 32 GB of memory, which is approximately 22% of the available memory. The team performed tests by tripling the memory but did not see any runtime improvement, and therefore kept the value of 32 GB for all the payload runs. Size of instance shared memory (4KB) (INSTANCE_MEMORY) -> 8192000STMM With STMM turned on, the monitoring and adjustment tasks were done by DB2, which automatically configured most of the memory settings and adjusted them at runtime to optimize performance. STMM did not require any DBA intervention to tune the memory parameters based on workload change. STMM tuned the following memory consumers within the database instance memory: • Database locking LOCKLIST and MAXLOCKS • Package cache size PCKCACHESZ • Sort memory SHEAPTHRES_SHR and SORTHEAP • Buffer pools 19
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioSTMM settings during the billing payload run (ET4):6 runstats and Reorganization Following best practice, the defaultSelf tuning memory value was kept on for AUTO_RUNSTATS (SELF_TUNING_MEM) ON and off for reorganization. WithSize of database shared memory (4KB) (DATABASE_MEMORY) AUTOMATIC(7044530) AUTO_RUNSTATS (a periodic back-Max storage for lock list (4KB) (LOCKLIST) AUTOMATIC(464480) ground process) and real-time statisticsPercent of lock lists per application (AUTO_STMT_STATS) set to on, the (MAXLOCKS) AUTOMATIC(97)Package cache size (4KB) catalog statistics were kept current so (PCKCACHESZ) AUTOMATIC(131566) that the optimizer determined the bestSort heap thres for shared sorts (4KB)(SHEAPTHRES_SHR) AUTOMATIC(916456) access path to the data for optimalSort list heap (4KB) performance. The runstats profile (SORTHEAP) AUTOMATIC(183291) definition for the large tables ensureddb2 select BPNAME, NPAGES, PAGESIZE FROM SYSCAT.BUFFERPOOLS that the impact of AUTO_RUNSTATS on system performance was negligible.BPNAME NPAGES PAGESIZEIBMDEFAULTBP -2 16384 (For more information, see the section about defining profile statistics on p. 16.)Figure 7 shows an example of how clearly how STMM can adapt the mem-STMM has aligned the database mem- ory on the fly, based on the workload. In the following description of the testory and the buffer pool over a period of The graph also shows that for the days project, the term evaluation refers to20 days. The regulation of the memory from November 18 to 20, the team exe- the processing that took place whenwithin the present time frame was cuted a series of billing runs and that, automatic statistics collection checkedcaused by a test series of billing and for example, the buffer pool memory whether or not specific tables requiredinvoicing runs. The graph demonstrates grew and shrunk by around 3 GB. statistics to be updated, deleted, or added, and then scheduled aFigure 7. STMM Alignment of the Database Memory and Buffer Pool runstats activity for the out-of-date Period of 20 days tables. 8,000,000 7,500,000 7,000,000 The first evaluation occurred within4 KB Pages 6,500,000 DB memory Buffer pool two hours of database activation. 6,000,000 Subsequent evaluations occurred 5,500,000 approximately every two hours after 5,000,000 that, as long as the database remained 31/10 31/10 31/10 31/10 31/10 16/11 17/11 17/11 17/11 17/11 18/11 18/11 18/11 18/11 19/11 19/11 19/11 19/11 20/11 20/11 1/11 1/11 1/11 1/11 2/11 2/11 2/11 2/11 3/11 3/11 3/11 3/11 4/11 STMM = Self-tuning memory management Days active.6 For more information about all memory parameter settings for run ET4, see p. 41. 20
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioAfter the load phase, the team reorganized DB2 and ran the runstats utility for allrelevant SAP tables to gain a stable state of the database. This state was saved viaFlashCopy as the “golden backup” and was later used to restore the database tothe defined state for the next series of test runs.db2 reorg table SAPSR3.”/1FE/0LT000IT” resetdictionarydb2 runstats on table sapsr3.”/1FE/0LT000IT” for detailedindexes allAutomatic maintenance (AUTO_MAINT) ONAutomatic runstats (AUTO_RUNSTATS) ONAutomatic statement statistics (AUTO_STMT_STATS) ONAutomatic reorganization (AUTO_REORG) OFFNew DBA Cockpit for DB2 from SAP This proof of concept was able to fully • Monitoring based on DB2 WorkloadThe DBA cockpit for DB2 from SAP is exploit the new cockpit. The following Management Service classesan integral part of all SAP solutions and list provides an overview of the newcovers the complete administration and monitoring features of IBM DB2 9.7 that • Easy navigation and guided proceduresmonitoring of local and remote data- have been added to the cockpit:bases. The team used the DBA cockpit • Uniform data collection with the DBAduring the proof of concept to carry out • History-based, back-end data collection cockpit and database performancemonitoring and performance analysis. warehouse (DPW) in SAP enhance-(For more information, refer to the IBM • Time-spent monitoring with drill-down ment package 2 for SAP NetWeavere-book SAP DBA Cockpit – Flight Plans capabilitiesfor DB2 LUW Administrator on the IBM • Monitoring of IBM DB2 pureScaleWeb site.) The new DBA cockpit from • New event monitors (for example, for (SAP transport available)SAP is a Web Dynpro–based user inter- locks)face and has been available since therelease of SAP enhancement package • New object metrics (for example:1 for the SAP NetWeaver® technology index access statistics, and databaseplatform. container read and write times) 21
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThe time-spent and historical data were Figure 8. DBA Cockpit from SAPextremely useful in identifying bottle-necks and interpreting runtime behavior.SUSE Linux Enterprise Server V11 TuningThe team carried out the following: defaults { polling_interval 30 failback immediate no_path_retry 5• Chose the SUSE Linux Enterprise rr_min_io 100 Server (SLES) V11.1 for this proof of path_checker tur user_friendly_names yes concept, and applied all available } devices { patches at project start (April 2011). device { vendor “IBM” product “2145” prio alua• Explicitly checked the patch level of path_grouping_policy group_by_prio the dm-multipath driver. }• Configured /etc/multipath.conf } as follows: 22
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario• Used two dual-port EMULEX HBA Figure 9. File System Layout of Test System cards, with the latest device driver from EMULEX. The EMULEX admin- / istration tool OneCommand Manager /usr /sap was installed as well; through this tool /db2 the default LUN queue length was /db2etl changed from default 30 to 64 per /ETL /sapdata1 LUN. /sapmnt /ETL /sapdata2 /sapdata...• Installed a dedicated SAN switch /sapdata32 (IBM 48 port model, 4 GB) installed – noncritical. /log_dir /db2dump• Formatted the file systems as ext3 File system Directory: Internal HDD V7000 MDG2 type; at project start, ext4 was not mount point: V7000 MDG1 V7000 MDG3 certified by SAP, and was in “experi- mentation” mode.• Installed the base OS as well as the During the installation of Linux and DB2, swap partition on internal disks – the team ran several workload tests; noncritical. when more files were used, the setup seemed to perform better. As a result,• Created a physical volume on every the team decided to use 32 file systems, LUN through pvcreate. as many as the number of installed processors.• Stored and installed all SAP and DB2 data on the Storwize V7000 system. The team used a one-to-one relation- ship between SAPDATA file systems and LUNs, did not use the Linux Logical Volume Manager, and chose the file system layout shown in Figure 9. 23
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioStorwize V7000 Setup Figure 10. Storwize V7000 ConfigurationFigure 10 shows the basic StorwizeV7000 configuration. 32 VDisks 1 VDisk 3 VDisksThe manage disk group (MDG) SAPDATA MDG SAPDATA, Easy Tier MDG EXEwas build up with 128 HDD, 2.5 inch, MDG LOG 16 HDD arrays: RAID 5, 7+1 1 HDD array450 GB capacity at 10,000 RPMs. The 3 SSD arrays: RAID 5, 5+1 1 SSD array RAID 10, 1+1 RAID 10, 2+216 arrays were configured as redundantarray of independent disks (RAID) 5, HDD MDisk SSD MDisk SSD MDisk HDD MDisk7+1. Within Storwize V7000, an internal total HDDs:142, spare:14RAID array was called managed disk total SSDs: 24, spare: 2(MDisk). In addition, two SSD-manageddisks were put into this MDG, with a Switching Between HDD, SSD, and During the billing and invoice scenarios,RAID 5, 5+1 configuration; each single Easy Tier the team tested HDD (only) and EasySSD has a capacity of 300 GB. The Storwize V7000 allowed changing Tier, due to capacity limitation on SSD. the physical storage setup or physical The command lsvdiskextent com-The DB2 log files were put into a storage layout while keeping the VDisk mand was used to identify the numberdedicated MDG, a RAID 10, 2+2 con- online. This flexibility was utilized to of extents to be moved, and then thefiguration, providing maximum I/O physically move the data (extents) of migrateexts command to move aperformance with minimum response VDisks between managed disks of type specific number of extends from onetime. A MDG with just one array was SSD and HDD. MDisk to another one. These commandsconfigured to store the EXE data, and provided the capability to move data froma RAID 10, 1+1 with two HDDs was During the upload test, the team used SSD to HDD and to switch between thechosen, providing 450 GB of usable the following storage configuration to configurations: HDD, SSD, and Easy Tier.storage. measure the performance differences: Using Storwize V7000 FlashCopyThe MDG was configured with the • HDD-only as Backupdefault extent size of 256 MB (non- The team used the Storwize V7000critical parameter); all SAPDATA VDisks • SSD only – this setup was used during space-efficient FlashCopy functionallywere configured as space-efficient; the first load phases, until the maxi- as data protection. For all VDisks, theeach SAPDATA VDisk had a size of 800 mum usable SSD capacity of 4.3 TB team created thin-provisioned VDisksGB. The EXE and LOG VDisks were was reached with the corresponding sizes, put themconfigured with thick provisioning. In the into a single consistency group, andtest environment, the LOG VDisk had a • HDD and SSD combined with Easy Tier issued the startfcmap command.size of 200 GB. 24
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioCommand sequence for backup: # mkfcconsistgrp -name BACKUP # mkfcmap -source SAPDATAxx -target SAPDATAxx_BACKUP -name SAPDATAxx_BACKUP -consistgrp BACKUP ... # startfcconsistgrp -prep -name BACKUPCommand sequence for restore: # mkfcconsistgrp -name RESTORE # mkfcmap -source SAPDATAxx_BACKUP -target SAPDATAxx -name SAPDATAxx_RESTORE -consistgrp RESTORE ... # startfcconsistgrp -prep -restore -name RESTOREOnly the command sequences for the • Billing of 2.5 million business partners, ment in other scenarios would dependSAPDATA VDisks were listed (step 2 which included the aggregation of on the hardware components used.in the list); the VDisks LOG and EXE 2.275 billion BITsneeded to be put into the consistency For the test, the team used 20 differentgroups as well. • Invoicing of 2.5 million business partners BIT classes and tested two different scenarios. In scenario 1, all the activitiesScenario Description The main objective of the test was to were done sequentially, mainly to cap-To review, the proof-of-concept project prove 1) that the solution was capable of ture data individually for each activity.took into account the following require- handling this volume within the 18-hour In the more realistic scenario 2, billingments of a large telecommunications period, and 2) scalability. Even though and invoicing still ran sequentially, butcompany: 50 million active customers the team was able to prove capability and there was, at the same time, a constantproducing a total of 1.5 billion BITs per scalability during this proof-of-concept stream of upload, running in parallel.day. Each BIT represents one phone project, further performance improve-call or SMS. Since every customer gets The steps in scenario 1 were executeda monthly invoice, the phone company in sequence.has to send out bills to 2.5 million cus-tomers every workday. Figure 11. Test Scenario 1 Concurrent processesThese typical requirements resulted inthe main KPIs of this test. In less than18 hours, the performance test had toaccomplish the following tasks:• Upload of 1.5 billion BITs, with a minimum Upload Billing Invoicing of 100,000 BITs uploaded per second Time 25
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIn scenario 2, billing and invoicing were Figure 12. Test Scenario 2run sequentially but in parallel with Concurrent processesupload. Billing Invoicing Upload TimeScenario Execution ing database load and SAP application interface of the BIT class in the ETLThe following section describes how server load. The load that the enqueue system, where they were written to thethe different runs were started. server puts onto the central instance is, database. Because synchronous BAPI® in this case, negligible. programming interface calls were used,Starting the SAP Batch Jobs the number of open RFC connectionsThe tests were mainly started by The upload activity was started on could not exceed the number of batchlaunching the appropriate SAP trans- the ETG system. The launched batch jobs in ETG, which was 150 for all tests.actions, FKKBIX_MA for billing and jobs created BITs using a report, whichFKKINV_MA for invoicing. Only for the was generated together with the cor- Billing as well as invoicing was startedcreation and transfer of the BITs did the responding BIT class to permit easy directly in the ETL system. The job dis-team need to provide a special report, testing for customers. The BITs thus tribution of scenario 1 was as follows:which ran in the ETG system. created were then sent via remote func- tion call (RFC) to the correspondingAll three activities were based on themass activity framework that permits Server Number of jobs during billing Number of jobsan easy way to launch multiple jobs in ETL App 1 25 42parallel and to distribute them over the ETL App 2 25 41 ETL App 3 25 42available servers. Though possible, nobatch job ran on the central instance;this had the benefit of clearly separat- 26
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThe upload used 150 parallel threads For scenario 2, the number of jobs was reduced because here the upload of BITson ETG; the called ETL system used a continued during billing and invoicing:load balancing mechanism to distribute Server Number of jobs during billing Number of jobsthe RFC calls equally over its applica- ETL App 1 17 14tion servers. ETL App 2 17 13 ETL App 3 17 14 ETL_CI 0 10In this scenario, the upload on ETG • vmstat -xyz during the run (OS, all DB2 Snapshot (Before and After Each Run)used only 34 concurrent threads. The servers) db2 get snapshot for databasedistribution of the RFC calls on the on <SID>receiving ETL system was done in the • iostat -xyz during the run (OS, allsame way as in scenario 1. servers) Configuration Information db2 get dbm cfg db2 get db cfg for <SID>The mass activity framework permits Clearly, some of these monitors provide db2set -all“through events” to perform special redundant information, but as each toolactions when jobs are launched or provides the information in a certainfinished, and this was used to automati- context, it makes the evaluation easier. During the proof of concept, SAPcally start and stop some monitors: released a new ad hoc data collection DB2 Monitoring tool for DB2 to collect a history of DB• STAD records written at the end of the For DB2 monitoring, the team used KPIs over a certain period. For perfor- run (SAP) the db2top utility in batch mode, the mance monitoring, the team used this database snapshot monitor, and the tool, which is provided as an attach-• SM37 information written at the end of DBA cockpit. Furthermore, the team ment to SAP Note 1603507 in the SAP the run (SAP) collected the database and database Notes tool. The data collection tool has manager configuration. a smaller footprint than the db2top• /SDF/MON capturing information dur- utility and therefore generates less per- ing the run (SAP) db2top formance overhead. db2top -d <sid> -i <interval>• nmon (OS, all servers) -m <duration> -b <suboption> -o <output file> b: batch mode• db2stat (DB2 LUW, DB server only) suboption: t = tablespaces d = database m = memory b = bufferpool 27
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioOS Monitoring Upload Each of the tests was run twice, andFor OS monitoring (processor, disk, The upload was tested in three different the following table gives the best resultsnetwork, and so on), the team used the configurations of the storage system: achieved for each test.IBM tool nmon, which was started with Test Total runtime Throughputthe SAP batch jobs. See the following • HDD (BITs per second)section for detailed results. (For more All data files have been placed on HDD 4:37:00 90,253information about nmon, see HDD, and log files on SSD. SSD 4:02:59 102,888 Easy Tier 4:02:59 96,587www.ibm.com/developerworks/aix/library/au-analyze_aix/.) • SSD All data and log files have been placedStorwize V7000 Monitoring on SSD. Figure 13 illustrates that the number ofIBM provides the PERL-based moni- BITs loaded per time interval did nottoring tool svcmon, providing very • Easy Tier deteriorate over time.detailed performance analyses about Easy tiering was switched on for datathe Storwize V7000 system. This tool files, HDD, and SSD within one MDG,runs continuously and creates reports log files on SSD within different MDGs.for a specified duration. Results for theperformance test are detailed in thesection that follows. (For more informa-tion about svcmon, see Figure 13. Scalability of Upload Runwww-03.ibm.com/support/techdocs Scalability BITs uploaded (SSD)/atsmastr.nsf/WebIndex/PRS3177.) 1,600,000,000 1,400,000,000Results and Achievements 1,200,000,000This section provides details of the run- 1,000,000,000 BITs uploadedtimes of the three different SAP batch 800,000,000jobs that were needed to process all 600,000,000 400,000,000billable items. The first part describes 200,000,000the results of scenario 1, where upload, 0billing, and invoicing run in sequence. 00:00:00 00:28:48 00:57:36 01:26:24 01:55:12 02:24:00 02:52:48 03:21:36 03:50:24 04:19:12The second part presents the results of Runtime [hh:mm:ss]scenario 2. 28
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioBilling Figure 14. Scalability of Billing RunAfter the upload was run, the database Scalability objects processed ETsize was 20 TB – too large to be stored 3,000,000on SSD only. Therefore the data file 2,500,000(SAPDATA) was stored either on HDD- Contract accounts billed 2,000,000only (and test runs are called HDD) orwith Easy Tier enabled (hot data stored 1,500,000on SSD, and these runs are called ET). 1,000,000During these runs, the log data was 500,000stored on a separate SSD-manageddisk group. The runtimes were: 0 00:00:00 02:24:00 04:48:00 07:12:00 09:36:00 12:00:00 Test Total Throughput Throughput Runtime [hh:mm:ss] runtime (Bills per second) (BITs per second) HDD 17:49:25 39 38,572 ET 11:27:20 61 60,015It is worth noting that there was a much Figure 15. Scalability with Respect to Concurrent Jobshigher throughput when the ET configu- Scaling behavior billingration was used, compared to HDD. The 80.00number of objects processed scaled in Scaling behavior invoicing 70.00a linear way with the runtime of the job, Linear (Scaling behavior invoicing) 60.00and there was no measurable degrada- 50.00 Bills per second 40.00tion in throughput. 30.00 20.00 10.00Another important aspect was the lin- 0.00 0 10 20 30 40 50ear scalability according to the number Concurrent jobsof concurrent jobs. This was requiredto increase the throughput wheneverneeded. Figure 15 shows that thisrequirement was fulfilled in this case. 29
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioInvoicingInvoicing was tested with HDD and Easy As Figure 16 shows, the application scaled with the number of objects processed.Tier for the same reasons as stated inthe billing section. The resulting run- Figure 16. Scalability of Invoicing Runtimes for each scheme were as follows: Scalability objects processed ET Test Total runtime Throughput 3,000,000 (invoices per second) HDD 0:35:49 1,163 2,500,000 Contract Accounts Invoiced ET 0:35:27 1,175 2,000,000 1,500,000 1,000,000As expected, the runtimes with Easy 500,000Tier were slightly better than those with 0HDD only. The small difference here – 00:00:00 00:07:12 00:14:24 00:21:36 00:28:48 00:36:00compared to the two other steps – can Runtime [hh:mm:ss]be explained by the fact that the billingdocuments and the resulting invoic-ing documents – compared to the size To reach still higher throughput figures, it was important that the number ofof all the BITs together – were rather invoices created per second scaled with the number of concurrent jobs. Figure 17small. Therefore, a lot of the informa- shows how this was the case.tion required for invoicing was verylikely still in the cache of the database Figure 17. Scalability of Invoicing with Respect to Concurrent Jobssystem; as a result, the different access Scalability behavior invoicingtimes between HDDs and SSDs are notimportant here. 700.00 Scaling behavior invoicing 600.00 Invoices per second Linear (Scaling behavior invoicing) 500.00 400.00 300.00 200.00 100.00 0.00 0 10 20 30 40 Concurrent jobs 30
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioTotal Runtime Test Runtime (ET) Runtime Runtime (HDD) factors The three runtime results totaled Upload 04:18:50 7 04:37:0016:21:37 hours. IBM Easy Tier reduced Billing 11:27:20 19 17:49:25the total runtime by 6:40:37 hours or Invoicing 00:35:27 1 00:35:4929%. ∑ 16:21:37 27 23:02:14Scenario 2 same activity wants to access the same response time could not be reducedUpload was run during billing and data page, it has to wait. But if there – this is a limitation of the 1 GB architec-invoicing to prove this was a possible are fewer jobs, the likelihood of a wait ture. Instead, a 10 GB Ethernet networkoption. This scenario was measured goes down and wait time is reduced. is recommended.only with the IBM Easy Tier option, and As activity B only wants to read fromthe results were as follows: the data blocks, it is not affected by the Storwize V7000 write lock. So the overall throughput is The proof of concept demonstrated Test Runtime (ET) higher. that the IBM Storwize V7000 is capable Billing 13:23:47 Invoicing 01:53:09 of both handling the given workload ∑ 15:16:58 Best Practices and Conclusions and achieving all KPIs. The detailed System X5 and SLES 11 illustrations of performance data in thisUpload is not included in this table as During the proof of concept, the IBM section show that when the Storwizeit ran concurrently with the two other System X5 servers that were used V7000 system was under load, notactivities. It required less time to fin- turned out not to be the limiting fac- all hot data could be placed on SSDsish and didn’t influence the total time tor. In fact, the processor and memory with the configuration used (24 SSDs).required for the processing. utilization was typically in the area of For the given KPIs, the team recom- between 60% and 70% for the ETG mended a total of three SSD MDGs forCompared with scenario 1, scenario 2 database server. Most likely, an increase SAPDATA, in a RAID 5, 7+1 configura-needed one hour less – due to reduced in processor capacity or memory would tion, leading to 26 SSDs, including twoconcurrency of parallel jobs of the same not have led to an additional, significant spares.activity. Usually one activity writes to reduction in runtime.a certain set of tables, while the otheractivity reads from them. For example, According to the nmon data, the 1 GBassuming activity A wants to write to a Ethernet link of the database servertable, it has to set locks on certain enti- was almost at the limit of its bandwidth.ties like data pages. If another job of the By bonding two Ethernet cards, the IP 31
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIf the team had required further runtime The team used the I/O simulation tool In addition, Storwize V7000 storagereduction, they would have had to install ndisk before the proof of concept was virtualization functionally allowedmore SSDs – for example: four MDisk started – to verify this performance changing the physical data placementarrays with a RAID 5, 7+1 configura- assumption about the Storwize V7000 of a VDisk (LUN), while keeping thetion, plus two spare drives, leading to system. (For more information about the VDisk (LUN) online. This was done with34 SSDs in total. It is not necessary to nstress tool kit and ndisk, see just a single command.place LOG files on SSDs. HDDs would www.ibm.com/developerworks/wikisdo the work just as well if the team /display/WikiPtype/nstress.) Detailed Performance Resultswere to recommend an HDD RAID 10, The following section shows the detailed8+8 configuration on a separate MDG. Value of Storwize V7000 Virtualization performance data for the billing run. InHowever, during the performance test, and Easy Tier order to display more detailed perfor-the team was not able to use this con- The combination of virtualization and mance data, the information is drawnfiguration, because all available HDDs Easy Tier functionality of Storwize from only a 20-minute time span –were needed to build the required V7000 eased storage administration selected from a total of more thancapacity of 50 TB (online DB and significantly. After setting up the storage 10 hours.backup). pools, the team needed to tune nothing else. Also, the combination of these two HDD-Only Run “Billing” functionalities eliminated, by design, The following figures show the per- the possibility of storage performance formance data for the HDD-only run bottlenecks. (SAPDATA placed on HDD). 32
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioFigure 18. DB Server File Systems I/O Performance File system I/O 180MB/s 160 140 120 100 SAPDATA read 80 SAPDATA write SAPDATA total 60 LOG write 40 20 0 00:00:24 00:01:24 00:02:24 00:03:24 00:04:24 00:05:24 00:06:24 00:07:24 00:08:24 00:09:24 00:10:24 00:11:24 00:12:24 00:13:24 00:14:24 00:15:24 00:16:24 00:17:24 00:18:24 00:19:24Figure 19. DB Server CPU Utilization CPU workload 100 90 80 70 % utilization Idle% 60 Wait% 50 40 Sys% 30 User% 20 10 0Figure 20. DB Server Ethernet Performance 80 Ethernet I/O MB/s 70 60 50 IP write 40 IP read 30 20 10 0 33
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario Figure 21. V7000 SAPDATA VDisk Performance SAPData VDISK throughput 6.0 MB/s 5.0 4.0 SAPDATA10 read SAPDATA10 write 3.0 SAPDATA11 read SAPDATA11 write SAPDATA12 read 2.0 SAPDATA12 write 1.0 0.0 Figure 22. V7000 Log VDisk Performance LOG VDISK throughput 60.0MB/s LOG write 50.0 40.0 30.0 20.0 10.0 0.0 Figure 23. V7000 MDisk I/O Performance MDISK throughput 60,0 50,0 SSD MD LOG write HDD MD1 read 40,0 HDD MD1 writeMB/s 30,0 HDD MD2 read HDD MD2 write 20,0 HDD MD3 read 10,0 HDD MD3 write 0,0 34
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario Figure 24. V7000 VDisk I/O Response Times VDISK I/O response times 35msec 30 LOG write 25 SAPDATA10 read 20 SAPDATA10 write SAPDATA11 read 15 SAPDATA11 write 10 SAPDATA12 read SAPDATA12 write 5 0 Figure 25. V7000 MDisk I/O Response Times MDISK I/O response times 100msec 90 80 SDD MD LOG write 70 HDD MD1 read 60 HDD MD1 write 50 40 HDD MD2 read 30 HDD MD2 write 20 HDD MD3 read 10 HDD MD3 write 0The write and read processes compete OS or measured through svcmon onagainst the HDD resource. If the non- the storage system.volatile RAM (NV RAM) gets filled up, thewrite gains a higher priority, resulting in So as not to overload the number ofa lower read performance. It is no sur- graphics, the figures above includedprise that all performance figures show just three printouts for SAPDATAthe same utilization characteristics, VDisks; likewise, only 3 SAPDATAeither measured through nmon on the MDisks out of 11 are included. 35
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioEasy Tier Run “Billing” Figure 26. DB Server File Systems I/O PerformanceThe following figures show the perfor- File systems I/Omance data for the Easy Tier-only run 300 MB/s(SAPDATA placed on HDD and SSD). 250 200 150 SAPDATA read SAPDATA write 100 SAPDATA total LOG write 50 0 21:00:52 21:01:52 21:02:52 21:03:52 21:04:52 21:05:52 21:06:52 21:07:52 21:08:52 21:09:52 21:10:52 21:11:52 21:12:52 21:13:52 21:14:52 21:15:52 21:16:52 21:17:52 21:18:53 21:19:53 Figure 27. DB Server CPU Workload CPU workload 100 % uƟlizaƟon 90 80 70 Idle% 60 Wait% 50 40 Sys% 30 User% 20 10 0 Figure 28. DB Server Ethernet I/O Performance Ethernet I/O 100 MB/s 90 80 70 60 IP write 50 IP read 40 30 20 10 0 36
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioFigure 29. V7000 SAPDATA VDisk Performance SAPDATA VDisk throughput MB/s 6 5 SAPDATA 10 read 4 SAPDATA10 write SAPDATA 11 read 3 SAPDATA11 write 2 SAPDATA 12 read SAPDATA12 write 1 0Figure 30. V7000 Log VDisk Performance LOG Disk throughput 80 MB/s 70 LOG write 60 50 40 30 20 10 0Figure 31. V7000 MDisk Performance MDisk throughput 80 MB/s SSD LOG write 70 SSD MD1 read SSD MD1 write 60 SSD MD2 read 50 SSD MD2 write SSD MD3 read 40 SSD MD3 write HDD MD4 read 30 HDD MD4 write HDD MD5 read 20 HDD MD5 write HDD MD6 read 10 HDD MD6 write 0 37
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIn contrast to the HDD billing run, the Figure 32. V7000 VDisk I/O response timesresource utilization of Easy Tier was VDisk response times 7almost constant and did not waver. msec 6 LOG writeThe read and write processes were 5 SAPDATA10 readnot competing against the storage SAPDATA10 write 4 SAPDATA11 readresource. Compared to HDD, SSD 3 SAPDATA11 writetechnology had no need to spend time 2 SAPDATA12 readpositioning the head over the requested 1 SAPDATA12 writetrack. Regardless of what data block 0needed to be accessed, the dataaccess time was always constant withSSD – less than a millisecond for both Figure 33. V7000 MDisk I/O response timesreads and writes. MDisk I/O response times 60If data was stored on SSD, then no stor-age cache was involved, including NV SSD LOG write 50RAM. Data was read and written to SSD SSD MD1 read SSD MD1 writedirectly, leading to more available cache 40 SSD MD2 readcapacity for the remaining data on HDD. SSD MD2 writeAs a result, the resource utilization over SSD MD3 read msec 30 SSD MD3 writetime was constant, and I/O throughput HDD MD4 readincreased from approximately 140 MB HDD MD4 write 20for HDD-only to 250 MB for the Easy HDD MD5 read HDD MD5 writeTier configuration. There was also a sig- HDD MD6 read 10nificant reduction in I/O response time HDD MD6 writefor VDisks and MDisks. 0From a DB2 perspective, the bufferpool performance of the average totalphysical read time improved impres-sively by over 60%, due to the better I/Othroughput and response time of theEasy Tier implementation, compared toHDD. 38
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioThis report focuses on the buffer pool Figure 34. DB2 Buffer Pool Physical Readsread times because they resulted in (in Milliseconds) During Billing Runs on HDD and Easy Tierread requests directly to disk, whereas DB2 buffer pool physical reads (msec) – billing runsthe write requests were handled by thecache of the storage subsystem, which avg_total_reads avg_async_reads avg_sync_readsstayed the same for the HDD and Easy 30.00Tier runs. Thus, the write time did not 24.96 25.04 25.00vary among the tests, whereas the read 21.41times did so heavily, as shown in Figure 34. 20.00 msec 15.00MDisk Heat Distribution 12.53 9.97The following table shows the heat dis- 10.00tribution of the SAPDATA MDisks, and 5.00 1.19the amount of data that was “hot” and 0 Billing – HDD Billing – ET“cool.” This data was collected duringthe ET billing run with the IBM StorageTier Advisor Tool. (For more information,see www-304.ibm.com/support Volume ID Configured size Capacity on SSD Heat distribution 0x0027 665.8 G 168.3 G 481.8 G 184.0 G/docview.wss?uid=ssg1S4000935.) 0x0026 649.5 G 165.8 G 469.0 G 180.5 G 0x0025 647.5 G 161.5 G 470.0 G 177.5 G 0x0024 648.3 G 171.0 G 461.8 G 186.5 G 0x0023 649.8 G 163.3 G 471.0 G 178.8 G 0x0022 651.3 G 170.0 G 465.5 G 185.8 G 0x0021 647.5 G 156.5 G 475.8 G 171.8 G 0x0020 652.0 G 181.8 G 455.3 G 196.8 G 0x001f 653.5 G 162.5 G 475.5 G 178.0 G 0x001e 647.5 G 168.8 G 463.5 G 184.0 G 0x001d 647.5 G 163.3 G 469.3 G 178.3 G 0x001c 650.8 G 171.3 G 463.5 G 187.3 G 0x001b 647.5 G 161.8 G 469.0 G 178.5 G 0x001a 650.3 G 163.8 G 472.0 G 178.3 G 0x0019 649.5 G 148.8 G 485.0 G 164.5 G 0x0018 647.5 G 171.8 G 459.3 G 188.3 G 0x0017 647.5 G 162.0 G 469.5 G 178.0 G 0x0016 650.3 G 166.8 G 467.8 G 182.5 G 39
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario 0x0015 648.3 G 162.0 G 470.3 G 178.0 G 0x0014 649.3 G 169.8 G 465.3 G 184.0 G 0x0013 649.0 G 153.3 G 480.8 G 168.3 G 0x0012 649.0 G 174.5 G 459.3 G 189.8 G 0x0011 649.0 G 173.5 G 460.5 G 188.5 G 0x0010 647.5 G 180.3 G 450.3 G 197.3 G 0x000f 647.5 G 174.0 G 459.3 G 188.3 G 0x000e 647.5 G 175.3 G 457.5 G 190.0 G 0x000d 649.8 G 171.3 G 463.0 G 186.8 G 0x000c 647.5 G 110.5 G 522.0 G 125.5 G 0x000b 647.5 G 165.0 G 466.3 G 181.3 G 0x000a 647.5 G 165.8 G 465.8 G 181.8 G 0x0009 647.5 G 164.8 G 467.5 G 180.0 G 0x0008 651.0 G 63.8 G 570.5 G 80.5 GDB2 Configuration Parametersfor Run ET4DB2 Registry (db2set -all)[e] DB2DBDFT=ETL[i] DB2_RESTORE_GRANT_ADMIN_AUTHORITIES=YES [DB2_WORKLOAD][i] DB2_BLOCKING_WITHHOLD_LOBLOCATOR=NO [DB2_WORKLOAD][i] DB2_AGENT_CACHING_FMP=OFF [DB2_WORKLOAD][i] DB2_TRUST_MDC_BLOCK_FULL_HINT=YES [DB2_WORKLOAD][i] DB2_CREATE_INDEX_COLLECT_STATS=YES [DB2_WORKLOAD][i] DB2_ATS_ENABLE=YES [DB2_WORKLOAD][i] DB2_RESTRICT_DDF=YES [DB2_WORKLOAD][i] DB2_DUMP_SECTION_ENV=YES [DB2_WORKLOAD][i] DB2_OPT_MAX_TEMP_SIZE=10240 [DB2_WORKLOAD][i] DB2_USE_FAST_PREALLOCATION=OFF[i] DB2_WORKLOAD=SAP[i] DB2_TRUNCATE_REUSESTORAGE=IMPORT [DB2_WORKLOAD][i] DB2_MDC_ROLLOUT=DEFER [DB2_WORKLOAD][i] DB2_ATM_CMD_LINE_ARGS=-include-manual-tables [DB2_WORKLOAD][i] DB2_SKIPINSERTED=YES [DB2_WORKLOAD][i] DB2_VIEW_REOPT_VALUES=YES [DB2_WORKLOAD][i] DB2_OBJECT_TABLE_ENTRIES=65532 [DB2_WORKLOAD][i] DB2_OPTPROFILE=YES [DB2_WORKLOAD][i] DB2_USE_ALTERNATE_PAGE_CLEANING=ON[i] DB2_IMPLICIT_UNICODE=YES [DB2_WORKLOAD][i] DB2STMM=APPLY_HEURISTICS:YES [DB2_WORKLOAD][i] DB2_INLIST_TO_NLJN=YES [DB2_WORKLOAD] 40
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario[i] DB2_MINIMIZE_LISTPREFETCH=YES [DB2_WORKLOAD][i] DB2_REDUCED_OPTIMIZATION=4,INDEX,JOIN,NO_TQ_FACT,NO_HSJN_BUILD_FACT, STARJN_CARD_SKEW,NO_SORT_MGJOIN,CART OFF,CAP OFF [DB2_WORKLOAD][i] DB2NOTIFYVERBOSE=YES [DB2_WORKLOAD][i] DB2TERRITORY=1[i] DB2_INTERESTING_KEYS=YES [DB2_WORKLOAD][i] DB2_EVALUNCOMMITTED=YES [DB2_WORKLOAD][i] DB2_LOGGER_NON_BUFFERED_IO=ON[i] DB2_EXTENDED_OPTIMIZATION=NLJOIN_KEYCARD,IXOR [DB2_WORKLOAD][i] DB2_ANTIJOIN=EXTEND [DB2_WORKLOAD][i] DB2COMPOPT=327685,131776 [DB2_WORKLOAD][i] DB2ATLD_PORTS=60000:65000[i] DB2ENVLIST=INSTHOME SAPSYSTEMNAME dbs_db6_schema DIR_LIBRARY LD_LIBRARY_PATH[i] DB2COMM=TCPIP [DB2_WORKLOAD][i] DB2_PARALLEL_IO=*:2[g] DB2FCMCOMM=TCPIP4[g] DB2SYSTEM=coe-he-16[g] DB2INSTDEF=db2etlDatabase Manager Configuration(db2 get dbm cfg)Database Manager ConfigurationNode type Enterprise Server EditionDatabase manager configuration release level 0x0d00CPU speed (millisec/instruction) (CPUSPEED) 2,68E-01Communications bandwidth (MB/sec) (COMM_BANDWIDTH) 1,00E+08Max number of concurrently active databases (NUMDB) 8Federated Database System Support (FEDERATED) NOTransaction processor monitor name (TP_MON_NAME)Default charge-back account (DFT_ACCOUNT_STR)Java Development Kit installation path (JDK_PATH) /db2/db2etl/sqllib/Diagnostic error capture level (DIAGLEVEL) 3Notify Level (NOTIFYLEVEL) 3Diagnostic data directory path (DIAGPATH) /db2/ETL/db2dumpSize of rotating db2diag & notify logs (MB) DIAGSIZE) 1000Default database monitor switches Buffer pool (DFT_MON_BUFPOOL) ON Lock (DFT_MON_LOCK) ON Sort (DFT_MON_SORT) ON Statement (DFT_MON_STMT) ON Table (DFT_MON_TABLE) ON Timestamp (DFT_MON_TIMESTAMP) ON Unit of work (DFT_MON_UOW) ONMonitor health of instance and databases (HEALTH_MON) OFFSYSADM group name (SYSADM_GROUP) DBETLADMSYSCTRL group name (SYSCTRL_GROUP) DBETLCTL 41
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioSYSMAINT group name (SYSMAINT_GROUP) DBETLMNTSYSMON group name (SYSMON_GROUP) DBETLMONClient Userid-Password Plugin (CLNT_PW_PLUGIN)Client Kerberos Plugin (CLNT_KRB_PLUGIN)Group Plugin (GROUP_PLUGIN)GSS Plugin for Local Authorization (LOCAL_GSSPLUGIN)Server Plugin Mode (SRV_PLUGIN_MODE) UNFENCEDServer List of GSS Plugins (SRVCON_GSSPLUGIN_LIST)Server Userid-Password Plugin (SRVCON_PW_PLUGIN)Server Connection Authentication (SRVCON_AUTH) NOT_SPECIFIEDCluster manager (CLUSTER_MGR)Database manager authentication (AUTHENTICATION) SERVER_ENCRYPTAlternate authentication (ALTERNATE_AUTH_ENC) NOT_SPECIFIEDCataloging allowed without authority (CATALOG_NOAUTH) NOTrust all clients (TRUST_ALLCLNTS) YESTrusted client authentication (TRUST_CLNTAUTH) CLIENTBypass federated authentication (FED_NOAUTH) NODefault database path (DFTDBPATH) /db2/ETLDatabase monitor heap size (4KB) (MON_HEAP_SZ) AUTOMATIC(90)Java Virtual Machine heap size (4KB) (JAVA_HEAP_SZ) 2048Audit buffer size (4KB) (AUDIT_BUF_SZ) 0Size of instance shared memory (4KB) (INSTANCE_MEMORY) 8192000Backup buffer default size (4KB) (BACKBUFSZ) 1024Restore buffer default size (4KB) (RESTBUFSZ) 1024Agent stack size (AGENT_STACK_SZ) 1024Sort heap threshold (4KB) (SHEAPTHRES) 0Directory cache support (DIR_CACHE) NOApplication support layer heap size (4KB) (ASLHEAPSZ) 16Max requester I/O block size (bytes) (RQRIOBLK) 65000Query heap size (4KB) (QUERY_HEAP_SZ) 1000Workload impact by throttled utilities(UTIL_IMPACT_LIM) 10Priority of agents (AGENTPRI) SYSTEMAgent pool size (NUM_POOLAGENTS) AUTOMATIC(100)Initial number of agents in pool (NUM_INITAGENTS) 5Max number of coordinating agents (MAX_COORDAGENTS) AUTOMATIC(200)Max number of client connections (MAX_CONNECTIONS) AUTOMATIC(MAX_ COORDAGENTS)Keep fenced process (KEEPFENCED) NONumber of pooled fenced processes (FENCED_POOL) AUTOMATIC(MAX_ COORDAGENTS)Initial number of fenced processes (NUM_INITFENCED) 0Index re-creation time and redo index build (INDEXREC) RESTARTTransaction manager database name (TM_DATABASE) 1ST_CONNTransaction resync interval (sec) (RESYNC_INTERVAL) 180SPM name (SPM_NAME)SPM log size (SPM_LOG_FILE_SZ) 256SPM resync agent limit (SPM_MAX_RESYNC) 20SPM log path (SPM_LOG_PATH) 42
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioTCP/IP Service name (SVCENAME) sapdb2ETLDiscovery mode (DISCOVER) SEARCHDiscover server instance (DISCOVER_INST) ENABLESSL server keydb file (SSL_SVR_KEYDB)SSL server stash file (SSL_SVR_STASH)SSL server certificate label (SSL_SVR_LABEL)SSL service name (SSL_SVCENAME)SSL cipher specs (SSL_CIPHERSPECS)SSL versions (SSL_VERSIONS)SSL client keydb file (SSL_CLNT_KEYDB)SSL client stash file (SSL_CLNT_STASH)Maximum query degree of parallelism (MAX_QUERYDEGREE) ANYEnable intra-partition parallelism (INTRA_PARALLEL) NOMaximum Asynchronous TQs per query (FEDERATED_ASYNC) 0No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) AUTOMATIC(4096)No. of int. communication channels (FCM_NUM_CHANNELS) AUTOMATIC(2048)Node connection elapse time (sec) (CONN_ELAPSE) 10Max number of node connection retries (MAX_CONNRETRIES) 5Max time difference between nodes (min) (MAX_TIME_DIFF) 60db2start/db2stop timeout (min) (START_STOP_TIME) 10Database Configuration (db2 get db cfg for etl)Database Configuration for Database ETLDatabase configuration release level 0x0d00Database release level 0x0d00Database territory en_USDatabase code page 1208Database code set UTF-8Database country/region code 1Database collating sequence IDENTITY_16BITNumber compatibility OFFVarchar2 compatibility OFFDate compatibility OFFDatabase page size 16384Dynamic SQL Query management (DYN_QUERY_MGMT) DISABLEStatement concentrator (STMT_CONC) OFFDiscovery support for this database (DISCOVER_DB) ENABLERestrict access NODefault query optimization class (DFT_QUERYOPT) 5Degree of parallelism (DFT_DEGREE) ANYContinue upon arithmetic exceptions (DFT_SQLMATHWARN) NODefault refresh age (DFT_REFRESH_AGE) 0Default maintained table types for opt (DFT_MTTB_TYPES) SYSTEMNumber of frequent values retained (NUM_FREQVALUES) 10Number of quantiles retained (NUM_QUANTILES) 20Decimal floating point rounding mode (DECFLT_ROUNDING) ROUND_HALF_EVENBackup pending NO 43
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioAll committed transactions have been written to disk NORollforward pending NORestore pending NOMulti-page file allocation enabled YESLog retain for recovery status NOUser exit for logging status NOSelf tuning memory (SELF_TUNING_MEM) ONSize of database shared memory (4KB) (DATABASE_MEMORY) AUTOMATIC(6993738)Database memory threshold (DB_MEM_THRESH) 10Max storage for lock list (4KB) (LOCKLIST) AUTOMATIC(422112)Percent. of lock lists per application (MAXLOCKS) AUTOMATIC(97)Package cache size (4KB) (PCKCACHESZ) AUTOMATIC(128005)Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) AUTOMATIC(113636)Sort list heap (4KB) (SORTHEAP) AUTOMATIC(22727)Database heap (4KB) (DBHEAP) AUTOMATIC(3745)Catalog cache size (4KB) (CATALOGCACHE_SZ) 2560Log buffer size (4KB) (LOGBUFSZ) 16384Utilities heap size (4KB) (UTIL_HEAP_SZ) 10000Buffer pool size (pages) (BUFFPAGE) 10000SQL statement heap (4KB) (STMTHEAP) AUTOMATIC(8192)Default application heap (4KB) (APPLHEAPSZ) AUTOMATIC(256)Application Memory Size (4KB) (APPL_MEMORY) AUTOMATIC(40000)Statistics heap size (4KB) (STAT_HEAP_SZ) AUTOMATIC(4384)Interval for checking deadlock (ms) (DLCHKTIME) 10000Lock timeout (sec) (LOCKTIMEOUT) 3600Changed pages threshold (CHNGPGS_THRESH) 40Number of asynchronous page cleaners (NUM_IOCLEANERS) AUTOMATIC(31)Number of I/O servers (NUM_IOSERVERS) 64Index sort flag (INDEXSORT) YESSequential detect flag (SEQDETECT) YESDefault prefetch size (pages) (DFT_PREFETCH_SZ) AUTOMATICTrack modified pages (TRACKMOD) ONDefault number of containers 1Default tablespace extentsize (pages) (DFT_EXTENT_SZ) 2Max number of active applications (MAXAPPLS) AUTOMATIC(408)Average number of active applications (AVG_APPLS) AUTOMATIC(3)Max DB files open per application (MAXFILOP) 61440Log file size (4KB) (LOGFILSIZ) 128000Number of primary log files (LOGPRIMARY) 150Number of secondary log files (LOGSECOND) 50Path to log files /db2/ETL/log_dir /NODE0000/Overflow log path (OVERFLOWLOGPATH)Mirror log path (MIRRORLOGPATH)First active log fileBlock log on disk full (BLK_LOG_DSK_FUL) YESBlock non logged operations (BLOCKNONLOGGED) NOPercent max primary log space by transaction (MAX_LOG) 0 44
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioNum. of active log files for 1 active UOW(NUM_LOG_SPAN) 50Group commit count (MINCOMMIT) 1Percent log file reclaimed before soft chckpt (SOFTMAX) 300Log retain for recovery enabled (LOGRETAIN) OFFUser exit for logging enabled (USEREXIT) OFFHADR database role STANDARDHADR local host name (HADR_LOCAL_HOST)HADR local service name (HADR_LOCAL_SVC)HADR remote host name (HADR_REMOTE_HOST)HADR remote service name (HADR_REMOTE_SVC)HADR instance name of remote server(HADR_REMOTE_INST)HADR timeout value (HADR_TIMEOUT) 120HADR log write synchronization mode (HADR_SYNCMODE) NEARSYNCHADR peer window duration (seconds) (HADR_PEER_WINDOW) 0First log archive method (LOGARCHMETH1) OFFOptions for logarchmeth1 (LOGARCHOPT1)Second log archive method (LOGARCHMETH2) OFFOptions for logarchmeth2 (LOGARCHOPT2)Failover log archive path (FAILARCHPATH)Number of log archive retries on error (NUMARCHRETRY) 5Log archive retry Delay (secs) (ARCHRETRYDELAY) 20Vendor options (VENDOROPT)Auto restart enabled (AUTORESTART) ONIndex re-creation time and redo index build (INDEXREC) SYSTEM (RESTART)Log pages during index build (LOGINDEXBUILD) OFFDefault number of loadrec sessions (DFT_LOADREC_SES) 1Number of database backups to retain (NUM_DB_BACKUPS) 12Recovery history retention (days) (REC_HIS_RETENTN) 60Auto deletion of recovery objects (AUTO_DEL_REC_OBJ) OFFTSM management class (TSM_MGMTCLASS)TSM node name (TSM_NODENAME)TSM owner (TSM_OWNER)TSM password (TSM_PASSWORD)Automatic maintenance (AUTO_MAINT) ON Automatic database backup (AUTO_DB_BACKUP) OFF Automatic table maintenance (AUTO_TBL_MAINT) ON Automatic runstats (AUTO_RUNSTATS) ON Automatic statement statistics ON (AUTO_STMT_STATS) Automatic statistics profiling (AUTO_STATS_PROF) OFF Automatic profile updates (AUTO_PROF_UPD) OFF Automatic reorganization (AUTO_REORG) OFFAuto-Revalidation (AUTO_REVAL) DEFERREDCurrently Committed (CUR_COMMIT) ONCHAR output with DECIMAL input (DEC_TO_CHAR_FMT) NEWEnable XML Character operations (ENABLE_XMLCHAR) YESWLM Collection Interval (minutes) (WLM_COLLECT_INT) 0 45
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications Scenario Monitor Collect Settings Request metrics (MON_REQ_METRICS) BASE Activity metrics (MON_ACT_METRICS) NONE Object metrics (MON_OBJ_METRICS) BASE Unit of work events (MON_UOW_DATA) NONE Lock timeout events (MON_LOCKTIMEOUT) WITHOUT_HIST Deadlock events (MON_DEADLOCK) WITHOUT_HIST Lock wait events (MON_LOCKWAIT) NONE Lock wait event threshold (MON_LW_THRESH) 5000000 Number of package list entries (MON_PKGLIST_SZ) 32 Lock event notification level (MON_LCK_MSG_LVL) 1 SMTP Server (SMTP_SERVER) SQL conditional compilation flags (SQL_CCFLAGS) Section actuals setting (SECTION_ACTUALS) NONE Connect procedure (CONNECT_PROC)SAP Convergent Invoicing Key TablesLoad tables: /1FE/0LT0xxITBilling tables: /1FE/0LT0xxIT00Other large tables DFKKCOH, DFKKCOHI DFKKINVDOC_H, DFKKINVDOC_I, DFKKINVDOC_P, DFKKINVDOC_S DFKKKO, DFKKOP, DFKKOPK DFKKSUMC DFKKINVBILL_H, DFKKINVBILL_I, DFKKINVBILL_S 46
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioRow Compression Results for the Largest SAP Convergent Invoicing Tables Table + index Table + index – compressed Table compressed Index compressed – uncompressed Total Total Total Data Data Data Index Index Index size savings savings size saved saved size saved saved Table name (KB) (GB) (%) (GB) (GB) (%) (GB) (GB) (%)/1FE/0LT022IT 1.421 829 592 42 582 313 35 247 279 53/1FE/0LT084IT00 612 252 361 59 213 319 60 39 42 52/1FE/0LT094IT00 607 249 357 59 211 316 60 39 42 52/1FE/0LT134IT00 605 249 357 59 210 315 60 39 41 52/1FE/0LT104IT00 605 249 356 59 210 315 60 39 42 52/1FE/0LT154IT00 602 247 354 59 209 314 60 38 41 52/1FE/0LT004IT00 614 247 367 60 208 325 61 39 42 52/1FE/0LT144IT00 612 246 366 60 208 325 61 38 41 52/1FE/0LT184IT00 610 245 365 60 206 323 61 39 42 52/1FE/0LT114IT00 609 245 364 60 206 323 61 39 41 52/1FE/0LT194IT00 607 244 363 60 206 321 61 39 42 52/1FE/0LT124IT00 605 243 361 60 205 320 61 39 41 52/1FE/0LT012IT 903 241 662 73 137 391 74 104 271 72/1FE/0LT164IT00 611 240 371 61 202 329 62 38 42 52/1FE/0LT174IT00 614 236 378 62 198 336 63 39 42 52/1FE/0LT024IT00 614 236 378 62 197 336 63 39 42 52/1FE/0LT014IT00 612 225 387 63 186 346 65 39 42 52Sum 11,463 4,724 59 47
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioResourcesSAP for Telecommunications and SAP Convergent Invoicing:www.sap.com/industries/telecom/businessprocesses/invoicing/index.epxnmon Linux OS performance monitoring tool:www.ibm.com/developerworks/aix/library/au-analyze_aixsvcmon performance monitoring tool for Storwize V7000 / SVC:www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3177ndisk (nstress) I/O simulation tool:www.ibm.com/developerworks/wikis/display/WikiPtype/nstresswww.ibm.com/developerworks/mydeveloperworks/blogs/svcmon/?lang=jaIBM Storage Advisor Tool:www-304.ibm.com/support/docview.wss?uid=ssg1S4000935DB2 9.7 Information Center:http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jspDB2 9.7 Information Center (db2top):http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=%2Fcom.ibm.db2.luw.admin.cmd.doc%2Fdoc%2Fr0025222.htmlDB2 for Linux, UNIX, and Microsoft Windows on SAP Developer Network (SDN):www.sdn.sap.com/irj/sdn/db6IBM APAR IC76792:ibm.com/support/docview.wss?uid=swg21502430IBM Developer Works - Distributed DBA: Storage, I/O, and DB2(DB2_PARALLEL_IO):ibm.com/developerworks/data/library/dmmag/DBMag_2009_Issue1/DBMag_Issue109_DistributedDBA/index.html 48
  • SAP and IBM Demonstrate Capability of Handling High Billing Volume in a Telecommunications ScenarioIBM Developer Works – DB2 tuning tips for OLTP applications:ibm.com/developerworks/data/library/techarticle/anshum/0107anshum.html#logbuffersizeIBM E-Book SAP DBA Cockpit – Flight Plans for DB2 LUW Administrator:http://public.dhe.ibm.com/common/ssi/ecm/en/imm14052usen/IMM14052USEN.PDFArticle on SAP SDN: The Evolution of the Memory Model in IBM DB2 LUW byJohannes Heinrich:sdn.sap.com/irj/scn/index?rid=/library/uuid/b0aabcc9-afc1-2a10-5091-b5cda33036b0SAP Notes:Note 1351160 – DB6: Using DB2 9.7 with SAP SoftwareNote 1329179 – DB6: DB2 V9.7 Standard Parameter SettingsNote 1603507 – DB6: AdHoc Analysis Tool for DB6 49
  • 50
  • 51
  • © Copyright IBM Corporation 2012 . All rights Reserved. Some information addresses anticipated future © 2012 by SAP. All rights reserved. capabilities. Such information is not intended as a References in this document to IBM products or services SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, definitive statement of a commitment to specific levels of do not imply that IBM intends to make them available in SAP BusinessObjects Explorer, StreamWork, SAP HANA, performance, function or delivery schedules with respect every country. and other SAP products and services mentioned herein as to any future products. Such commitments are only well as their respective logos are trademarks or registered IBM, the IBM logo, and ibm.com are trademarks or made in IBM product announcements. The information is trademarks of SAP AG in Germany and other countries. registered trademarks of International Business Machines presented here to communicate IBM’s current investment Corporation in the United States, other countries, or both. and development activities as a good faith effort to help Business Objects and the Business Objects logo, If these and other IBM trademarked terms are marked on with our customers’ future planning. BusinessObjects, Crystal Reports, Crystal Decisions, their first occurrence in this information with a trademark Web Intelligence, Xcelsius, and other Business Objects Performance is based on measurements and projections symbol (® or ™), these symbols indicate U.S. registered products and services mentioned herein as well as their using standard IBM benchmarks in a controlled or common law trademarks owned by IBM at the time this respective logos are trademarks or registered trademarks environment. The actual throughput or performance information was published. Such trademarks may also be of Business Objects Software Ltd. Business Objects is an that any user will experience will vary depending upon registered or common law trademarks in other countries. SAP company. considerations such as the amount of multiprogramming A current list of IBM trademarks is available on the Web at in the user’s job stream, the I/O configuration, the storage Sybase and Adaptive Server, iAnywhere, Sybase 365, “Copyright and trademark information” at www.ibm.com/ configuration, and the workload processed. Therefore, no SQL Anywhere, and other Sybase products and services legal/copytrade.shtml. assurance can be given that an individual user will achieve mentioned herein as well as their respective logos are SAP, SAP NetWeaver , and SAP logos are trademarks or throughput or performance improvements equivalent to trademarks or registered trademarks of Sybase Inc. registered trademarks of SAP and/or its affiliates. the ratios stated here. Sybase is an SAP company. Java and all Java-based trademarks and logos are Photographs shown are of engineering prototypes. Crossgate, m@gic EDDY, B2B 360°, and B2B 360° trademarks or registered trademarks of Oracle and/or its Changes may be incorporated in production models. Services are registered trademarks of Crossgate AG affiliates. in Germany and other countries. Crossgate is an SAP Any references in this information to non-IBM Web sites company. Microsoft, Windows, Windows NT, and the Windows logo are provided for convenience only and do not in any are trademarks of Microsoft Corporation in the United manner serve as an endorsement of those Web sites. The All other product and service names mentioned are the States, other countries, or both. materials at those Web sites are not part of the materials trademarks of their respective companies. Data contained for this IBM product and use of those Web sites is at your in this document serves informational purposes only. Intel, Intel Inside (logos), MMX, and Pentium are trademarks own risk. National product specifications may vary. of Intel Corporation in the United States, other countries, or both. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated UNIX is a registered trademark of The Open Group in the [The following specific IBM trademarks appeared in the companies (“SAP Group”) for informational purposes United States and other countries. text of the original draft] only, without representation or warranty of any kind, and Linux is a trademark of Linus Torvalds in the United States, IBM System i® SAP Group shall not be liable for errors or omissions with other countries, or both. respect to the materials. The only warranties for SAP IBM System x® X5 server Group products and services are those that are set forth SET and the SET Logo are trademarks owned by SET IBM System z® in the express warranty statements accompanying such Secure Electronic Transaction LLC. products and services, if any. Nothing herein should be IBM Storwize® V7000 Other company, product, or service names may be construed as constituting an additional warranty. trademarks or service marks of others. IBM DB2® Information is provided “AS IS” without warranty of any SUSE® Linux® Enterprise Server kind. IBM Easy Tier® All customer examples described are presented as Enterprise X-Architecture® illustrations of how those customers have used IBM products and the results they may have achieved. Actual IBM System Storage® SAN Volume Controller environmental costs and performance characteristics IBM DB2 pureScale® may vary by customer. FlashCopy® Information concerning non-IBM products was obtained from a supplier of these products, published IBM BladeCenter® announcement material, or other publicly available Tivoli® sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.