This document discusses persistent IU pacing, also known as Extended Distance FICON. It begins with legal disclaimers and an abstract. Then it provides an agenda and discusses the basics of persistent IU pacing, including what it is and is not. It explains sequences, open exchanges, and information units in FICON protocols. It discusses IU pacing and how persistent IU pacing functions to help avoid performance degradation over long distances by implementing a new pacing protocol. It notes the hardware requirements and discusses how IU pacing and persistent IU pacing work to control data flows.

1. Demystifying Extended Distance FICON (aka Persistent IU Pacing) Dr. Steve Guendert Principle Engineer/Global Solutions Architect Brocade Communications

2. Legal Disclaimer All or some of the products detailed in this presentation may still be under development and certain specifications, including but not limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in this presentation. NOTHING IN THIS PRESENTATION SHALL BE DEEMED TO CREATE A WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT OF THIRD-PARTY RIGHTS WITH RESPECT TO ANY PRODUCTS AND SERVICES REFERENCED HEREIN. Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 2 8/16/2011

3. Abstract CMG 2011 paper submission In 2008 IBM announced IU pacing enhancements that allow customers to deploy z/OS Global Mirror (zGM) over long distances without a significant impact to performance. This is more commonly known by the marketing term Extended Distance FICON. The more technically accurate term as defined in the FC-SB3/4 standards is persistent IU pacing. How this functionality works, and what it actually does for the end user has remained a mystery, thanks in large part to the marketing term used. This paper/presentation will demystify how the technology works, and how it can benefit the end user. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 3 8/16/2011

4. Agenda Basics: What is Extended Distance FICON (and what it is not)? Sequences, Open Exchanges, and Information Units (oh my!) IU Pacing and Persistent IU Pacing How does it really function? Performance Implications Do I still need channel extension? Concluding thoughts and observations © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 4 8/16/2011

5. Basics of Extended Distance FICON 8/16/2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 5

6. Extended Distance FICON The rumor of the demise of channel extension is greatly exaggerated Extended Distance FICON is an enhancement to the industry-standard FICON architecture (FC-SB-3) that can help avoid degradation of performance at extended distances by implementing a new protocol for "persistent" Information Unit (IU) pacing Some storage vendors will position the Extended Distance FICON feature as eliminating the need for “Those expensive channel extenders” 8/16/2011 6 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

7. What is Extended Distance FICON? Announced by IBM 26 February 2008 It is more of a marketing term than a technological term. IBM announced IU pacing enhancements for z/OS Global Mirror (zGM) The announcement really announced that the z10 and DS8000 supported the new FC-SB3 amendments for Persistent IU Pacing. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 7 8/16/2011

8. What Extended Distance FICON is (and is not) It is a performance enhancement mechanism for FICON I/O traffic traversing “long” distances. It really does not increase the distance the FICON I/O traffic can go. Distance capabilities depend on other things Buffer to buffer credits © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 8 8/16/2011

9. Hardware requirements System z196 System z10 Driver 73 with MCL F85898.003 Driver 76 with MCL N10948.001 DASD IBM DS 8000/8700/8800 EMC VMAX with Enginuity 5874 Q4 service release HDS USP-V © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 9 8/16/2011

10. Sequences, Open Exchanges, and Information Units (Oh my!) © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 10 8/16/2011

11. The FICON Upper Layer Protocol (ULP) Provides 3 resources for managing the underlying physical credits used to transmit the actual data in an I/O. FC-SB3/4 Sequences Exchanges (OEs) Information Units (IUs) A credit is a 2K physical data transmission buffer More commonly known as buffer to buffer credits (BB credits) Used for flow control between two optically adjacent fibre channel ports. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 11 8/16/2011

12. Sequences The FC-PH standard provides a mechanism that assembles sub-blocks of data contained in the payloads of one or more frames into a single data block when receiving frames. Each FC-4 ULP defines what the contents of the sequences are for its specific ULP functions. FICON: IUs © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 12 8/16/2011

13. Information Units (IUs) Command Mode An IU is a logical resource comprised of 1-4 credits Also synonymous with a Channel Command Word (CCW) FICON channels support a maximum of 16 concurrent IUs for each Open Exchange IUs contain commands, device-level controls, link controls, and other related functions Common examples of IUs: Command IUs Solicited/unsolicited data IUs Solicited/unsolicited control IUs © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 13 8/16/2011

14. Open Exchanges (OEs) An Open Exchange (OE) is a logical resource representing an I/O being processed by a channel. FICON and FICON Express/Express 2 channels on zSeries servers can support 32 concurrent OEs. FICON Express 2/4/8 channels on System z9 and later servers can support 64 concurrent OEs. IUs sent by a FICON channel during the execution of an SB-3 link-control function or device-level function are restricted to one exchange, known as an outbound exchange. IUs which a FICON channel receives during an I/O operation are restricted to a different exchange, known as an inbound exchange. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 14 8/16/2011

15. Inbound and Outbound Exchanges Command Mode An inbound exchange and an outbound exchange transfer IUs in a single direction. An existing pair occurs when both an inbound and an outbound exchange simultaneously exist between a channel and control unit for the execution of the same device-level or link-level function. The control unit is then said to be connected to the channel. A channel program, which is executed in a single connection, uses only one exchange pair. If the connection is removed by closing the exchanges during the channel program, a new exchange pair is required to complete the channel program. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 15 8/16/2011

16. IU Pacing and Persistent IU Pacing How does it really work? © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 16 8/16/2011

17. IU pacing Each FICON channel port is capable of multiple concurrent data exchanges (currently a maximum of 64) in full duplex mode. Information for active exchanges is transferred in Fibre Channel sequences mapped as FICON Information Units (IUs) which consist of one to four Fibre Channel frames As with prior Z channel protocols, there is a concept of a channel to control unit "connection." In its most primitive form, a connection is associated with a single channel program. In practice, a single channel program may result in the establishment of several sequential connections. In order to prevent a single channel/control-unit data flow from monopolizing the majority of the FICON channel bandwidth, the FICON protocol uses a pacing algorithm which is negotiated during the ELP-LPE sequence. For most cases, the pacing window is 16 IU’s. 8/16/2011 17 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

18. IU Pacing FC-SB-3 Rev 1.6 defines an IU pacing protocol that controls the number of IUs that can be in flight from a channel to a control unit. IU pacing is an FC-SB-3 level-4 function that limits the number of Channel Command Words (CCWs), and therefore the number of IUs, that can either transmit (write) or solicit (read) without the need for additional control-unit-generated acknowledgements called command responses. The control unit may increase the pacing count (the number of IUs allowed to be in flight from channel to control unit) in the first Command Response IU sent to the channel. The increased pacing count is valid only for the remainder of the current outbound exchange. In certain applications, at higher link speeds and at long distances, a performance benefit is obtained by the increase in the allowed pacing count. Basics © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 18 8/16/2011

19. IU Pacing What it really is IU pacing is not a flow control mechanism. Flow control is adequately addressed by the FC-PH level buffer-to-buffer crediting function. IU pacing is a mechanism intended to prevent I/O operations that might introduce very large data transfers from monopolizing access to Fibre Channel facilities by other concurrent I/O operations. Essentially it provides a load-sharing or fair-access mechanism for multiple competing channel programs. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 19 8/16/2011

20. IU Pacing Limitations The first burst of IUs from the channel to the control unit can be no larger than a default value of 16. This causes a delay in the execution of channel programs with more than 16 commands at long distances because a round trip to the control unit is required before the remainder of the IUs can be sent by the channel, upon the receipt of the first command response, as allowed by the increased pacing count. Pacing credits can be adjusted dynamically from these values by control unit requests for specific protocol sequences; however, the channel is not bound to honor control unit requests for larger IU pacing windows. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 20 8/16/2011

21. IU Pacing Mixed results? It yields mixed results insures more predicable I/O response times on heavily loaded channels. produces less optimal results for very-long-distance deployments. Increased link latencies can introduce dormant periods on the channel and its Wide Area Network (WAN) link. Dormant periods occur when delays waiting for anticipated command responses increase to the point where the pacing window prohibits the timely execution of CCWs that might otherwise be executed to insure optimal performance. Mixed results, and limitations led to new enhancement © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 21 8/16/2011

22. Persistent IU Pacing What is it? Persistent IU pacing is a method for allowing FICON channels to have more IUs in flight, and retain a pacing count that may be used at the start of execution of a channel program. This may improve the performance of long I/O programs at higher link speeds and long distances by allowing the channel to send more IUs to the control unit, thereby eliminating the delay of waiting for the first command response. The channel retains the pacing count value, presented by the control unit in accordance with the standard, and uses that pacing count value as its new default pacing count for any new channel programs issued on the same logical path. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 22 8/16/2011

23. Persistent IU Pacing Exploiting the technology The control unit must support the new IU pacing protocol, which is detected during the Establish Logical Path/Logical Path Established (ELP/LPE) sequence. The channel will default to the current pacing values when operating with control units that cannot exploit persistent IU pacing. Control units that exploit the enhancement to the architecture can increase the pacing count (the number of IUs allowed to be in flight from channel to control unit). This can allow the channel to remember the last pacing update for use on subsequent operations to help avoid degradation of performance at the start of each new operation © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 23 8/16/2011

24. Persistent IU Pacing Functionality Each FICON channel provides an IU pacing credit that is initialized at either the start of each channel program or at the reconnection to continue the execution of a channel program. Remember that the IU pacing credit is the maximum number of IUs that a channel may send on a given outbound exchange before it receives a command-response IU. A FICON channel may operate in the default IU pacing mode or in the persistent IU pacing mode. A FICON channel operates in persistent IU pacing mode on each logical path for which both the channel and control unit indicate support for the persistent IU pacing optional feature. On any logical path for which the persistent IU pacing optional feature is not supported, a FICON channel operates in the default IU pacing mode. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 24 8/16/2011

25. Persistent IU Pacing Functionality At the channel When a FICON channel is operating in default IU pacing mode, the IU pacing credit in effect at the start of a channel program shall be set to a model-dependent (DASD array specific) value no greater than the default value of 16. When a FICON channel is operating in persistent IU pacing mode, the IU pacing credit is initialized with the current pacing credit in effect for the logical path. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 25 8/16/2011

26. Persistent IU Pacing Functionality At the control unit When a control unit is operating in persistent IU pacing mode, the control unit may request that the IU pacing credit for the logical path be modified at the start of a channel program. Any modification of the IU pacing credit shall take effect at the start of the next channel program on the logical path and shall persist until changed again by the control unit, or until a system reset is performed for the logical path. The control unit may request that the IU pacing credit be modified by providing an IU pacing parameter in either the command response or in the status sent in response to the first command of a channel program. A control unit may increase, decrease, or reset the pacing credit to the default value. A pacing parameter value of zero shall have the effect of resetting the credit to the default value of 16. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 26 8/16/2011

27. Persistent IU Pacing Functionality At the control unit In order to avoid resetting the pacing count to the default value, the control unit must retain its desired setting and include this value in the pacing parameter for all Command Response IUs and Status IUs in which the pacing parameter is valid. This must be done for all device operations on the same logical path. If the control unit sets the IU pacing parameter to a value less than or equal to the default value, the FICON channel shall not increase the IU pacing credit above the default value. If the control unit sets the IU pacing parameter to a value greater than the default value, then the FICON channel may increase the IU pacing credit by any amount up to the value indicated by the IU pacing parameter. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 27 8/16/2011

28. Performance Implications Do I still need channel extension technology? © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 28 8/16/2011

29. Why extension of storage networks? Safeguard critical business data Provide near continuous access to applications and services in the event of a localized disaster © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 29 8/16/2011

30. High Level Design Considerations For your extended storage network Classification of your stored data How important are the subsets of data for your business operations? How often do you need to backup this data? In event of failure, how quickly do you need to recover? Recovery Point/Time Objectives (RPO/RTO) Synchronous/Near-Synchronous? Asynchronous? Electronic tape vaulting? © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 30 8/16/2011

31. Objectives of extending FICON channels Protect the integrity of the data transaction Keep the FICON channel busy doing useful work Reduce the number of round trips for channel transactions © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 31 8/16/2011

32. Z/OS Global Mirror (formerly XRC) Configurations typically utilize channel extension technology between the remote host and the primary controller. The channel extension technology in these configurations typically uses advanced emulation/acceleration techniques to analyze the CCW chains, and modify them to perform local acknowledgements so as to avoid throughput performance degradation (commonly known as “data droop”) caused by extra round trips in the long distance environment. This technology is typically an additional software license added to the channel extension hardware. Channel extension emulation © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 32 8/16/2011

33. FICON XRC Emulation over Distance XRC System Data Mover IP Network Primary Datacenter Recovery Datacenter IBM z/OS Global Mirror (formerly XRC) data replication is performed by pulling information from a primary data store by SDM (remote read operation) Distance (latency) impacts read performance When FICON emulation is enabled, Brocade 7800 and FX8-24 automatically go into emulation mode Emulates XRC RRS (Read Record Set) sequences, accelerating data flows to the host, and queuing at the channel for SDM Maintains integrity of command and acknowledgement sequences Z-Series MF Hosts Brocade FX8-24 (or 7800) Brocade 7800 (or FX8-24) Primary DASD Target/Secondary DASD © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 33 8/16/2011

34. Z/OS Global Mirror (formerly XRC) Do we need emulation technology? Persistent IU pacing produces performance results similar to XRC Emulation at long distances. It does not really extend the distance supported by FICON, but can provide the same benefits as XRC Emulation. In other words, there may be no need to have XRC Emulation running on the channel extension hardware. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 34 8/16/2011

35. Some performance comparisons © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 35 8/16/2011

36. XRC Extended Distance FICON Channel Extension 4K writes to 1 Volume, 4 Enhanced SDM Readers Application I/O Rate (thousand I/Os per second) 36 8/16/2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

37. Extended Distance FICON Single-Reader XRC Extended distance FICON channel extension 4k writes to 1 volume, single reader I/O Ranges Where Emulation Provides Greater Capacity Than Persistent IU Pacing Application I/O Rate (thousand I/Os per second) 37 8/16/2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

38. Some performance comparisons The workload consists of 64 jobs performing 4 KB sequential writes to 64 data sets with 1113 cylinders each, which all reside on one large disk volume. There is one SDM configured with a single, non-enhanced reader to handle the updates. When turning the XRC Emulation off the performance drops significantly, especially at longer distances. However, after the persistent IU pacing functionality is installed on the IBM DASD array, the performance returned to where it was with XRC Emulation on, except at very high I/O rates. For this case, Persistent IU pacing allows the same functionality and performance benefits without the need for the additional license. For very high I/O rates, the emulation technology outperforms the perisistent IU pacing results. persistent IU pacing performance comparisons for a sequential write workload © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 38 8/16/2011

39. Some performance comparisons persistent IU pacing performance comparisons for a sequential write workload Achieved for the z/OS Global Mirror (XRC) SDM “Read Record Set” (RRS) data transfer from a DASD array (that supports persistent IU pacing) to the SDM host address space. The control units that support persistent IU pacing are able to increase the pacing count. The pacing count is the number of IUs that are “in flight” from a channel to a control unit. FICON supports IU pacing of 16 IUs in flight. Persistent IU pacing increases this limitation for the RRS CCW Chain to permit 240 IUs in flight without waiting for an acknowledgement from the control unit, eliminating handshakes between channel and control unit. Improved IU pacing with 240 IU instead of 16 will improve the utilization of the link. © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 39 8/16/2011

40. IU Pacing and Extended Distance A good enhancement Extended Distance FICON removes IU pacing completely and provides for an optimized path to send no more than approximately 240 IUs from the channel to a control unit without waiting for a command response to open the IU pacing window. Extended IU Pacing is only operational on FICON XRC connections between a z/OS system and a Extended Distance FICON enabled XRC Primary control unit DASD that supports this larger IU pacing value include IBM DS8xxx Enginuity 5874 Q4 service release (i.e. requires the VMAX DASD array) HDS with their USP-V 40 8/16/2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

41. IU Pacing The real story Some of the collateral, including sales presentations by Storage Vendors state that the Extended Distance FICON feature eliminates the need for those expensive channel extenders. This could lead a customer to believe the feature address devices and applications other than Global Mirror (which it does not) It is also is sometime true; BUT…… Brocade does more for Global Mirror than just address the impact latency has on the IU pacing protocol The Brocade solution also emulates DVC-Ack Processing and (under debug flag control) status accept processing 41 8/16/2011 © 2011 Brocade Communications Systems, Inc. Company Proprietary Information

42. OK, what do you mean by “Maybe no need” Extended Distance FICON is for DASD What if you are using the extension device(s) for something else? remote tape connectivity? © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 42 8/16/2011

43. Who Needs FICON emulation technology? Much more use than with just zGM Mainframe installations who need to extend FICON channels beyond natively supported distances (beyond 100-120km) Extend FICON applications over distance via FCIP Facilitate remote connectivity between Primary DASD and remote host running SDM (System Data Mover) for XRC Host and IBM VTS or TS7700 IBM VTC and VTS for peer-to-peer VTS Host and remote Oracle VSM controller Oracle VSM controller and remote tape library/drives Clustered Oracle VSMs Host and Teradata systems Anyone looking to conserve WAN bandwidth (reducing costs) © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 43 8/16/2011

44. Concluding Thoughts and Observations Persistent IU pacing, AKA “Extended Distance FICON” does not increase the distance support for FICON. It enhances the performance of FICON traffic going over long distance This performance enhancement is intended primarily for use with z/OS Global Mirror (formerly known as XRC). This technology does not alter anything with respect to the amount of buffer credits needed to support long distance FICON data transmission. Hence, it does not change the need for FICON directors and/or channel extension hardware used for multi-site disaster recovery/business continuity architectures. Persistent IU pacing may eliminate the need for XRC emulation technology on the channel extension hardware and this may reduce the cost of this hardware. For high I/O rates XRC emulation technology is still a good idea. Emulation technology does more for Global Mirror than just address the impact latency has on the IU pacing protocol © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 44 8/16/2011

45. Key References CMG 2011 Paper “Demystifying Extended Distance FICON” zJournal August/September 2011 issue. FC-SB2/3/4 standards FS-FS 2/3 standards © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 45 8/16/2011

46. Thank You © 2011 Brocade Communications Systems, Inc. Company Proprietary Information 46