Binder1
Upcoming SlideShare
Loading in...5
×
 

Binder1

on

  • 4,174 views

 

Statistics

Views

Total Views
4,174
Views on SlideShare
4,174
Embed Views
0

Actions

Likes
0
Downloads
75
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Binder1 Binder1 Presentation Transcript

  • IBM’s Web 2.0 Strategy: Products And Themes Raymond Josef Edward A. Lara Regional Technical Team, ASEAN SW Lab Services
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What Is Web 2.0 Term coined by Tim O’Reilly for the next generation of web applications and software in general Shows a paradigm shift where applications are not dependent on technology but by the social factor…the “American Idol” phenomenon Web applications, since their inception, has been one big proof-of-concept showing that the technology works, Web 2.0 is about what the technology is used for: to serve content and incite participation
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Concepts Behind Web 2.0 Embrace the “long tail” Current web applications just expose the “head” Web 2.0 apps reaches back as far as needed to get content Content focus Web 2.0 hinges on content, lots of it Critical mass needs to be achieved to bring out the worthy 20% but the other still 80% needs to be there Power to the users Web 2.0 is what the Internet is all about: choice Users must be given freedom to determine what “clicks” and this will lead others to follow How many times have you heard: “Did you see that new video in YouTube?” Webmasters demoted to administrators Webmasters are delegated to just managing the infrastructure and for technical support Content control and even presentation is entirely up to the users The inmates have taken over the asylum The more freedom users have the more they can enrich the application Some rights may have to be given up to the user Constant refinement There is no such thing as a “final version” in Web 2.0 web apps As long as users keep coming back the web application is in “perpetual beta” Cooperate, don’t control Work with the users, build a framework for them to operate with but don’t restrict See “On The Lot” and “Google Gadgets” Multi-channel The browser is not the only platform to present Web 2.0 Mobile Phones, PDAs, Web Kiosk Terminals, and even desktop applications View slide
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore IBM’s Web 2.0 Strategy Technology AJAX REST JSON RSS ATOM Techniques/Services Mashups Composite Applications Tagging Folksonomy Blogging Wikis View slide
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AJAX Stands for Asynchronous JavaScript and XML AJAX is not a new technology or programming language, but a new way to use existing standards and mechanisms With AJAX you can create better, more dynamic, and more user- friendly web applications that incorporates features that were previously only seen in desktop applications (e.g. drag and drop) Dojo Toolkit – Widely supported AJAX framework
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Traditional Web Applications Client Application Server User interaction with page triggers HTTP request… Request processed and HTML page generated Time Client refreshes whole page based on new HTML HTTP request HTTP response User interaction…
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AJAX Web Applications Client Application Server User interaction with page triggers AJAX action… Request processed and XML data or HTML page fragment generated Time Client refreshes part of a page based on new XML data or HTML JavaScript JavaScript invoked to handle request HTTP request HTTP response User interaction…
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore REST Stands for Representational State Transfer Conceptualized by Roy Fielding in his doctoral dissertation which became the architectural model for HTTP (and therefore the World- Wide Web) Characteristics: Client-Server – Uses a request-response type of interaction Stateless – The request must hold all information necessary to accomplish the interaction and should not rely on any server context Uniform Interface – The request makes use of set methods to determine the type of access (GET, PUT, POST, DELETE) Named Resources – The request must name the resource to be accessed from the server Works well with AJAX
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore More On REST REST is not a standard W3C will not put out a specification for REST Vendors will not sell a toolkit for REST REST is a design pattern It is not packaged, it is implemented REST uses standards HTTP URI/URL XML, HTML, GIF, JPEG (resource representations) text/xml, text/html, image/gif, image/jpeg (MIME types)
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore REST In Action This is an example of how REST can be used in an application such as a customer relationship management system AtomGET../crm/customer/cust1225/itemRetrieve a customer’s purchases PUT GET POST GET GET Method ../crm/customer/cust1225/item ../crm/customer/cust1225/item/it301 ../crm/customer/cust1225 ../crm/customer/cust1225 ../crm/customer URI Enter a customer’s new purchase Retrieve a customer’s purchase Update a customer’s information Retrieve a customer Retrieve all customers Action JSON XML JSON XML Atom Representation
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore REST vs. RPC The URI is the methodThe URI calls the method Designed for HTTPProtocol neutral Unique URI per resourceAll resources use the same URI Document orientedData oriented RESTRPC
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What About Web Services? Web services is an excellent technology For systems integration and business processes Generally, for computer-to-computer solutions Web services is not appropriate for all situations Not easily consumable (can’t code “by hand”) Generally, it is not good for user-to-computer solutions We only need this… …so what’s all this!!!
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore JSON Stands for JavaScript Object Notation A specification for the simple and lightweight representation of data JSON is text-based and easy to parse (compared to XML) JSON is not tied to a particular programming language Works well with REST services { "accountNo": "12250301", "name": "Jack B. Nimble", "acctType": "CA", "balance": 1000 }
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore RSS Stands for Rich Site Summary (or Really Simple Syndication) A protocol used for subscription and delivery of content, called syndication Returns an XML document summarizing the updated content
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Atom A syndication protocol similar to RSS but with a richer and more strictly defined message format (works well with REST) Request GET /reilly/ HTTP/1.1 Host: example.org Content-Type: application/atom+xml Response HTTP/1.1 200 Ok Content-Type: application/atom+xml Content-Length: nnnn <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom "> <title>My Submitted Expenses</title> <link href="http://example.org/reilly/"/> <entry> <title>Lotusphere 2006</title> <link rel=“alternate” href="http://example.org/reilly/3.html"/> <link rel=“edit” href=“http://example.org/reilly/3.html”/> <updated>2006-12-05T12:29:29</updated> <content type="XHTML" xml:lang="en-us"> <p xmlns="...">I am a speaker and am staffing a stand in the innovation lab</p> </content> </entry> </feed>
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Mashups And Composite Applications A mashup is… A collection of two or more components bound together through content An application derived by seamlessly combining content from more than one data source A composite application is… A rapidly created application to address an immediate need A typically (but not necessarily) short-lived solution Informal, “good enough” (lacks product quality)
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Tagging Tagging is the collaborative effort to organize information in which users are encouraged to assign freely chosen keywords to identify web sites (i.e. bookmarks) in ways that are meaningful to themselves Tagging results in a new way of classifying data known as a folksonomy Gives rise to tag clouds It’s not what you know… it’s not who you know… it’s who knows what you need to know!
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Blogs And Wikis
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What Does Web 2.0 Mean For My Company? Syndication can be used to disseminate information quickly both internally (e.g. meeting recordings/podcast) and externally (e.g. product updates) Companies can allow mashups and composite applications for specialized functions (e.g. CRM, proposal generation) to increase user productivity Search engines can offer searching based on folksonomies as an alternative to standard taxonomy-based searches Companies can derive ideas for new products by deploying a blog for it’s customer community Product documentation can be refined and made more complete through wikis, errors in documentation can be corrected quickly
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What Happened To SOA? WEB 2.0 Componentized Interoperable Modular Scaleable
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Where Is The Web 2.0 In WebSphere? WebSphere sMash Based on the Project Zero public incubator REST services platform Supports JSON and ATOM Supports PHP and Groovy scripting WAS 6.1 Feature Pack for Web 2.0 AJAX proxy with support for REST connection JSON messaging capability (JSON-RPC) ATOM support Dojo toolkit + IBM extensions
  • Thank you… larara@ph.ibm.com
  • Speaker Name YOGESH BHATIA Title PORTAL AND WEMP CLUSTERING CONCEPTS
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AGENDA INTRODUCTION TO WAS CLUSTERING CONCEPTS WAS CLUSTERING STEPS INTROUDUCTION TO PORTAL WHY WAS STEPS NOT APPLICABLE IN PORTAL CONCERNS IN PORTAL CLUSTERING STEPS FOR PORTAL CLUSTERING INTRODUCTION TO WEMP WHY PORTAL STEPS NOT APPLICABLE IN WEMP
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WAS CORE CONCEPTS An application server is a Java Virtual Machine (JVM) that is running user applications. The application server collaborates with the Web server to return a dynamic, customized response to a client request. Application code, including servlets, JavaServer Pages (JSP) files, enterprise beans and their supporting classes, runs in an application server. Conforming to the Java 2 platform, Enterprise Edition (J2EE) component architecture, servlets and JSP files run in a Web container, and enterprise beans run in an Enterprise JavaBeans (EJB) container.
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Cluster Concept Clusters are groups of servers that are managed together and participate in workload management. A cluster can contain nodes or individual application servers. A node is usually a physical computer system with a distinct host IP address that is running one or more application servers. Clusters can be grouped under the configuration of a cell, which logically associates many servers and clusters with different configurations and applications with one another depending on the discretion of the administrator and what makes sense in their organizational environments. Clusters are responsible for balancing workload among servers. Servers that are a part of a cluster are called cluster members. When you install an application on a cluster, the application is automatically installed on each cluster member.
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Vertical Scaling and Horizontal Scaling
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore FROM STAND ALONE TO FEDERATED
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore REPOSITORY STRUCTURE IN CELL
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore PORTAL SERVER OVERVIEW Product of WebSphere family for displaying portlets ,web site and personalization From WAS point of view , application sitting on top of WAS Upon Installation , its give option to use existing WAS set up or install new WAS Its huge application with lots of libraries and create its own folder structure and config repository
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore FUNDAMENTAL CHARACTERISTICS Information aggregation Targeted and personalized information Accessibility Single sign-on
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Typical Portal Set up
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore PORTAL AND WAS CLUSTERING TRADEOFF In WAS no importance of default applications Portal has set of applications required for its functionality ANY SOLUTIONS ?
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WAS CLUSTERING DEPLOYMENT MANAGER NODE 1 SERVER 1 NODE 2 SERVER 1 addNode addNode
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CLUSTERED WEBSPHERE PORTAL
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore PORTAL CLUSTERING DEPLOYMENT MANAGER NODE 1 SERVER 1 NODE 2 SERVER 1 addNode - includeApps addNode PRIMARY NODE SECONDARY NODE
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore STEPS FOR PORTAL CLUSTER SET UP Install Portal on both nodes Install Deployment Manager Migrate Cloudescape to Db2 of primary node Enable LDAP security on primary node Add Primary node with includeapps Add Secondary node Create Cluster
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore INTRODUCTION TO WEMP
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore It is an extension to WebSphere Portal Enables device specific markup capability Extends Portal navigation to mobile devices Adds mobile administration portlets Provides a framework that can be used to create device independent portlet applications (XDIME Portlets) XDIME Aggregator – displays navigation nodes which are configured and stored in Portal. XDIME Portlets – render applications that generate XDIME. Mutli Channel Server (MCS) Integration – converts XDIME to device-specific markup. Easily supports new devices with mobile device update service It takes advantage of many Portal Features Configuration (xml access) Navigation (Aggregator/Model API) Security (e.g. TAI/ACL) Scalability (cluster) Virtual Portal It includes an extensive tooling suite built on IBM’s Eclipse-based Rational Software Development Platform What is WebSphere Everyplace Mobile Portal?
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore With unlimited applications and thousands of devices, the challenge becomes how to solve an expanding "M x N" matrix N devices M applications...
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore The solution is to define a Mobile Platform that dynamically and intelligently adapts to the specific device N devices M applications... IBM WebSphere® Everyplace® Mobile Portal software
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WebSphere Application Server WebSphere Everyplace Mobile Portal Overview WebSphere Portal MCS runtime MCS Repository Layout Theme Components Device XDIME Portlets XDIME Aggregator Devices
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Runtime Component Interaction XDIME Aggregator Portlet Filters (IBM Portlets only) XHTML MP Portlet Container XDIME (XDIME) XHTML Basic WML PortletRequest PortletRequest PortletResponse PortletResponse Manage Mobile Pages Portal Navigation and Content Model ServletRequest ServletResponse Multi - Channel Server MCS Policy Repository XDIME Device - specific 6 XHTML Basic XHTML MP WML WebSphere Portal (WP) 7 5 4 3 2 1 MCS Servlet Filter MCS components WEMP extensions to WPLegend XDIME XDIME
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore REQUIREMENT FOR WEMP INSTALLATION MCS to be installed after Portal Problem is in secondary node as well , MCS needs to be installed and it requires database ANY SOLUTIONS ?
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOLUTION INSTALL BOTH PORTAL NODES MIGRATE PRIMARY NODE CONNECT SECONDARY NODE WITH SAFE MODE USE SAME DATABASE FOR SECONDARY NODE
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore VARIOUS STRATEGIES FOR WEMP CLUSTERING Install WEMP on federated Portal Node Install WEMP on node not managed by DMGR ( discussed ) Install WEMP to a Portal Cluster
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore INSTALL WEMP ON FEDERATEDE NODE Install WebSphere Portal on WP01 Create the Installing and configuring a WebSphere Portal cluster . Verify that WebSphere Portal works with the Web server configuration. Install the mcs Multi-Channel Server for DB2 or Install Multi-Channel Server for Oracle. Install the Everyplace Mobile Portal extensions. Replace the ./WPSconfig.sh init mcs-deploy-mcs command with: ./WPSconfig.sh init mcs-config-cluster-secondary-node -DMcsRepositoryType=odbc -DMcsDbVendor=<db_vendor> -DMcsDbHost=<hostname> -DMcsDbPort=<port_number> -DMcsDbSource=<db_name> -DMcsDbUser=<db_instance> -DMcsDbPassword=<db_instance_password> -DMcsProject=mobile-portalUse the same Multi-Channel Server database information used by the WP01 instance.
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore INSTALL WEMP NOT MANAGED BY DMGR Install WebSphere Portal on this instance following steps in Installing WP02 in Installing WebSphere Portal on a separate WebSphere Application Server node not managed by the deployment manager. Follow the remaining required steps and actions in the topic Before you install Everyplace Mobile Portal. Install WebSphere Portal on this instance and then follow the remaining required steps and actions in the topic Before you install Everyplace Mobile Portal. Before reconfiguring the database from Cloudscape to the database used by the WP01 instance: Follow the procedures in the topic Installing Multi-Channel Server for DB2 or Installing Multi-Channel Server for Oracle and use the following information:
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore ….continued Use the same Multi-Channel Server database information used by the WP01 instance. Ensure that the following parameters have identical values for both WP01 and WP02: DMcsRepositoryType, DMcsDbVendor, DMcsDbHost, DMcsDbPort, DMcsDbSource, DMcsDbUser, DMcsDbPassword Do not run the following command since the default device policies were imported when you installed Multi-Channel Server on WP01: ./WPSconfig.sh init mcs-import-devices -DMcsProject=mobile-portal When installing the Everyplace Mobile Portal extensions, do not run the following commands since the Everyplace Mobile Portal extensions policies were imported when you installed the Everyplace Mobile Portal extensions on WP01: ./WPSconfig.sh mwp-import-policies ./WPSconfig.sh mwp-import-portlet-policies
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore INSTALLING WEMP TO PORTAL CLUSTER Perform the recommended steps and actions in the topic Before you install Everyplace Mobile Portal. Verify that WebSphere Portal works properly. Follow the procedures in the topic Installing Multi-Channel Server for DB2 or Installing Multi-Channel Server for Oracle. Follow the procedures in the topic Installing Everyplace Mobile Portal extensions and replace the ./WPSconfig.sh mwp-config or ./WPSconfig.sh mwp-config-all command with ./WPSconfig.sh mwp- config-cluster-primary-node
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore ….CONTINUED Following steps should be performed after installing WP02: Load the Administrative Console from DM01 and perform the following steps: Run Full Resynchronize on both nodes. Restart the cluster. Configure the Everyplace Mobile Portal extensions for cluster using one of the following options: Option 1 - To install and configure all of the Everyplace Mobile Portal extensions, run the following command: ./WPSconfig.sh mwp-config-cluster-allOption 2 - To install and configure all of the Mobile Portal extensions, except for the sample XDIME portlets and the sample XDIME pages, run the following command: ./WPSconfig.sh mwp-config-clusterTo activate the portlets, run the following command: ./WPSconfig.sh activate-portlets
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore THANK YOU QUESTIONS ???
  • © 2008 IBM Corporation IBM Software Group WSTC Workshop Collaborative Development Using Rational Team Concert
  • © 2008 IBM Corporation WSTC 2008 Lab Overview 2 Agenda Introduction to Rational Team Concert Lab Overview Module 1 Setting up the Team Module 2 Planning Your Work Module 3 Keeping Track of All Our Work Module 4 Performing and Sharing Your Work Module 5 Remembering Well Known SCM Configurations Module 6 User’s View of Build Module 7 Endgame and a Tightened Process Session Summary
  • © 2008 IBM Corporation WSTC 2008 Objectives Describe development and delivery challenges with distributed teams Explore how Team Concert can Enable development teams to collaborate in real time in the context of the work they are doing, especially in globally diverse environments Enable projects to be managed more effectively by providing visibility into accurate project health information drawn directly from actual work Automate traceability and auditability by managing artifacts and their inter-relationships across the lifecycle empowering teams to deliver more value Provide customizable process design and enactment through rule-based process guidance, automation and definable checkpoints Provide a hands on experience using Team Concert to automate the software delivery process Collaborative Development Using Rational Team Concert 3
  • © 2008 IBM Corporation IBM Software Group WSTC Workshop Introduction to Rational Team Concert
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 5 Global development and delivery challenges 50% of outsourced projects are expected to underperform Mismatched & Misunderstood Process Communication Issues Cultural Issues Decreased Productivity Increased Rework Mistakes in Work-Transfer Higher Co-ordination Costs Political Issues Lack of Security IP Protection Lack of Project Visibility, Agility and Control Lack of Project Metrics – Unable to Measure Success Geographically Distributed Development Creates Gartner survey of 219 clients who outsourced projects offshore & domestically – half are expecting to fail to deliver anticipated savings. Acquisitions & Mergers Application Backlogs Budget cuts/ Cost Reductions Variable Staffing Model Increased flexibility to adapt quickly Business Drivers Customer Pain Points
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 6 Trends toward SOA & geographic distribution Impact of growing modularity More granular service functionality in composite business applications Large number of projects & assets including custom, outsourced & packaged Impact of accelerating change Asset will be impacted by frequent updates and changing interdependencies Effective cross-organizational visibility and synchronization becomes an imperative! Drives the need for enabling better performance of organizationally diverse people & assets across the software supply chain Business management Business-IT alignment Development Software delivery Operations Service management SOA enabled Business Flexibility
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 7 Customers speak: What’s needed to break through “What’s needed is a collaborative application lifecycle management approach that gets business and technology to work as a team.” ““Something that doesn’t get in the way and doesn’t add overhead.” “Something that gives both the top down view and the low level view and links back to configuration, bug tracking, and build trending” Development Lead “Something that helps me plan iterations and balance work loads across my distributed teams and onboard team members and projects in a day instead of a month.” Development Manager “Something that automates my development process with short iteration cycles so we’re able to quickly fulfill business requests.” Program Manager “A more transparent view into my development teams allowing me to make better decision and to deliver on time to my executive team” CIO Developer
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 8 What’s different today? 2008 and beyond: Shifts in software delivery These shifts are setting the agenda for a new approach to software delivery... What we’re seeing… “2/3 of our projects span multiple business units” “Our architecture has to be simpler... we can’t afford this cost model anymore” “We’ve done two acquisitions in six months and we can’t lose customers” “Our last ERP upgrade took six months… that isn’t going to fly next time” “We have to go where the best talent is, but we have IP and compliance realities” What we’re hearing… Increasingly diverse and cross organizational global project teams Interest in replicating the proven models of open, community- governed software delivery models Popularity of Agile development and social networking models SOA, Web-centric and Web 2.0 enabling new business models
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 9 Enable team transparency of “who, what, when, why” Build team cohesion and presence Automate hand-offs so nothing falls through the cracks Automate team workflow improving productivity Automate data collection eliminating administrative overhead Real time reporting and alerts reduces project risk Dynamic provisioning of projects and teams Real-time iteration planning and workload balancing Unify teams with tools choice Real-time integration of people, process and projects across the lifecycle Collaborate in Context Right-size Governance Day One Productivity Robust, extensible and scaleable Globally distributed, fluid and dynamic Community-based and open at Jazz.net A open technology initiative to transform how people work together to deliver greater value and performance from their software investments Envisioning a platform that can transform software delivery
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 10 How Jazz is different Dynamic integration of People Desktop/database integration Global integration Function first Team first Pre-determined tool function Dynamically extensible Rigid rules Creative collaboration Manual tasks Process aware and transparent Proprietary infrastructure Internet standards Lengthy project onboarding Day one productivity Hindsight analysis Continuous steering Status guesstimates Real-time insight ...across the software lifecycle Process Projects
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 11 Open Lifecycle Service Integrations Jazz platform architecture JAZZ TEAM SERVER Search and Query In Context Collaboration Team Awareness Dashboards Events Notification Security Server Integrations Eclipse Web 2.0 Visual Studio (4Q) Others to come… Client Integrations Rational Team Concert Rational Quality Manager Rational Requirements Composer Enterprise Reporting Project Management Others… Business Partner & IBM Offerings Best Practice Processes Rational ClearCase 7.1 Rational ClearQuest 7.1 Rational Build Forge 7.1 Rational Asset Manager 7.1 Subversion
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 12 Software innovation through collaboration Real time, in-context team collaboration Make software development more automated, transparent and predictive "Think and work in unison" Integrated source control, work item and build management Assess real-time project health Capture data automatically and unobstrusively Automate best practices Dynamic processes accelerate team workflow Out-of-the-box or custom processes Unify software teams Integrate a broad array of tools and clients Extend the value of ClearQuest and ClearCase Support for System z and System i servers (Beta 3Q) IBM Rational Team Concert transparent integrated presence wikis OPEN real-time reporting chat automated hand-offs Web 2.0 custom dashboards automated data gathering EXTENSIBILITY Eclipse plug-ins services architecture FREEDOM TO CREATE Open and extensible on Collaborate in context Right-size governance Day one productivity
  • © 2008 IBM Corporation WSTC 2008 Delivering customer value with Rational Team Concert Collaborative Development Using Rational Team Concert 13 The evolution of software delivery Supporting the software economics of an increasingly global, dynamic and web-based technology environment Project initiation in days rather than weeks Reduction in time to first demonstrable release Improved cycle times through automated and reusable patterns, services and architectures Less overhead to stay compliant with standards Earlier detection and accelerated resolution of defects Improved predictability of project outcomes Fewer meetings and status reports Reduction in late scrap and rework Reduced attrition of skills and greater flexibility to leverage top technical and business talent Day one productivity Right-size governance Collaborate in context
  • © 2008 IBM Corporation WSTC 2008 Rational Team Concert: A Deeper Look Collaborative Development Using Rational Team Concert 14 Team advisor for defining / refining “rules” and enabling continuous improvement Process enactment and enforcement In-context collaboration shows team members and status of their work Customizable project dashboard Real-time reporting and queries Integrated iteration planning and execution Real-time project status window Single structure for project related artifacts World-class team on-boarding / offboarding including team membership, sub-teams and project inheritance Role-based operational control for flexible definition of process and capabilities Jazz Team Server Flow browser shows developer to team flow Integrated stream management Component level baselines Server-based sandboxes Identifies component in streams and available baselines ClearCase connector SCM Work Items Defects, enhancements and conversations Query results view and share queries with team or member Support for approvals and discussions ClearQuest connector Query editor interface Work item and change set traceability Local or remote build servers Supports ant and command line tools Integration with build forge Build definitions for team and private builds Build
  • © 2008 IBM Corporation WSTC 2008 Unify distributed teams in mid-sized or enterprise settings Collaborative Development Using Rational Team Concert 15 A complete, standalone solution for departmental and medium-sized distributed teams Supports departmental “satellite” or “agile” projects optionally integrated with existing enterprise-proven Rational ALM infrastructure offerings Enhances existing ALM offerings with new collaborative & process capabilities Bi-directional auto-synchronization Project Management Europe Development China Config & Build China Team Concert Development Australia EclipseTM, Linux®, Microsoft® Windows®, UNIX®, IBM z/OS®, IBM i5/OS Rational ClearQuest Rational Enterprise-proven ALM Infrastructure Rational Build Forge Rational ReqPro Rational ClearCase Define Build Test Deploy New York City North Carolina Testing Ireland
  • © 2008 IBM Corporation WSTC 2008 Delivering global team collaboration and visibility Allows individuals to focus on what they do best Collaborative Development Using Rational Team Concert 16 Deployment Manager Developer Business Stakeholder Architect Build Engineer Know exactly what to fix when something breaks Create great software, unencumbered by tedious status reports Build design reviews into your software lifecycle-no more chasing after rogue development streams. Provision a new team member across the world in a day Track progress on the updates that your customers care about. Make confident go/no decisions based on facts, not estimates. Project Manager
  • © 2008 IBM Corporation WSTC 2008 We eat our own dog-food! Collaborative Development Using Rational Team Concert 2-way Xeon Server running application server (WAS) and another running DB2 Jazz Project – Using bi-weekly iteration builds Jazz and Rational Team Concert self hosting since 4Q06 Global team in 7 locations in NA and Europe ~100 developers, plus jazz.net web client access Repo > 10G, 66K files, 43K work items Over 10 internal Rational Development Team leveraging RTC
  • WSTC 2008 © 2008 IBM Corporation Collaborative Development Using Rational Team Concert 18 Customer Feedback “Rational Team Concert not only brings the communications channel inside the development platform, but it integrates everything. The integration with the complete workflow for the life cycle ... is a big improvement.” -- Major health care insurance provider "Features provided by Rational Team Concert are equal or more sophisticated than current used tool development environment at RB (JIRA, CVS and Release Engine).“ -- Global supplier of industrial, consumer and building technology "I found that it was straightforward to install, which made a nice change. Starting up the server and getting the clients talking to it was simple as well.“ -- Leading UK financial services organization
  • © 2008 IBM Corporation WSTC 2008 Collaborative Development Using Rational Team Concert 19
  • © 2008 IBM Corporation WSTC 2008 Lab Overview 20 Agenda Introduction to Rational Team Concert Lab Overview Module 1 Setting up the Team Module 2 Planning Your Work Module 3 Keeping Track of All Our Work Module 4 Performing and Sharing Your Work Module 5 Remembering Well Known SCM Configurations Module 6 User’s View of Build Module 7 Endgame and a Tightened Process Session Summary
  • WSTC 2008 © 2008 IBM Corporation Scenario for PoT Labs You are joining an exciting new project called Squawk that has recently been started in your company. You will be using Rational Team Concert as the project’s collaborative development environment. The Squawk project is run according to the Eclipse Way, a development process based on the wildly successful Eclipse project (www.eclipse.org). You have joined the project at the start of Milestone 2 and have a chance to make a real difference to this project. Lab Overview 21
  • WSTC 2008 © 2008 IBM Corporation Scenario for PoT Labs Squawk is a (simple) program that will print out different sounds depending on who “squawks”. The Dog squawker goes “bark”, the Cat squawker goes “meow”, etc. Your main task is to create a new squawker, along with tests and documentation. At the same time as creating new squawkers, you will get to participate in some planning activities, interact with your fellow team members, deliver your work to the project, trigger automated builds and various tasks typical for project teams everywhere. The project team structure mimics the four major components: Core Library Documentation User Interface Release Engineering (build) You are assigned to the Core Library and Documentation teams with a team leader (one of the instructors). Welcome to the team! Lab Overview 22
  • WSTC 2008 © 2008 IBM Corporation Eclipse Overview Lab Overview 23 Current Perspective Change Perspective A View All screen elements below the button bar are called Views Add new Views using the Window- >Show View menu Menu Button bar View tabs This View has different areas accessed via the View tabs
  • WSTC 2008 © 2008 IBM Corporation Team Concert Terminology Jazz artifacts are stored in a repository. The repository contains project areas, which are the system's representation of a software projects. Each project area has an associated process, which governs how the project is run. Project Areas are decomposed into a set of team areas, which describe the teams that work on the project. Teams use a stream to store the master copy of project’s files. Team Members use a personal repository workspace to work on project files. Lab Overview 24
  • WSTC 2008 © 2008 IBM Corporation Squawk Project in Team Concert Lab Overview 25 Project Area Streams Repository Workspaces Team Areas Process Definition
  • WSTC 2008 © 2008 IBM Corporation Team LeadTeam Lead (instructor)(instructor) TeamTeam MemberMember (student)(student) Lab 2Lab 1 Chat with the Team Chat with the Team Create Work Items Create Work Items Examine Iteration Plan Examine Iteration Plan Accept Team Invite Accept Team Invite Create & Distribute Iteration Plan Create & Distribute Iteration Plan Sequence of events – Lab 1 & 2 Lab Overview 26
  • WSTC 2008 © 2008 IBM Corporation Team LeadTeam Lead (instructor)(instructor) TeamTeam MemberMember (student)(student) Lab 4Lab 3 Project Status with Team Central View Project Status with Team Central View Track your Work with My Work View Track your Work with My Work View Create & Deliver Work Create & Deliver Work Sequence of events – Lab 3 & 4 Lab Overview 27 Accept all Changes Accept all Changes
  • WSTC 2008 © 2008 IBM Corporation Team LeadTeam Lead (instructor)(instructor) TeamTeam MemberMember (student)(student) Lab 5 Create & Promote Workspace Snapshots Create & Promote Workspace Snapshots Accept New Baselines Accept New Baselines Create Component Baselines Create Component Baselines Explore Snapshots Explore Snapshots Replace contents of Workspace Replace contents of Workspace Sequence of events – Lab 5 Lab Overview 28
  • WSTC 2008 © 2008 IBM Corporation Team LeadTeam Lead (instructor)(instructor) Lab 6 Request Personal Build Request Personal Build Request Integration Build Request Integration Build Explore BuildExplore Build Sequence of events – Lab 6 & 7 Lab Overview 29 Lab 8 Change to Development Iteraton Change to Development Iteraton Experience Process Enactment Experience Process Enactment Change to the Endgame Iteration Change to the Endgame Iteration TeamTeam MemberMember (student)(student)
  • WSTC 2008 © 2008 IBM Corporation Lab Conventions The hostname used to connect to the Jazz Team Server is jazz-server Each student is assigned a unique user id of the form student<N> based on their student number, e.g. student1 Examples in the lab workbooks use student1, you will need to adjust per your assigned id Every student creates their own unique Squawker. Examples in the lab workbooks use Lion Students can choose any squawker they want but should include your student id in the name <squawker>_<student id>, e.g. Lion_student1 Work items created should include the full squawker name in the summary title <squawker name> Implementation and <squawker name> Documentation e.g. Lion_student1 Implementation Lab Overview 30
  • WSTC 2008 © 2008 IBM Corporation Lab Overview 31
  • WSTC 2008 © 2008 IBM Corporation Lab Overview 32 Agenda Introduction to Rational Team Concert Lab Overview Module 1 Setting up the Team Module 2 Planning Your Work Module 3 Keeping Track of All Our Work Module 4 Performing and Sharing Your Work Module 5 Remembering Well Known SCM Configurations Module 6 User’s View of Build Module 7 Endgame and a Tightened Process Session Summary
  • WSTC 2008 © 2008 IBM Corporation Session Summary 33 Session summary We have described current collaboration challenges with distributed teams We have explored how Team Concert can Enable development teams to collaborate in real time in the context of the work they are doing, especially in globally diverse environments Enable projects to be managed more effectively by providing visibility into accurate project health information drawn directly from actual work Automate traceability and auditability by managing artifacts and their inter-relationships across the lifecycle empowering teams to deliver more value Provide customizable process design and enactment through rule-based process guidance, automation and definable checkpoints We have provided a hands on experience using Team Concert to automate the software delivery process
  • WSTC 2008 © 2008 IBM Corporation Session Summary 34 Next steps Engage your local Rational team Provide a customized demo for your team Conduct a targeted proof of concept Register on jazz.net and explore learning tutorials and videos http://www.ibm.com/developerworks/spaces/jazz
  • WSTC 2008 © 2008 IBM Corporation Session Summary 35 Additional resources Learn more about and download free trials of Rational Team Concert at http://ibm.com/rational/rtc Explore Rational Team Concert tutorials, demos and other developer learning resources http://ibm.com/developerworks/spaces/jazz Participate in the open commercial development of Jazz by joining the community http://jazz.net Learn more about the Jazz technology and the future IBM Rational product roadmap http://ibm.com/rational/jazz/roadmap
  • WSTC 2008 © 2008 IBM Corporation Session Summary 36
  • WSTC 2008 © 2008 IBM Corporation Session Summary 37 We appreciate your feedback in order to improve this educational event. Please fill out the survey form.
  • Melvin Cheng Advisory IT Specialist
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Topics to Cover Pain Points Typical Software Delivery Challenges Overview of Build Forge
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Pain Points: “Spaghetti Infrastructure” Issues Lack of consistency and/or repeatability of configuration changes Staff bogged down in keeping environments synchronized No change history / audit trail for WebSphere configuration Costly automation of configuration changes and deployments Requires custom coding Time consuming to deploy without a framework
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Best Practices for WebSphere Administration There are many Best Practice procedures recommended by WebSphere but not implemented within the product Backup current version of application before updating with new build Use source control to manage & control WebSphere configuration settings Embed build id in J2EE EAR file Manifest.mf file as it is built Fall back to previous version of the application if the update fails or is unacceptable Store a record of who updates the application and when (and why)
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Test Environment Deploy Source Code Repository Typical Software Delivery Lifecycle Roles Source Build Scripts Build Results Deploy Package + = Why I am I always the last to know that the build broke? I wish I knew immediately when a build breaks and could restart it from there!! Manually configure the server, the test framework and deploy the build… what environments on how many servers? Hello! I need to get some testing done!!! TesterDeployerBuild EngineerDeveloper
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Problem: Silos between development, build, test and deployment Inconsistent policies and processes cannot be practiced among all stakeholders Silos of information contribute to reducing team productivity Difficult to facilitate communication, visibility, sharing, and leveraging globally distributed teams without mass chaos Limited to no traceability among the artifacts Minimal reuse of artifacts and controls Difficulty tracking status of multiple branches and multiple builds Labor Intensive Tribal Knowledge Error Prone Handoffs Implement & Iterate Build & Stage Test & Validate Software) Build Test Build Assets Development Assets Test Assets Develop Test Environment Deploy
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Application deployment challenges Inability to ensure the test environment matches production results in quality problems and slow problem resolution Customer Dissatisfaction Lost Revenue or Productivity Deployment errors result in roll-backs and unplanned down-time of business critical applications Lack of an auditable deployment process hinders ability to comply with external regulations Penalties, Shareholder Value Manually configuring complex applications to deployment servers results in inefficient use of IT resources High Labor Costs
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore System Deployments become increasingly complex
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build & deployment “motivators” Compliance Requires the ability to track software from development to production and back again Repeatability & Reliability How to reliably deploy software through multiple test phases and ultimately to production. Process – a need to apply a standardized set of steps in an organization Server Configurations – production server configurations don’t always match test lab configurations, which leads to errors in deployment Change – updates to existing systems often cause failure due to a lack of understanding of how the application is currently deployed, and what impact the new application will have on the existing system Security – authentication and electronic signature approvals Lifecycle Automation Be able to queue multiple automation efforts into a single automated process – Build -> Deploy ->Test as a single automated process
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Can you trace your development artifacts? Development Operations Release Build Artifacts Source Code Environment Changes Implemented with this Built using these In this Executable This Deployment Artifacts Which deployed these System on this Application Of this Tests Test Results Where these Produced these Build Scripts Which required these Requirements Approval Implemented these Approved by
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Development Matters: Can you pass this audit? What is your deployment process ? Who approves the deployment of applications into test and production environments. Where is the audit trail? What build artifacts are associated with this release of the application? What version of build artifacts are deployed in the Test Environment? The Production Environment? What version of source is associated with this build artifact? Who deployed the code on this server? How is that controlled? How are approvers authenticated & how do they “sign off”?
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deployment Questions How can I create and enforce a repeatable process for deploying applications through a variety of test environments prior to going into production How can I ensure that the build that was approved during testing is the same build deployed into production? How can I prevent unauthorized tampering of the build artifacts? Can I recreate the build that is currently deployed in a specific environment? What build artifacts are loaded on Testing Servers? The Production Servers? How can we load different machines with the same application version simultaneously?
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Software Delivery Lifecycle Challenges Manual, error-prone processes Proprietary, internally-developed systems Inconsistent processes for different products and platforms Dependence on the build team for execution, status, and troubleshooting Separation of developer environments from production systems Difficult and time consuming to resolve problems Difficulty proving adherence with compliance requirements Delays and missed deadlines for application releases/updates Unpredictable product release cycles Costly systems to support, troubleshoot and maintain with knowledge held by few More work requires more people – limited repeatability or portability for new projects and platforms Burdens staff and requires increased headcount to take on new projects Requires better tracking and auditing of application lifecycle Current ConditionsTypical Challenges Business Impact
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge is a process execution framework Automates any set of tasks that can be executed from command line or scripts Eliminates manual handoffs between code, build, test, package, and deploy activities Self documenting processes that record all results in one auditable location Manages the activities in the software development lifecycle Application build and release process management Runtime customization and application deployment The “conveyor belt” analogy for a software assembly line Automating Software Delivery with Build Forge
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Implement & Iterate Build & Stage Test & Validate Software Test Environment Build Forge Streamlines Software Delivery Automate, Accelerate, Coordinate, Track, Analyze Reduces global development cost by automating processes, handoffs, and communication Accelerates development cycles by enabling rapid iterations and rapid feedback Improves quality by managing product assembly tasks for consistent results each time Practical solution that leverages existing scripts and tools for rapid implementation and results Provides traceability and control for compliance management and better software governance Develop Test IBM Rational Clear Case - Source Control Deploy IBM Rational Clear Quest - Defect Tracking Build IBM Rational Testing Suite - Testing Tools IBM Rational Build Forge Development Assets Build Assets Test Assets
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore In Production Source Build & Deploy Artifacts Implement Build & Stage Provision & Validate Server(s) Function Test Approve Defects E-SIG Integration Production Project ManagerTesterDeployerBuild EngineerDeveloper Automation Orchestrates The Roles Requirements
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Pushing the Quality Agenda Upstream Why Customers Choose Build Forge Hold developers to a higher level of quality Identify & resolve quality issues faster Enable agile development Meeting Compliance Requirements Avoid the dreaded failed application audit Get audit control and traceability Secure the build environment Achieving The Software Factory Software production as a “service” Standardize and reuse across teams Remove obstacles to distributed development Time-to-Market Advantages Remove the software production bottlenecks Automate the assembly process Remove manual handoffs
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge Extensive Tool Integration Deployment & Provisioning Tools Tivoli Provisioning Manager WebSphere Admin (wsadmin) WebSphere Service Registry VMWare, VMLogix Ascendant WICA Software Quality Tools Mecury Quality Center, LoadRunner, TestDirector WinRunner, Junit Rational Test Lab Manager, Rational Functional, Performance & Manual Tester Agitar Monitor, Record, Report Automate and manage lifecycle processes IBM Rational Build Forge Source Control & Change Management Tools Rational ClearCase Rational ClearQuest StarTeam, Perforce CVS, PVCS, VSS Synergy Subversion Remedy, Bugzilla Endevor, ChangeMan Development Tools Rational Application Developer Eclipse, VisualStudio Rational Automated Software Analyst Rational Team Concert (Jazz) Klocwork, Black Duck Compile & Assemble Tools Compilers (any) Linkers (all) Ant, Make, NMake Maven, Rational ClearMake Jar, rpm, zip, Governance Tools Rational Method Composer Rational ClearQuest WebLayers Rational Asset Manager
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Process Automation Automated, Distributed, Repeatable, Consistent Software Delivery Lifecycle Solution Overview Python, VBscript, Batch, Perl, KSHScripting Source Control Product Build Quality Assurance Package ReleaseDevelopment Control Acceleration Environment Notification Scheduling Log Analysis Tracking Analysis Platforms UNIX, Windows Mac, Linux, zSeries, iSeries, Proprietary • ClearCase • StarTeam • Perforce • CVS • PVCS • VSS • Synergy • Subversion • etc. • ClearMake • Ant • NAnt • Make • GNUMake • NMake • Open Source • CruiseControl • Maven • etc. • Java • C • C++ • C# • etc. Source ControlLanguages Build Tools • TestManager • Performance • Functional • Robot • LoadRunner • TestDirector • WinRunner • Junit • etc. Test Tools • Tivoli • WebSphere • WebLogic • etc. Release • RAD • Eclipse • Visual .NET IDEs • ClearQuest • Remedy • ChangeMan • DevTrack • Bugzilla • etc. Change Mgmt • Wise • InstallShield • jar • RPM • Package • etc. Package Tools Management Console Centralized Web-based Management, Collaborative Distributed Access, Role-Based Security IDE Plug-Ins Developer Self-Service, Role-Based Security SOFTWARE DELIVERY AUTOMATION FRAMEWORK
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Example Build Forge Automated Process Build Forge GO Rational Clear Case Rational Clear Quest RAM 1. Extract Source Code 6. Update Defect Tracking Aggregated Reports across entire Process 2. Extract Asset Metadata 3. Enforce SOA Development Policies 4. Build Source Code 5. Package Build Results 7. Deploy Build Results 8. Execute Tests Effect of Class Sharing on Memory Footprint in WAS V6.1 126 245 363 481 600 719 140 233 328 422 516 609 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 Number of Running App Servers Megabytes WAS 6.02 WAS 6.1 System configuration Linux: IBM X-Series 335, 2x 3.2 GHz Intel Xeon, 8.00 GB RAM, Suse Enterprise Linux 9 SP3
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Functional Architecture Build Forge Master Database • Projects and Steps • Server Configurations • Environment Configurations • User/Permission Information • Build Statistics • System Master Log Build Forge Engine • 3 Tier Architecture • Centrally Managed • Orchestrates BuildForge Tasks Server Pools • Manage - Users - Resources • Author - Capture Processes and Steps • Execute - Rune/Schedule Jobs - View Real-Time Status - Troubleshoot Build Issues • Analysis - Reporting & Analysis Linux HP-UX zSeries MacWindows Solaris AIX Linux Windows Windows Windows Integration • Command Line • Adaptors • API • Developer Self Service DB2 Oracle MySQL SQL Server Sybase Linux HP-UX AIX MacWindows Solaris Quality Assurance Build Systems Linux HP-UX zSeriesWindows Solaris Production Build Forge Management Console Build Forge IDE Plugin Build Forge Server Build Forge Agents Agents • Real Time Environment Configuration • Execute Project Steps • Return Output / Logs to Mgmt Server ACCESS EXECUTEIMPLEMENT Source Code Repositories Test Suites Deployment Tools SCM Applications iSeries Database… • Build Stats • BOM data Engine…
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Master Database • Admin & Server Configurations • Environment Configurations • User/Permission Information • All project record data Server Pool LinuxWindows SolarisAIX Linux Windows Windows Windows Linux HP-UX AIX MacWindows Solaris Braintree BangaloreDallas Build Log Database • Build Statistics • BOM data Build Log Database • Build Statistics • BOM data Console / Engine Console / Engine Console / Engine Console / Engine Build Log Database • Build Statistics • BOM data Log DB Log DB Log DB Master DB GDD features focuses on performance improvements for remote agents writing to the database Architected for Centralized Management Across Geographically Distributed Development
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Building Blocks… Containers that hold a series of steps necessary to execute a process. A project can contain a series of diverse tasks beyond the compile, including automated tests, running installers, updating databases, or a website update.Project Individual tasks performed as part of a project. Each contains one or more command line statements to be executed on a server machine. A step can run a script, invoke another application, move or copy files, export a project for backup, and more.Steps Environment Collections of environment variables which can be maintained separate from projects and steps, then assigned to projects and steps as needed. 1. RELEASE=Release_1.1 2. JAVA_HOME=C:Program FilesJavajdk1.5.0_06 3. PATH=C:Program FilesJavajdk1.5.0_06bin 4. … Build Test Package DeploySource … 1a. cleartool mkview -snapshot -tag $BF_TAG -vws $VIEW_STG$BF_TAG.vws $BF_SERVER_ROOT$BF_TAG 1b. cleartool setcs -tag $BF_TAG config.spec 2a. gcc main.c -o main.o 2b. gcc main.c -o ui.o 2c. make Release 3a. testscript.sh –run –r $RELEASE –module HelloWorld.exe
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge project
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge project Threading: Threading allows for multiple Steps to run in parallel, thereby increasing the speed of your build. Multiple, adjacent, Threaded steps are thought to be a Thread Block Threaded Steps are indented on the Project Page
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge projects - Threading Get Src Compile Solaris Compile Linux Test Package DeployInit Compile Windows Get Src Compile SolarisCompile Linux Test Package DeployInit Compile Windows
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge environments The BuildForge system allows you to manage environment variables separately from your Projects, Servers, and Steps You can define Environment Groups – Containing one or more Environment Variables You can assign Environment Groups to Servers, Projects, and Steps – At runtime, the system assembles the final working environment for a Step from all of the relevant Groups assigned to the Server, Project, and Step
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge environments
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge security Concepts in the BuildForge security system: Users – A user in the BuildForge environment corresponds to a single, named person who interacts with the system in some way. Access Groups – An access group is a collection of users who are assigned permissions to perform specific tasks within the BuildForge system. An Access Group is usually associated with a logical Role within an organization – the .NET User Interface development team, for example. Permissions – Permissions are typically set on system level functionality and are assigned to Access Groups only – not individual Users. Ownership – Specific resources – Projects, Servers, Steps, etc. – in the BuildForge system have a single, named Access Group as Owner. Ownership over an entity allows for full edit/delete/execute privileges over that resource.
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge security Role-based system Access Groups represents a role a User has in organization Roles have Permissions and Ownership User access is determined by the union of the Permissions of all groups the user belongs to Combination of Permissions and Ownership define what a group can do and/or see To edit a give Project, a User would have to have Ownership on the Project AND Permissions to Edit Projects Example User who is a member of the Guest group (and no other groups) sees only Projects which have the Guest group assigned as their Access property User can only launch projects with Guest access
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge security
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge servers A Server is a logical resource in the BuildForge environment that represents any physical system running the BuildForge Agent. A Server is defined within the BuildForge system with certain properties, such as a default path. A Server Pool is a group of Servers with similar build properties (running the same operating system and with the same compilers available, for example). The BuildForge system can run a Project on a different Server in the same Pool when the default Server for the Project is busy.
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge servers
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Build Forge project execution Example Schedule
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Project/Process Putting it all together… Environment Get Src Compile Solaris Compile Linux Test Package DeployInit Compile Windows (main) Reusable Steps Compile Windows (ui) Link Windows (main, ui) Control Environment Notification Scheduling Log Analysis Tracking Analysis Pooling Threading Error/Cleanup
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Project/Process Putting it all together… Environment Get Src Compile Solaris Compile Linux DeployInit Compile z/OS (main) Reusable Steps TestPackage Compile z/OS (cobol, cics) Link (main, cics) Control Environment Notification Scheduling Log Analysis Tracking Analysis Pooling Threading
  • © 2008 IBM Corporation Page 36 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Ensure Traceability Among the Artifacts Create an automated, documented audit trail Release Build Artifacts Source Code Environment Changes Implemented with this Built using these In this Executable This Deployment Artifacts Which deployed these System on this Application Of this Tests Test Results Where these Produced these Build Scripts Which required these Requirements Implemented these Approved by Approval Development Pre-Prod Transparent process for turning source code into software Records of how software was released into production
  • © 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Summary The Build Forge architecture provides a robust, flexible foundation for build (and more) process automation. We have discussed Build Forge Architecture Projects Environments Security Servers Project Execution In the following labs we will dive into a Build Forge environment and see how to use these concepts.
  • © 2008 IBM Corporation Page 38 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • Java Garbage CollectionJava Garbage Collection Best Practices for Sizing and Tuning the Java HeapBest Practices for Sizing and Tuning the Java Heap
  • 22© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. ObjectivesObjectives • Overview • Selecting the Correct GC Policy • Sizing the Java heap • Questions/Answers
  • 33© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Garbage Collection PerformanceGarbage Collection Performance • GC performance issues can take many forms • Definition of a performance problem is user centric User requirement may be for: Very short GC “pause” times Maximum throughput A balance of both • First step is ensure that the correct GC policy has been selected for the workload type Helpful to have an understanding of GC mechanisms • Second step is to ensure heap sizing is correct • Third step us to look for specific performance issues
  • Selecting the Correct GC PolicySelecting the Correct GC Policy
  • 55© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Understanding Garbage CollectionUnderstanding Garbage Collection • Responsible for allocation and freeing of: Java objects, Array objects and Java classes • Allocates objects using a contiguous section of Java heap • Ensures the object remains as long as it is in use or “live” Determination based on a reference from another “live” object or from outside of the Heap • Reclaims objects that are no longer referenced • Ensures that any finalize method is run before the object is reclaimed
  • 66© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Object AllocationObject Allocation • Requires a contiguous area of Java heap • Driven by requests from: The Java application JNI code • Most allocations take place in Thread Local Heaps (TLHs) Threads reserve a chunk of free heap to allocate from Reduces contention on allocation lock Keeps code running in a straight line (fewer failures) Meant to be fast Available for objects < 512 bytes in size • Larger allocates take place under a global “heap lock” These allocations are one time costs – out of line allocate Multiple threads allocating larger objects at the same time will contend
  • 77© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Object Reclamation (Garbage Collection)Object Reclamation (Garbage Collection) • Occurs under two scenarios: An “allocation failure” An object allocation is requested and not enough contiguous memory is available A programmatically requested garbage collection cycle call is made to System.GC() or Runtime.GC() the Distributed Garbage Collector is running call to JVMPI/TI is made • Two main technologies used to remove the garbage: Mark Sweep Collector Copy Collector • IBM uses a mark sweep collector or a combination for generational
  • 88© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Global Collection PoliciesGlobal Collection Policies • Garbage Collection can be broken down into 2 (3) steps Mark: Find all live objects in the system Sweep: Reclaim unused heap memory to the free list Compact: Reduce fragmentation within the free list • All steps are in a single stop-the-world (STW) phase Application “pauses” whilst garbage collection is done • Each step is performed as a parallel task within itself • Four GC “Policies”, optimized for different scenarios -Xgcpolicy:optthruput optimized for “batch” type applications -Xgcpolicy:optavgpause optimized for applications with responsiveness criteria -Xgcpolicy:gencon optimized for highly transactional workloads -Xgcpolicy:subpools optimized for large systems with allocation contention
  • 99© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Parallel Mark Sweep Collector, with compaction avoidance Created to make use of additional processors on server systems Designed to increase performance for SMP and not degrade performance for uni-processor systems • Optimized for “Throughput” Best policy for “batch” type applications • Consists of a single “flat” Java heap: Parallel GC (Parallel GC (optthruputoptthruput)) 0 GB 2 GB Heap Base Heap Limit Heap Size LOA
  • 1010© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Parallelism achieved through the use of “GC Helper Threads” “Parked” set of threads that wake to share GC work Main GC thread generates the root set of objects Helper threads share the work for the rest of the phases Number of helpers is one less than the number of processing units So helper threads and main GC thread equals the number of processing units Configurable using -Xgcthreads GC Helper ThreadsGC Helper Threads
  • 1111© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Parallel Mark/Parallel Sweep view of GCParallel Mark/Parallel Sweep view of GC
  • 1212© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Reduces and makes more consistent the time spent inside Stop the World GC Reduction usually between 90 and 95% • Achieved by carrying out some of the STW work whilst application is running 1.4.2: Concurrent Marking 5.0: Concurrent Marking and Concurrent Sweeping • Slight overhead on thruput for greatly reduced STW times • Policy is ideal for systems with responsiveness criteria eg. Portal applications Concurrent GC (Concurrent GC (optavgpauseoptavgpause))
  • 1313© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Parallel and Concurrent Mark/SweepParallel and Concurrent Mark/Sweep Concurrent Kickoff
  • 1414© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Concurrent MarkConcurrent Mark –– hidden object issuehidden object issue • Higher heap usage…
  • 1515© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Higher heap usage… • …because not all garbage removed Concurrent MarkConcurrent Mark –– hidden object issuehidden object issue Dangling pointer!
  • 1616© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Similar in concept to that used by Sun and HP Parallel copy and concurrent global collects by default • Motivation: Objects die young so focus collection efforts on recently created objects Divide the heap up into a two areas: “new” and “old” Perform allocates from the new area Collections focus on the new area Objects that survive a number of collects in new area are promoted to old area (tenured) • Ideal for transactional and high data throughput workloads Generational and Concurrent GC (Generational and Concurrent GC (gencongencon)) 0 GB 2 GB Heap Base Heap LimitHeap Size LOA Nursery (new) Space Tenured (old) Space Allocate Survivor
  • 1717© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Allocate Space Survivor Space • Nursery is split into two spaces (semi-spaces) Only one contains live objects and is available for allocation Minor collections (Scavenges) move objects between spaces Role of spaces is reversed Nursery/Young Generation Survivor Space Allocate Space • Movement results in implicit compaction Nursery (new) Space Copy CollectionNursery (new) Space Copy Collection
  • 1818© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. SubpoolingSubpooling ((subpoolsubpool)) • Goals: Reduce allocation lock contention by distributing free memory into multiple lists Reduce allocation contention through use of atomic operations instead of a heap lock Prevent premature garbage collections by using a “best fit” (or closer to best fit) policy instead of address ordered • Ideal for very large SMP systems where large amounts data is being allocated … where there is heap lock contention
  • 1919© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Looking for Heap Lock ContentionLooking for Heap Lock Contention • All locks can be profiled using Java Lock Analyzer (JLA) http://www.alphaworks.ibm.com/tech/jla (AlphaWorks) Provides time accounting and contention statistics for Java and JVM locks Functionality includes: Counters associated with contended locks Total number of successful acquires Recursive acquires – times a thread acquires a lock it already owns Number of times a thread blocks because a monitor is already owned Cumulative time the monitor was held.
  • 2020© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. JLA Sample ReportJLA Sample Report System (Registered) Monitors %MISS GETS NONREC SLOW REC TIER2 TIER3 %UTIL AVER-HTM MON-NAME 87 5273 5273 4572 0 710708 18487 1 95408 JITC Global_Compile lock 9 6870 6869 631 1 113420 2976 0 11807 Heap lock 5 1123 1123 51 0 11098 286 1 248385 Binclass lock 0 1153 1147 5 6 1307 33 0 47974 Monitor Cache lock 0 46149 45877 134 272 36961 877 1 6558 JITC CHA lock 0 33734 23483 19 10251 6544 150 1 17083 Thread queue lock 0 5 5 0 0 0 0 0 9309689 JNI Global Reference lock 0 5 5 0 0 0 0 0 9283000 JNI Pinning lock 0 5 5 0 0 0 0 0 9442968 Sleep lock 0 1 1 0 0 0 0 0 0 Monitor Registry lock 0 0 0 0 0 0 0 0 0 Evacuation Region lock 0 0 0 0 0 0 0 0 0 Method trace lock 0 0 0 0 0 0 0 0 0 Classloader lock 0 0 0 0 0 0 0 0 0 Heap Promotion lock Java (Inflated) Monitors %MISS GETS NONREC SLOW REC TIER2 TIER3 %UTIL AVER-HTM MON-NAME 15 68 68 10 0 2204 56 2 11936405 test.lock.testlock1@A09410/A09418 2 42 42 1 0 186 5 0 300478 test.lock.testlock2@D31358/D31360 0 70 70 0 0 41 1 0 7617 java.lang.ref.ReferenceQueue$Lock@920628/920630
  • 2121© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. JLA: Fields in the reportJLA: Fields in the report
  • 2222© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Choosing the Right GC PolicyChoosing the Right GC Policy •Four GC “Policies”, optimized for different scenarios -Xgcpolicy:optthruput optimized for “batch” type applications -Xgcpolicy:optavgpause optimized for applications with responsiveness criteria -Xgcpolicy:gencon optimized for highly transactional workloads -Xgcpolicy:subpools optimized for large systems with allocation contention • How do I know whether to use “optavgpause” or “gencon”? Monitor GC activity Look for certain characteristics
  • 2323© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Monitoring GC ActivityMonitoring GC Activity • Use of Verbose GC logging only data that is required for GC performance tuning Graph Verbose GC output using GC and Memory Visualizer (GCMV) from ISA • Activated using command line options -verbose:gc -Xverbosegclog:[DIR_PATH][FILE_NAME],X,Y where: [DIR_PATH] is the directory where the file should be written [FILE_NAME] is the name of the file to write the logging to X is the number of files to Y is the number of GC cycles a file should contain • Performance Cost: (very) basic testing shows a 2% overhead for GC duration of 200ms eg. if application GC overhead is 5%, it would become 5.1%
  • 2424© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Important Characteristics for ChoosingImportant Characteristics for Choosing GC PolicyGC Policy • Rate of Garbage Collection High rates of object “burn” point to large numbers of transitional objects, and therefore the application may well benefit from the use of gencon • Large Object Allocations? The allocation of very large objects adversely affects gencon unless the nursery is sufficiently large enough. The application may well benefit from optavgpuse • Large heap usage variations The optavgpause algorithms are best suited to consistent allocation profiles Where large variations occur, gencon may be better suited • Rule of thumb: if GC overhead is > 10%, you’ve most likely chosen the wrong one
  • 2525© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Rate of Garbage CollectionRate of Garbage Collection optavgpause gencon • Gencon could handle a higher “rate of garbage collection” Completing the test quicker • Gencon had a smaller percentage of time in garbage collection • Gencon had a shorter maximum pause time
  • 2626© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Rate of Garbage CollectionRate of Garbage Collection • Gencon provides less frequent long Garbage Collection cycles • Gencon provides a shorter longest Garbage Collection cycle
  • 2727© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Large Object AllocationsLarge Object Allocations • (Very) Large Object allocations affects the gencon GC policy If object is larger than the Nursery size, the object is immediately tenured Removes the benefit of generational heaps Still has the additional overhead of running generational If object is fits in the nursery but fills it, frequent nursery collects will have to occur Too frequent nursery collects mean objects are likely to survive and need copying Copying is an expensive process • If (Very) Large Objects are being used, a sufficiently large enough nursery is required
  • Sizing the Java HeapSizing the Java Heap
  • 2929© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Sizing the Java HeapSizing the Java Heap • Maximum possible Java heap sizes • The “correct” Java heap size • Fixed heap sizes vs. Variable heap sizes • Heap Sizing for Generational GC
  • 3030© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Maximum Possible Heap SizeMaximum Possible Heap Size • 32 bit Java processes have maximum possible heap size Varies according to the OS and platform used Determined by the process memory layout • 64 bit processes do not have this limit Limit exists, but is so large it can be effectively ignored Addressability usually between 2^44 and 2^64 Which is 16+ TeraBytes
  • 3131© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • An Operating System process like any other application: Subject to OS and architecture restrictions 32bit architecture has an addressable range of: 2^32 which is 0x00000000 – 0xFFFFFFFF which is 4GB • Not all addressable space is available to the application The operating system needs memory for: The kernel The runtime support libraries Varies according to Operating System How much memory is needed and where that memory is located 0 GB 4 GB 0x0 0xFFFFFFFF 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Java Process Memory LayoutJava Process Memory Layout
  • 3232© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Memory Available to the Java ProcessMemory Available to the Java Process • On Windows: • On AIX: 0 GB 4 GB 0x0 0xFFFFFFFF 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Operating System Space Libraries 0 GB 4 GB 0x0 0xFFFFFFFF 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Kernel Libraries
  • 3333© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Java Process RestrictionsJava Process Restrictions • Not all Java Process space is available to the Java application The Java Runtime needs memory for: The Java Virtual Machine Backing resources for some Java objects This memory area as well as some other allocations, is part of the “Native” Heap Memory not allocated to the Java Heap is available to the native heap Available memory space – Java heap = native heap Effectively, the Java process maintains two memory pools
  • 3434© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. TheThe ““NativeNative”” HeapHeap • Allocated using malloc() and therefore subject to memory management by the OS • Used for Virtual Machine resources, eg: Execution engine Class Loader Garbage Collector infrastructure • Used to underpin Java objects: Threads, Classes, AWT objects, ZipFiles • Used for allocations by JNI code
  • 3535© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Native Heap available to ApplicationNative Heap available to Application • On Windows • On AIX (1.4.2 with small heaps) 0 GB 4 GB 0x0 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Operating System Space Libraries Java Heap 0xFFFFFFFF VM Resources Native Heap 0 GB 4 GB 0x0 0xFFFFFFFF 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Kernel LibrariesJava Heap VM Resources Native Heap
  • 3636© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Layout with Large Java Heaps on AIXLayout with Large Java Heaps on AIX • Applies to heaps > 1GB in size and Java 5.0 • Java heap becomes allocated using mmap() • Segments used start at 0xC and work downwards understanding memory layout important for monitoring 0 GB 4 GB 0x0 0xFFFFFFFF 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 Kernel LibrariesJava Heap VM Resources 0x7 Native Heap 0xD0x3
  • 3737© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. • Linux: • z/OS: Memory Layout for LinuxMemory Layout for Linux 0 GB 4 GB 0x0 2 GB 0x80000000 1 GB 3 GB 0x40000000 0xC0000000 KernelJava Heap 0xFFFFFFFF VM Resources Native Heap PAGE_OFFSETTASK_SIZE 0 GB 0x0 2 GB 0x7FFFFFFF 1 GB 0x40000000 Java Heap VM Resources
  • 3838© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Theoretical and Advised Max Heap SizesTheoretical and Advised Max Heap Sizes • The larger the Java heap, the more constrained the native heap • Advised limits to prevent native heap from becoming overly restricted, leading to OutOfMemoryErrors • Exceeding advised limits possible, but should be done only when native heap usage is understood Native heap usage can be measured using OS tools: Svmon (AIX), PerfMon (Windows), RMF (zOS) etc 1.8GB1.8GB/3GB 2.5GB3 GBHugemem Kernel Advised MaximumMaximum PossibleAdditional OptionsPlatform 1.3GB1.7GBz/OS 1.5GB1.8GBWindows 1.5GB2 GBLinux 2.5GB3.25 GBautomaticAIX
  • 3939© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Moving to 64bitMoving to 64bit • Moving to 64bit remove the Java heap size limit • However, ability to use more memory is not “free” 64bit applications perform slower More data has to be manipulated Cache performance is reduced 64bit applications require more memory Java Object references are larger Internal pointers are larger • Major improvements to this in Java 6.0 due to compressed pointers
  • 4040© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. TheThe ““correctcorrect”” Java heap sizeJava heap size • GC will adapt heap size to keep occupancy between 40% and 70% Heap occupancy over 70% causes frequent GC cycles Which generally means reduced performance Heap occupancy below 40% means infrequent GC cycles, but cycles longer than they needs to be Which means longer pause times that necessary Which generally means reduced performance • The maximum heap size setting should therefore be 43% larger than the maximum occupancy of the application Maximum occupancy + 43% means occupancy at 70% of total heap Eg. For 70MB occupancy, 100MB Max heap required, which is 70MB + 43% of 70MB
  • 4141© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. TheThe ““correctcorrect”” Java heap sizeJava heap size Long Garbage Collection Cycles To Frequent Garbage Collection Memory Time 70% 40% Heap Occupancy Heap Size
  • 4242© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Fixed heap sizes vs. Variable heap sizesFixed heap sizes vs. Variable heap sizes • Should the heap size be “fixed”? ie. Minimum heap size (-Xms) = Maximum heap size (-Xmx)? • Each option has advantages and disadvantages As for most performance tuning, you must select which is right for the particular application • Variable Heap Sizes GC will adapt heap size to keep occupancy between 40% and 70% Expands and Shrinks the Java heap Allows for scenario where usage varies over time Where variations would take usage outside of the 40-70% window • Fixed Heap Sizes Does not expand or shrink the Java heap
  • 4343© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Heap Expansion and ShrinkageHeap Expansion and Shrinkage • Act of heap expansion and shrinkage is relatively “cheap” • However, a compaction of the Java heap is sometimes required Expansion: for some expansions, GC may have already compacted to try to allocate the object before expansion Shrinkage: GC may need to compact to move objects from the area of the heap being “shrunk” • Whilst expansion and shrinkage optimizes heap occupancy, it (usually) does so at the cost of compaction cycles
  • 4444© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Conditions for Heap ExpansionConditions for Heap Expansion • Not enough free space available for object allocation after GC has complete Occurs after a compaction cycle Typically occurs where there is fragmentation or during rapid occupancy growth (ie, application startup) • Heap occupancy is over 70% Compaction unlikely • More than 13% of time is spent in GC Compaction unlikely
  • 4545© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Conditions for Heap ShrinkageConditions for Heap Shrinkage • Heap occupancy is under 40% • And the following is not true: Heap has been recently expanded (last 3 cycles) GC is a result of a System.GC() call • Compaction occurs if: An object exists in the area being shrunk GC did not shrink on the previous cycle • Compaction is therefore likely to occur
  • 4646© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Introduction toIntroduction to ––XmaxfXmaxf andand ––XminfXminf • The –Xmaxf and –Xminf settings control the 40% and 70% occupancy bounds -Xmaxf: the maximum heap space free before shrinkage (default is 0.6 for 60%) -Xminf: the minimum heap space before expansion (default is 0.3 for 70%) • Can be used to “move” optimum occupancy window if required by the application eg. Lower heap utilization required for more infrequenct GC cycles • Can be used to prevent shrinkage -Xmaxf1.0 would mean shrinkage only when heap is 100% free Would completely remove shrinkage capability
  • 4747© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Introduction toIntroduction to ––XmaxeXmaxe andand --XmineXmine • The –Xmaxe and –Xmine settings control the bounds of the size of each expansion step -Xmaxe: the maximum amount of memory to add to the heap size in the case of expansion (default is unlimited) -Xmine: the minimum amount of memory to add to the heap size in the case of expansion (default is 1MB) • Can be used to reduce/prevent compaction due to expansion Reduce expansions by setting a large -Xmine
  • 4848© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. GC Managed Heap SizingGC Managed Heap Sizing Long Garbage Collection Cycles To Frequent Garbage Collection Memory Time -Xminf -Xmaxf Heap Occupancy Heap Size Expansion (>= -Xmine)
  • 4949© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Fixed or Variable??Fixed or Variable?? • Again, dependent on application • For “flat” memory usage, use fixed • For widely varying memory usage, consider variable • Variable provides more flexibility and ability to avoid OutOfMemoryErrors Some of the disadvantages can be avoided: -Xms set to lowest steady state memory usage prevents expansion at startup -Xmaxf1 will remove shrinkage -Xminf can be used to prevent compaction before expansion -Xmine can be used to reduce expansions
  • 5050© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Nursery Tenured Fix both nursery and tenured space Allow them to expand/contract General Advice: Fix the new space size Size the tenured space as you would for a “flat” heap Heap Sizing for Generational GCHeap Sizing for Generational GC • Options are:
  • 5151© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. Sizing the NurserySizing the Nursery • “Copying” from Allocate to Survivor or to Tenured space is expensive Physical data is copied (similar to compaction with is also expensive • Ideally “survival” rates should be as low as possible Less data needs to be copied Less tenured/global collects that will occur • The larger the nursery: the greater the time between collects the less objects that should survive However, the longer a copy can potentially take • Recommendation is to have a nursery as large as possible Whilst not being so large that nursery collect times affect the application responsiveness
  • 5252© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. SummarySummary • GC Policy should be chosen according to application scenario • Java heap should ideally be sized for between 40 and 70% occupancy • Min=Max heap size is right for some applications, but not for others
  • 5353© 2008 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM. QuestionsQuestions • Please remind students to complete their session evaluation form • Thank you!!
  • D02D02 –– A Behind the Scenes Look at the Performance Improvements inA Behind the Scenes Look at the Performance Improvements in WPS/WESB v6.1WPS/WESB v6.1 Speaker Name Vijay K Peachimuthu/India/IBMVijay K Peachimuthu/India/IBM E-Mail vpeachim@in.ibm.comvpeachim@in.ibm.com Title Consulting IT ArchitectConsulting IT Architect
  • WebSphere Technical Conference 2008 Agenda Recent WPS Performance Improvement Themes Key Performance Enhancements in Previous WPS/WESB Versions B i P E i (BPE)Business Process Engine (BPE) SCA Performance Improvements Overall WPS Code Path length WESB S ifi P f I tWESB Specific Performance Improvements WBM v6.1 Performance Improvement Highlights Key Performance Enhancements in WPS/WESB v6.1 Business Process Engine (BPE) General WPS Runtime Areas of Investigation © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 2
  • WebSphere Technical Conference 2008 Recent WPS Performance Themes Core SCA invocation improvements M “h ” li i tiMessage “hop” elimination Eliminate over-reliance on messaging Reduce SCA overhead on system services Improve efficiency of generated codeImprove efficiency of generated code Optimize expensive interactions with EMF “Bread and butter” profile-driven performance analysis based improvements © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 3 p
  • WebSphere Technical Conference 2008 WPS v6.02 BPE Improvements Exploiting Materialized Views The BPC data model (schema) is optimized for create, retrieve, and d t ti d t id d li ti f d t i th d t bupdate operations and to avoid duplication of data in the database Navigation is very fast but queries against the model often require access to multiple tables Materialized Views enables “single table” performance levels for queries that access data from multiple tables © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 4
  • WebSphere Technical Conference 2008 Materialized Views With materialized views, complex queries become simple queries that are executed against a singlesimple queries that are executed against a single table only MV_1 WorkItem Process Instance Task Task Description Query Properties © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 5
  • WebSphere Technical Conference 2008 Exploiting Materialized Views with HTM Human task queries benefit the most from materialized views since human task queries are complex and materialized views improves complex database queries Starting with WPS v6.0.2.1g Requires “materialized views” support from associated DB provider Steps to configure C t th t l t bl f th t i li d iCreate the control table for the materialized views Create the table spaces for the materialized views Give the user that accesses Business Process Choreographer database (<BPEDB>) the rights that are required to administer the materialized views Define the materialized viewsDefine the materialized views Activate the changes by restarting the WPS server/cluster WPS Information http://www ibm com/support/docview wss?rs=2307&uid=swg27009623http://www.ibm.com/support/docview.wss?rs=2307&uid=swg27009623 DB2 Materialized Views Information (DB2 Infocenter) http://public.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=/com.ibm.websphere.ii.db2udb. nav.doc/navcontainers/iiypnavc tuning mqt.html © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 6 nav.doc/navcontainers/iiypnavc_tuning_mqt.html
  • WebSphere Technical Conference 2008 WPS v6.02 Key Enhancements Benefits from Materialized ViewsBenefits from Materialized Views 14000 14000 Query Response Time [ms] 97 90 100 DB Server CPU Utilization [%] 6000 8000 10000 12000 40 50 60 70 80 90 No optimizations [ms] 60 2000 4000 No optimizations Utilization [%] 22.5 0 10 20 30 No optimizations With materialized view s No optimizations With materialized view s Additional details can be found in the paper “Performance Tuning of Human Workflows Using Materialized Views” © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 7 Additional details can be found in the paper Performance Tuning of Human Workflows Using Materialized Views (see http://www-1.ibm.com/support/docview.wss?rs=2307&uid=swg27009623 )
  • WebSphere Technical Conference 2008 WPS v6.02 SCA Improvements MQ and MQJMS Bindings Direct access to MQ queues from SCA based imports and exports MQ bindings use MQ APIs to access MQ Series hosted queues MQJMS bindings use the version of the JMS API supported by MQ Series to access MQ Series hosted queuesSeries to access MQ Series hosted queues 2 to 2.2 times faster than MQ Link based bindings © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 8
  • WebSphere Technical Conference 2008 MQ Link Benchmark Topology JMS Consumer JMS Producer MQ Queue Manager JMS Consumer JMS Producer MQ Queue Manager JMS Queue JMS Import Mediation JMS Export JMS Queue JMS Queue JMS Queue JMS Import JMS Import MediationMediation JMS Export JMS Export JMS Queue JMS Queue DB2DB2 © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 9
  • WebSphere Technical Conference 2008 MQ JMS Benchmark Topology MQ Queue Manager JMS Consumer JMS Producer MQ Queue Manager JMS Consumer JMS Producer MQ JMS Import Mediation MQ JMS Export MQ JMS Import MQ JMS Import MediationMediation MQ JMS Export MQ JMS Exportpo tpo tpo t © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 10
  • WebSphere Technical Conference 2008 WPS v6.02 Performance Enhancements Web Services BindingsWeb Services Bindings Local Web Services Optimization If the target of the Web Services import binding is hosted locally in the same application server the performance can be further improved by exploiting theapplication server, the performance can be further improved by exploiting the optimized communication path provided by the Web container. Normally requests from the Web Services clients are sent through the network connection between the client and the service provider. For local Web Services calls, however, WAS offers a direct communication channel, , bypassing the network layer completely. Steps to enable – Set Web Container custom propery “enableInProcessConnections” to true – Do not use a wildcard for the host name of the Web Container port Use the hostname or IPDo not use a wildcard for the host name of the Web Container port. Use the hostname or IP address instead – Use “localhost” instead of host name in the Web Services client binding – Make sure there is not an entry for your server hostname and IP address in your server’s hosts file for name resolution © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 11
  • WebSphere Technical Conference 2008 Contact Manager on Windows Local versus Remote BindingsLocal versus Remote Bindings Messages Local synchronous SCA bindings are 3 1 ti f t th t ContactManager - Windows 2000 Local versus Remote Bindings 3.1 times faster than remote synchronous SCA bindings Optimized ocal Web Services bindings are faster than remote Web Services bindings Throughput Results Local versus Remote Bindings 100% 200 250 Throughput Results 198 BTPS – local SCA binding ~64 BTPS – remote SCA bindings 110 BTPS – local optimized Web Services 100% 100% 100%100 150 BTPS 1 WPS JVM (WS Opt) 1 WPS JVM Services 96 BTPS – local Web Services (no optimization) 88 BTPS – remote Web Services bindings Benchmark Details 100% 0 50 2 WPS JVMs Benchmark Details Windows Host System – 1 CPUs @ 2.8 GHz 1 SCA Sync WebServices © Copyright IBMCorporation 2005, 2008. All rights reserved WPS SystemCPUUtilization Above Each Bar © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 12
  • WebSphere Technical Conference 2008 JMS Binding Improvements since v6.01 Invocation Hop Elimination JMS Import intra-module invocation hop elimination JMS Export intra-module invocation of downstream component hop elimination © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 13
  • WebSphere Technical Conference 2008 SCA JMS Binding One-way WPS 6 0 1WPS 6.0.1 M1 SCA M2 MDB ServiceSIBusMessageBean SCA Runtime SCA Runtime MDB Export.E1EIS MDB ServiceSIBusMessageBean J2C Adapter sca/M1/ActivationSpec J2C Adapter (ActivationSpec) J2C Adapter sca/M/ActivationSpec g sca/M1/import/I1* sca/M1 Forward Routing Path SIBus SCA.SYSTEM.<cellname>.Bus JMS Queue sca/M Forward Routing Path sca/M/export/E1* SIBus SCA.SYSTEM.<cellname>.Bus © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 14
  • WebSphere Technical Conference 2008 Hop Reduction: SCA JMS Binding One-way WPS 6 0 1 1WPS 6.0.1.1 M1 SCA M2 SCA Runtime SCA Runtime MDB Export.E1EIS J2C Adapter (ActivationSpec) JMS Queue © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 15
  • WebSphere Technical Conference 2008 WESB v6.02 Enhancements Deferred Parsing Deferred Parsing enables WESB to “avoid SDO inflation (deserializing) in cases where it is not required”( g) q – Assuming incoming SDO is in a serial form that SCA understands – No benefit if you convert from an incoming custom format to SDO then serialize using SCA serialization functionality. If you get to a fully de-serialized SDO, pass the de-serialized SDO to the mediation module using synchronous bindings Benefits flows where the primitives do not explicitly or implicitly access the data via the SDO API Primitives that benefit from deferred parsing – Route on header – XSLT Transformation – Endpoint Lookup without Xpath user properties Primitives that require full parsingq p g – Custom mediation – Route on body – Any mediation with an Xpath expression against the message body – RPC Encoded WSDL © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 16 RPC Encoded WSDL – Note: Deferred parsing is achieved by using WID generated WSDL
  • WebSphere Technical Conference 2008 WBM v6.1 Performance Improvements Highlights Significant throughput improvements in WPS v6.1 illustrated by the core workload benchmarks SOABench automated mode improvements up to 132% and SOABench manual mode improvements up to 82%, relative to WPS v6.0.2 Banking throughput improved by 33% relative to WPS v6 0 2 and 3 9 times higher than WPS 6 0 0 0Banking throughput improved by 33% relative to WPS v6.0.2 and 3.9 times higher than WPS 6.0.0.0 Contact Manager throughput improved by up to 69% relative to WPS v6.0.2 and 6.2 times higher than WPS v6.0.0.0 WPS v6.1 SMP scaling data that demonstrates outstanding vertical scaling; SOABench automated mode achieved 7 1x scaling on an 8-way AIX system and 13 8x scaling on a 16-automated mode achieved 7.1x scaling on an 8-way AIX system and 13.8x scaling on a 16- way AIX system WPS v6.1 clustering data that shows near-linear horizontal scaling with up to 8 nodes; Contact Manager achieved 7.3x scaling on an 8 node topology WESB v6.1 using Web Services bindings delivers throughput improvements from 94% to 258% relative to WESB v6.0.2 WESB v6.1 using JMS bindings delivers throughput improvements from 72% to 198% relative to WPS v6 0 2relative to WPS v6.0.2 WESB v6.1 clustering data shows linear horizontal scaling up to 4 nodes; the Transform Value workload delivered 4x scaling on a 4 node topology © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 17
  • WebSphere Technical Conference 2008 WBM v6.1 Performance Improvements Highlights WPS and WESB v6.1 dramatically improved large message/object capability across all scenarios; WESB achieved up to an 8 times improvement and WPS achieved up to a 5 times improvement Performance data for i5/OS is introduced in this report. WESB v6.1 demonstrates the ability to process up to 1,400 messages per second WPS v6.1 delivers throughput up to 418 transactions per second with the SOABench automated mode benchmark on a 4-way system WebSphere Adapters The WA SAP Adapter achieves a 2 times improvement in throughput over v6.0.2 when using the BAPI module for request processing The WA SAP Adapter achieves a 45% improvement over v6.0.2 when using the ALE module forThe WA SAP Adapter achieves a 45% improvement over v6.0.2 when using the ALE module for event delivery The WA Flat File Adapter achieves a 2 times improvement in throughput over v6.0.2 for event delivery. The WA Flat File Adapter was able to successfully process a 32 GB file when using SplitBySize capabilitycapability WebSphere Business Monitor WBM v6.1 achieved almost a 2 times improvement in throughput over v6.0.2 WBM v6.1 achieved a event processing rate of almost 2,000 events per second while sustaining © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 18 WBM v6.1 achieved a event processing rate of almost 2,000 events per second while sustaining 100,000 active process instances
  • WebSphere Technical Conference 2008 WID Performance Improvements Significant improvement in authoring productivity since WID V6.0.2.0 Build response time reduced by 65% Build memory usage reduced by 60% Publish response time reduced by 40% Significantly reduced memory leaks in WIDg y y © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 19
  • WebSphere Technical Conference 2008 Key Enhancements in WPS v6.1 Business Process Engine G l WPS E h tGeneral WPS Enhancements © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 20
  • WebSphere Technical Conference 2008 WPS v6.1 Performance Enhancements Business Process EngineBusiness Process Engine Key BPE Enhancements New “High Speed” Business Process Navigation Engine for Long Runningg p g g g g Processes – Previous engine relied on JMS messages – Improved engine leverages WAS “async beans” Improved engine also caches data and leverages server affinity to associate to– Improved engine also caches data and leverages server affinity to associate to the server that holds the cache Data compression during serialization – Significantly reduces the size of XML business data stored and retrieved from theg y BPC database Materialized views for “Task List Queries” – Introduced in WPS v6.021 © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 21
  • WebSphere Technical Conference 2008 WPS v6.1 Key Performance Enhancements Business Process Engine (BPE)Business Process Engine (BPE) Performance of the Business Process Engine (BPE) is significantly improved in WPS v6.1.0.1 14 Business Process Performance 8 10 12 6 1 (hi speed)2 4 6 SCA async EJB Web service JMS POJO 6.0.2 6.1 (classic) 6.1 (hi-speed) 0 2 © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 22
  • WebSphere Technical Conference 2008 WPS v6.1 Key Performance Enhancements General Enhancements to the Common RuntimeGeneral Enhancements to the Common Runtime Exploitation of WAS v6.1 Usage of Java 5 Improvements in Synchronous JCA performancep y p 58% improvement in WPS v6.1 over WPS v6.02 for the ContactManager benchmark Improvements in Synchronous SCA performance Over 60% improvement in WPS v6.1 over WPS v6.02 for the ContactManager benchmark SMP Scaling Optimizations Illustrated by Travel Booking 8-way results SIBus File Store Use the file system instead of a database to hold messagesUse the file system instead of a database to hold messages Configuration used for the Banking benchmark results in the WPS v6.1 Performance Report Memory and Path Length Optimization Reduce memory usage to support larger objects and messagesReduce memory usage to support larger objects and messages Additional BPC and SCA path length reductions © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 23
  • WebSphere Technical Conference 2008 WPS v6.1 Key Performance Enhancements WID EnhancementsWID, Enhancements Utilized shared resource management system for all of the SCA component builders and validatorsp Enhancements to the WID Indexer and Resolver Caching of widgets used by the WID Indexer Reduced number of file copies that the Web ServicesReduced number of file copies that the Web Services deployment builder performed Leverage Java 5 and the GENCON garbage collector WID v6.02 used Java 1.4.2 J9 © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 24
  • WebSphere Technical Conference 2008 WESB v6.1 Performance Enhancements Native Form Re-use Incoming native data streams (non-SCA streams) will be “re-used” if the associated mediation does not perform data or object modifications Applies to “routing only” mediationspp g y Not applicable to “custom mediations” even if the “custom mediation” does not modify the data/object © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 25
  • WebSphere Technical Conference 2008 WebSphere Business Monitor v6.1 Monitor will re-order CBE that arrive out of order so more binding options with Monitor are enabled Improvements in the data representation of BPEL headers and application data inp p pp CBEs Support for previous formats preserved Over 2 times improvement using this new data representation New multi-processor mode called Serial Multi-Threaded (SerialMT) that achieves around an improvement based on the number of processors beyond 1 when compared with the previous SerialST implementation Support for previous operation preserved SerialMT is heavier per transaction so SerialST will still be faster on single processor host systems Remote Model Logic removes CPU contention between the “model logic” function and the “moderator” function when the two functions are run on different host systemssystems “Required Events Only” improves BPEL throughput Monitor throughput is reduced because of additional load on WPM © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 26
  • WebSphere Technical Conference 2008 Areas of Investigation in SOA Performance Remaining Pain Points “The cost of dynamicity” – Dynamic invocations (SCA)y ( ) – Dynamic data (SDO) – Process Engine (BPEL) More specificallyp y Transformations to/from wire format (XML e.g.) to object format (SDO) “Interpreted” BPEL engine navigation BPEL engine state management for long-running flowsg g g g Side effects of programming model – Pass by value across SCA modules © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 27
  • WebSphere Technical Conference 2008 Summary Recent WPS Performance Improvement Themes Key Performance Enhancements in Previous WPS/WESB Versions B i P E i (BPE)Business Process Engine (BPE) SCA Performance Improvements Overall WPS Code Path length WESB S ifi P f I tWESB Specific Performance Improvements WBM v6.1 Performance Improvement Highlights Key Performance Enhancements in WPS/WESB v6.1 Business Process Engine (BPE) General WPS Runtime Areas of Investigation © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 28
  • WebSphere Technical Conference 2008 Questions Please remind students to complete their session evaluation form Thank you!! © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 29
  • WebSphere Technical Conference 2008 Credits Gary Hunt - gthunt@us.ibm.com © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 30
  • WebSphere Technical Conference 2008 Hindi Thai Gracias Ob i d Russian Spanish Traditional Chinese Thank You Obrigado D k Brazilian Portuguese Arabic Merci Grazie Dank e French German Italian Simplified Chinese Tamil Korean © 2008 IBM Corporation Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 31 Japanese
  • Tuning WebSphere App Server with WhatTuning WebSphere App Server with What You Already HaveYou Already Have Session Number: D03Session Number: D03 Kevin TobinKevin Tobin –– kevintob@au1.ibm.comkevintob@au1.ibm.com
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 2 ObjectivesObjectives To demonstrate you do not need high-end, expensive tools to “adequately” tune WebSphere. • Learn how to make the WebSphere Application Server perform to its potential without additional IT budget for tools. • Learn about different performance testing scenarios including performance data collection and analysis. • Learn about common causes of performance issues. • Learn WebSphere tuning and service tips. • Learn issues that can be caused due to load distribution, caching and different infrastructures.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 3 AgendaAgenda • Intro to tools OS tools, Application tools, Free Tools. • Scenarios Performance testing scenarios, collecting and analysing data, common causes of performance issues. • Tuning JVM memory tuning, Connect pool tuning, DB problem analysis. • Concerns Load distribution, Caching, Infrastructure.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 4 Intro to toolsIntro to tools • Intro to tools OS tools, Application tools, Free Tools • Scenarios Performance testing scenarios, collecting and analysing data, common causes • Tuning JVM memory tuning, Connect pool tuning, DB problem analysis • Concerns Load distribution, Caching concerns, Infrastructure concerns.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 5 ToolsTools –– Operating SystemOperating System Network monitoring • ipconfig / netstat / iostat / Management Console • snoop / iptrace / tcpdump / nettl / Network Monitor Process monitoring • top / ps / Task Manager • free / vmstat / Management Console Disk Monitoring • df / Disk Manager
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 6 ToolsTools –– JVM AnalysisJVM Analysis PMAT (Pattern Modeling Analysis Tool) IBM HeapAnalyzer / HAT (Heap Analysis Tool) IBM Thread and Monitor Dump Analyzer IBM Request and Trace Analyzer
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 7 JVMJVM -- PMATPMAT Pattern Modeling Analysis Tool developed by Jinwoo Hwang from WAS technical support. Features: GC analysis GC table view Allocation failure summary GC usage summary GC duration summary GC graph view GC pattern analysis Recommends key configurations based on pattern modeling of Java heap usage Download http://www.alphaworks.ibm.com/tech/pmat
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 8 Sample Verbose GC OutputSample Verbose GC Output AF[1]: Allocation Failure. need 1544 bytes, 0 ms since last AF> <AF[1]: managing allocation failure, action=1 (0/3983128) (209640/209640)> <GC(1): GC cycle started Tue Apr 24 10:49:58 2001> <GC(1): freed 1421672 bytes, 38% free (1631312/4192768), in 9 ms> <GC(1): mark: 8 ms, sweep: 1 ms, compact: 0 ms> <GC(1): refs: soft 0 (age >= 32), weak 17, final 16, phantom 0> <AF[1]: completed in 10 ms>
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 9 JVMJVM –– PMATPMAT
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 10 JVMJVM –– HeapAnalyzerHeapAnalyzer (1)(1) •HeapAnalyzer provides: List of Java heap leak suspects. Recommendation of the size of kCluster. List of gaps among allocated objects/classes/arrays. Java objects/classes/arrays search engine. List of objects/classes/arrays by: •Type name •Object name •Address •Size ( including children ) •Frequency List of available heap spaces by size. Tree view of Java heap dump. Supports persistence of processed heap dumps. JRE 1.3.1, or above ( including IBM & SUN).
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 11 JVMJVM –– HeapAnalyzerHeapAnalyzer (2)(2)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 12 JVMJVM –– HeapAnalyzerHeapAnalyzer (3)(3)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 13 JVMJVM –– HeapAnalyzerHeapAnalyzer (4)(4)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 14 JVMJVM –– HeapAnalyzerHeapAnalyzer (5)(5)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 15 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (1)(1) Java Core Dump Analyzer V2.5 (WAS 5.1 & up) JRE 5 Identifies hang suspects ( HP, SOLARIS, IBM 1.3.1 – 6.0 JVM’s) Can report on multiple dump files. Analyzes each thread and provides diagnostic information, such as: • Current thread information, • The signal that caused the javacore • Maximum Java heap size • Initial Java heap size • Garbage collector & Allocation failure counter • Free and allocated Java heap size • Number of runnable threads, total number of threads, number of monitors locked, and deadlock information. Compares all monitor information in javacore and detects deadlock and resource contention or monitors bottlenecks, if there are any.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 16 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (2)(2) Thread States
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 17 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (3)(3)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 18 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (4)(4)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 19 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (4)(4)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 20 JVMJVM –– Thread Monitor Dump AnalyzerThread Monitor Dump Analyzer (5)(5)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 21 JVMJVM --Trace and Request Analyzer (TRA)Trace and Request Analyzer (TRA) This tool processes WebSphere Application Server JVM trace files Allows you to find delays and possible hangs in the application server. The tool reads HTTP plug-in trace files. User definable thresholds. The tool parses method call tree, time stamp, process ID, thread ID, trace entry, trace exit, method entry and method exit information. Parsing information is customizable. TRA • Calculates actual times spent in each method and trace. • Finds delays longer than the threshold. • Sorts the delay list by delay time. • Displays a list of clocked trace entries.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 22 JVMJVM --Trace and Request Analyzer (TRA)Trace and Request Analyzer (TRA) Trace method analysis
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 23 JVMJVM --Trace and Request Analyzer (TRA)Trace and Request Analyzer (TRA) HTTP request analysis
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 24 JVMJVM --Trace and Request Analyzer (TRA)Trace and Request Analyzer (TRA) Trace gap analysis
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 25 Other analysis tools: WireShark • Open source, network monitoring /packet sniffing ktmatu Relax Webalizer Xtended AWStats Network & HTTP Log ToolsNetwork & HTTP Log Tools
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 26 WireSharkWireShark • Popular network protocol analyzer Live capture and offline analysis. Standard three-pane packet browser. Multi-platform: Runs on Windows, Linux, OS X, Solaris,and many others. Captured network data can be browsed. Read/write many different capture file formats. Capture files compressed with gzip. Live data can be read from Ethernet, IEEE 802.11, PPP/HDLC, ATM, Bluetooth, USB, Token Ring, Frame Relay, FDDI, and others. Decryption support for many protocols, including IPsec, ISAKMP, Kerberos, SNMPv3, SSL/TLS, WEP, and WPA/WPA2. Output can be exported to XML, PostScript®, CSV, or plain text. • Download from http://www.wireshark.org/
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 27 WireSharkWireShark configurationconfiguration
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 28 WireSharkWireShark Capture OptionsCapture Options
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 29 ktmatuktmatu RelaxRelax • Relax is a free specialized web server log analysis tool for referrer information processing written in PERL. http://ktmatu.com/software/relax/ • It answers the question: "Which search engines, are directing traffic to this site?". • Relax reports the number of users you are getting from the leading search engines. • Relax generates lists of URLs directing to your pages with a count of the number of times each referring URL was detected.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 30 WEBALIZER XTENDEDWEBALIZER XTENDED • HTTP logs analyzer • Statistics for all hits, number of requests, percentage requests and "HTTP 404 Not Found“ - error URL's. • Automatic sorting and incremental or full log file processing. • Additional response code pie chart. • Fully user definable colors for all statistics. • User definable table and chart borders (3D-effects). • HTTP response codes with additional percentage hits.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 31 WEBALIZER XTENDEDWEBALIZER XTENDED Example output
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 32 WEBALIZER XTENDEDWEBALIZER XTENDED Example output
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 33 AWSTATSAWSTATS • AWStats is a feature rich, HTTP log analyzer. • AWStats generates web, streaming, ftp or mail server statistics, graphically. • The log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 34 AWSTATSAWSTATS • Number of visits, and number of unique visitors. • Visit durations and last visits. • Authenticated users, and last authenticated visits. • Days of week and rush hours: pages, hits, KB for each hour and day of week. • Hosts list, last visits and unresolved IP addresses list. • Most viewed, entry and exit pages. • Files types. • Web compression statistics (for mod_gzip or mod_deflate).
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 35 AWSTATS ( continued )AWSTATS ( continued ) • OS used (pages, hits, KB for each OS). • Browsers used (pages, hits, KB by browser, version). • Visits of robots. • Worms attacks. • HTTP errors (e.g. Page Not Found with last referrer). • Other personalized reports based on URL. • Number of times a site is “added to favorites“.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 36 AWSTATS ( continued )AWSTATS ( continued )
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 37 Apache JMeterApache JMeter • Popular load test performance tester Open source Java desktop application. Originally developed by Stefano Mazzocchi. Designed for functional/load/performance/stress testing. Extensible… write your own test component. Simulate heavy load (application, server and network). Gives instant visual feedback. Distributed testing support. Various protocols - HTTP, FTP, JDBC, JMS, LDAP, SOAP. Multi-platform. Full multi-threading framework. Caching and offline analysis/replaying of test results. • Download from http://jakarta.apache.org/jmeter/
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 38 Apache JMeterApache JMeter • Application Overview GUI, command line interface. Tests can be run and analyzed interactively, or run in batch mode and analyzed offline. Test plans consist of: •Thread groups: organize threads of execution. •Samplers: sends requests to a server. •Logical controllers : control test plan flow (loops, conditionals,ordering, etc.). •Listeners: record, summarize and display record and response data. •Timers: introduce delays in test plan. •Assertions: assert facts about responses, for functional testing. •Configuration elements. •Pre-processors and post-processors.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 39 Apache JMeterApache JMeter • Testing Process
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 40 Apache JMeterApache JMeter –– Test Plan ElementsTest Plan Elements
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 41 ScenariosScenarios • Intro to tools OS tools, Application tools, Free Tools. • Scenarios Performance testing scenarios, collecting and analysing data, common causes. • Tuning JVM memory tuning, Connect pool tuning, DB problem analysis. • Concerns Load distribution, Caching concerns, Infrastructure concerns.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 42 ScenariosScenarios • Scenarios What you should know before you start. Performance testing scenarios. Collecting performance test data. Analysis of performance test data. Common causes of performance issues.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 43 ScenariosScenarios –– PrePre--knowledgeknowledge • What are the business policies? Security policies • Access restrictions? • Sniffer policy? • Privacy policy? Change control • Who has access? • Lead times? • Exceptions? • Can you deploy investigation tools? Monitoring • What is being monitored? • How are issues responded to?
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 44 ScenariosScenarios -- Performance testingPerformance testing • Load generation tools can assist (JMeter). Remember to monitor: • Network and Network Interfaces. • Application Interfaces. • Database. Requires scheduled time and special data on a production system Use the same mix of transactions as seen in production.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 45 ScenariosScenarios -- Performance testing (cont)Performance testing (cont) • Soak testing (JMeter) Good for finding leaks (PMAT). Detecting external system events that potentially impact performance. • Code path testing (JMeter) Targets a particular operation in isolation. Can find operational optimizations. Remember to monitor • The network, • Application interfaces, • Database Should only be done in a test environment.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 46 ScenariosScenarios -- Collecting dataCollecting data • Logs Ensure logging is appropriate and without errors. • Tracing Be aware of any tracing enabled during performance tests. • Dumps (thread and heap) CPU stays low even with heavy loading. Tool such as heap and thread analyzers help. May not be permitted on production systems.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 47 ScenariosScenarios -- Collecting data (cont)Collecting data (cont) • Monitoring Is monitoring enabled? What can be derived from what is being monitored? • Configuration Backup the current system configuration before any testing. Look for abnormal settings (heap, pools, cache, etc.) • Time Ensure time is synchronised on all systems. • Manually check even is NTP is in use. Ensure you know the exact time of tests.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 48 ScenariosScenarios -- Analysis of dataAnalysis of data • Memory Analysis • Log Analysis • Heap / Thread Analysis
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 49 ScenariosScenarios -- Memory AnalysisMemory Analysis • What type of JVM is in use? Java 1.4 (Flat( IBM) or Generational (Sun)) Java 1.5 (Generational) Sun HotSpot (Generational +) • Analyze VerboseGC (PMAT) How often is a compact being done? How often is a sweep being done? Is there a lot of pinned and dosed objects? What is the steady state heap size after startup? • Add flag to remove explicit garbage collects Explicit GC always performs a compact. Java flag (-XX:+DisableExplicitGC)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 50 ScenariosScenarios -- Log AnalysisLog Analysis • What logs are available? SystemOut.log • Are there warnings or errors? SystemErr.log • Should be small. What errors are being reported?. System_stdout.log and System_stderr.log • Should be empty trace.log (is this production?) activity.log • Requires LogAnalyzer to read • Is it being monitored (at all)? FFDC • Number? • Cleanup?
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 51 ScenariosScenarios -- Heap / Thread AnalysisHeap / Thread Analysis • Heap Analysis Limited value for WAS tuning. Look for p & k objects outside the designated clusters. • Thread Analysis Look for multiple threads waiting for a single resource. Look for large numbers of idle threads.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 52 ScenariosScenarios –– Common issuesCommon issues • Network Too narrow a pipe (ifconfig / iostat) Intervening devices like firewalls ( nmap )
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 53 ScenariosScenarios –– Common issues (cont.)Common issues (cont.) • Database Connection pool • Check using TPV Poorly constructed queries • DB2 analyser Non-indexed data Cleanup not being performed • Runstats • Connection Pool Too large (Server swamped) TPV Too small (Server starved) TPV
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 54 ScenariosScenarios –– Common issues (cont.)Common issues (cont.) • Memory Initial heap too low (slow startup) (-verbosgec / PMAT) Max heap too large (long gc’s) (-verbosgec / PMAT) • Keep-alive Too short (multiple re-connects from same client) Too long (reduces max system throughput) • DynaCache Check if its being used and enquire if not. Not warmed on startup (CacheMonitor) Too small (CacheMonitor) Invalidations (all, timed) • Prepared Statement Cache Not tuned (TPV)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 55 TuningTuning • Intro to tools OS tools, Application tools, Free Tools • Scenarios Performance testing scenarios, collecting and analysing data, common causes • Tuning JVM memory tuning, Connect pool tuning, DB problem analysis • Concerns Load distribution, Caching concerns, Infrastructure concerns.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 56 TuningTuning • Tuning JVM memory tuning Connection pool tuning DB communication problem analysis
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 57 TuningTuning -- JVM memoryJVM memory • Requires a JVM memory tool -verboseGC Tivoli Performance Viewer Other PMI tool, (ITCAM , etc..) PMAT HeapAnalyzer • What to look for: Long garbage collects (max heap too large). Memory leaks (available heap shrinks). Frequent garbage compacts (heap too small). Explicit garbage collects (should be discouraged). Initial startup size (guide for initial heap size). Average heap size after mark-sweep (guide for max heap). k and p cluster sizes (tune iteratively).
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 58 TuningTuning -- Connection poolsConnection pools • Tools required TPV Other PMI tool, (ITCAM , etc..) Application logging Netstat • Web Container pools Number of concurrent requests. Average request response time. Average data volume returned. • EJB Container pools Number of concurrent operations. Average operation response time. Average data volume communicated. • DB Connections (covered next slide)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 59 TuningTuning -- DB communicationsDB communications • Tools required TPV Other PMI tool, (ITCAM , etc..) SQL logs netstat • Required Information Expected peak operations (guide to max connections) Expected ramp-up and ramp-down (guide to reap time) Set min connections to zero • Firewalls !!! • OK for slow ramp-up • Only set non-zero in exceptional circumstances SQL optimization (long requests, indexing)
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 60 ConcernsConcerns • Intro to tools OS tools, Application tools, Free Tools • Scenarios Performance testing scenarios, collecting and analysing data, common causes • Tuning JVM memory tuning, Connect pool tuning, DB problem analysis • Concerns Load distribution, Caching concerns, Infrastructure concerns.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 61 ConcernsConcerns • Concerns Load distribution Infrastructure concerns Caching concerns
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 62 ConcernsConcerns -- Load distributionLoad distribution • Active-active • Active-passive • Cell topology - covered next
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 63 ConcernsConcerns -- Infrastructure concernsInfrastructure concerns • Cell Topology - Gold Standard? • Multiple Data Centres • Wide Area Communication • Backup and Recovery
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 64 ConcernsConcerns -- CachingCaching • Startup • Size in memory • Disk offload speed • Invalidation • Cache Distribution
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 65 ReferencesReferences • Sources
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 66 SummarySummary • We discussed existing and free tools that can be used to help tune WebSphere. • We discussed different performance testing scenarios. • We discussed common causes of performance issues. • We discussed WebSphere tuning and service tips. • We discussed possible issues caused by load distribution, caching and infrastructures. You do not need high-end, expensive tools to “adequately” tune WebSphere.
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 67 QuestionsQuestions Please complete your session evaluation!
  • © 2008 IBM Corporation WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 68 Weak/Soft referencesWeak/Soft references •Weak references •A weak reference, simply put, is a reference that isn't strong enough to force an object to remain in memory. Weak references allow you to leverage the garbage collector's ability to determine reachability for you, so you don't have to do it yourself. You create a weak reference like this: •WeakReference weakWidget = new WeakReference(widget); and then elsewhere in the code you can use weakWidget.get() to get the actual Widget object. Of course the weak reference isn't strong enough to prevent garbage collection, so you may find (if there are no strong references to the widget) that weakWidget.get() suddenly starts returning null. •To solve the "widget serial number" problem above, the easiest thing to do is use the built-in WeakHashMap class. WeakHashMap works exactly like HashMap, except that the keys (not the values!) are referred to using weak references. If a WeakHashMap key becomes garbage, its entry is removed automatically. This avoids the pitfalls I described and requires no changes other than the switch from HashMap to a WeakHashMap. If you're following the standard convention of referring to your maps via the Map interface, no other code needs to even be aware of the change. •Reference queues •Once a WeakReference starts returning null, the object it pointed to has become garbage and the WeakReference object is pretty much useless. This generally means that some sort of cleanup is required; WeakHashMap, for example, has to remove such defunct entries to avoid holding onto an ever- increasing number of dead WeakReferences. •The ReferenceQueue class makes it easy to keep track of dead references. If you pass a ReferenceQueue into a weak reference's constructor, the reference object will be automatically inserted into the reference queue when the object to which it pointed becomes garbage. You can then, at some regular interval, process the ReferenceQueue and perform whatever cleanup is needed for dead references. •Different degrees of weakness •Up to this point I've just been referring to "weak references", but there are actually four different degrees of reference strength: strong, soft, weak, and phantom, in order from strongest to weakest. We've already discussed strong and weak references, so let's take a look at the other two. •Soft references •A soft reference is exactly like a weak reference, except that it is less eager to throw away the object to which it refers. An object which is only weakly reachable (the strongest references to it are WeakReferences) will be discarded at the next garbage collection cycle, but an object which is softly reachable will generally stick around for a while. •SoftReferences aren't required to behave any differently than WeakReferences, but in practice softly reachable objects are generally retained as long as memory is in plentiful supply. This makes them an excellent foundation for a cache, such as the image cache described above, since you can let the garbage collector worry about both how reachable the objects are (a strongly reachable object will never be removed from the cache) and how badly it needs the memory they are consuming. •Phantom references •A phantom reference is quite different than either SoftReference or WeakReference. Its grip on its object is so tenuous that you can't even retrieve the object -- its get() method always returns null. The only use for such a reference is keeping track of when it gets enqueued into a ReferenceQueue, as at that point you know the object to which it pointed is dead. How is that different from WeakReference, though? •The difference is in exactly when the enqueuing happens. WeakReferences are enqueued as soon as the object to which they point becomes weakly reachable. This is before finalization or garbage collection has actually happened; in theory the object could even be "resurrected" by an unorthodox finalize() method, but the WeakReference would remain dead. PhantomReferences are enqueued only when the object is physically removed from memory, and the get() method always returns null specifically to prevent you from being able to "resurrect" an almost-dead object. •What good are PhantomReferences? I'm only aware of two serious cases for them: first, they allow you to determine exactly when an object was removed from memory. They are in fact the only way to determine that. This isn't generally that useful, but might come in handy in certain very specific circumstances like manipulating large images: if you know for sure that an image should be garbage collected, you can wait until it actually is before attempting to load the next image, and therefore make the dreaded OutOfMemoryError less likely.
  • Yogesh Bhatia SDP Introduction and Architecture
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AGENDA SDP INTRODUCTION CURRENT MARKET REQUIREMENTS SDP RELEVANCE IBM COMPONENTS IBM SOLUTION ARCHITECTURE COMPONENTS DISCUSSION
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Motivation Subscribes: 2M to 100M ARPU: $30 to $7
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore The Opportunity
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Delivery Platform - Introduction ! " # " $ % & ' ( ) “A common architecture for the creation and delivery of services across multiple network types and service enabling platforms” ' * ( ) ' ( )+ ' " "
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Market Context – Service Creation Current Process Commercial Launch Soft LaunchTrials Proof of concept Ideation – Today As Is Service 18monthsto2years Ideas source: Internal idea idea idea Long testing and integration period due to proprietary solutions Little user interaction for market feedback Services developed in silos Little reuse Leads to delays in launch time
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Market Context – Service Creation Innovation Process Commercial Launch Soft LaunchTrials Proof of concept Ideation Service Ideas source: Internal, partners and customers Weekstomonths Breadth Tomorrow – To Be idea idea idea idea idea idea idea idea idea idea idea idea idea Modelling and Prototype designed for iterative development 2Use of exposed common capabilities to enable rapid service assembly 3 Ideation and Product Inception form the expanding source for innovation Use Collaborative tools to solicit interactive feedback. 1 Service Service
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Rapid Service Creation depends on the common building blocks available in order to create new services quickly Service enabler Service enabler Our Service Innovation proposition provides operators with the ability to “productionise” service assembly and deployment Telecom services can be created from common building blocks – a similar concept to that used in manufacturing Service enabler Service enabler Service enabler Service enabler CommonCapabilitiesPallet Service enabler Service enabler Service 1 Service enabler Service 2 Service enabler Service 3 Service enabler Service enabler Service enabler Service 4Service enabler Service enabler Service enabler Service enabler Service enabler Service enabler Service enabler Service enabler Service enabler
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Convergence is driving TSP’s towards service based models To attract and serve an expanded customer base, Service Providers are becoming more customer-centric and moving to services that target the “long tail”. Telecom Biz Model Web 2.0 Model CentralizedCustomerdefined Fewer,slowerMore,faster Technology Focused Customer Focused Network-centric Marketing-centric Network Investments Business Investments Rapid Market Response Any service at anytime Customer self-management Broad partner ecosystem Tomorrow Today Situational Applications 1. Industry Specific 2. Ad hoc group specific Situational Applications 1. Cross Industry 2. Cross Community Legacy Network Services New Innovations / New Services / New Markets Capex,OpexandTimetoMarket Number of New Products Services & Markets Legacy Telecom Partner Ecosystem Customer Participation Reduced Cost Effort & Time Voice SMS Voice Mail Subscription Billing Location Services Mobile Video MVNO Content Services Virtual Worlds Community User Shared Content Blogs Wikis Forums Tagging Web 2.0 Mashups Video Broadcasts This shift to a customer-centric model requires the adoption of new sets of customer-driven data sources, tools and business processes for a more thorough understanding of customer behavior. Services will become more targeted, focused, personalized, and based on reusable components in many combinations. Service Providers need to re-invent how new services are created, launched and supported.
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What SDP provides A SDP (Service Delivery Platform) solution is a service implementation layer across a diverse set of capabilities This layer allows Marketing, Network, and IT to create and implement new services faster, more efficiently and with lesser cost Provides a unified service creation and control environment where integrated value added services can be defined, deployed and controlled Service Creation Order Capture Provisioning Activation Service Delivery Billing Service Retirement SDP SDP architecture provides an end-to-end control and enables – Selection of a Product or a Service – Placement of an Order – Provisioning of a Service or a Product – Activation of a Service or a Product – Delivering a Service or a Product – Billing for a Service or a Product – Disconnecting a Service or a Product
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP – Context
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deploying Value Added Service (with SDP)
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Services in Scope Content Gateway Portal Management System Content Management System Digital Rights Management Transcoding Directory Services Messaging Gateway A2P and P2A messaging for SMS, MMS Unified API(s) for applications such as televoting Subscription Engine for contents such as alerts, joke of the day, horoscope, etc Bulk Messaging capabilities for promotion, campaigns Application Gateway Enterprise Framework 3rd Party Framework Service Creation Environment Service Management Service Creation Gateway Open access for various applications Enterprise Framework Unified API(s) - Parlay compliant for developing various applications such as Virtual PBX Consumer - User Domain Mobility Users Infotel Users Business User Domain Content Providers Application Providers Developer Community 1 2 4 5 3
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Portal Solution – (IBM WebSphere Portal Server + WebSphere Everyplace Mobile Portal)) Broadband Portal Solution - (IBM WebSphere Portal Server) Web services Access Gateway Solution (IBM Telecom Web Services Server) Process Server and Workflow Engine Solution (IBM WebSphere Process Server) ITCAM Solution for System Management & Performance Monitoring (IBM Tivoli Composite Application Manager) Content Gateway Solution (MPower by Alcatel/Lucent – earlier Mobilitec) + (CoreMedia DRM) + (Adamind MediaSphire) Messaging Gateway Solution - (First Hop) Network Directory Services for SDP Solution – (Apertio One NDS) Real Time Charging Solution – (Differential Billing Platform) Application Gateway Solution – (Aepona Causeway) Campaign Management Solution – (CMS + CIT Systems) SDP Integrated Solution SDP Integrated NFR SDP Integrated Deployment Solution
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Functional Architecture
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Content Gateway: Problem Statement
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CMS Introduction Centralized Content Management – Common platform, for content ingestion, categorization, publishing. (Flexible workflow based) – Content Partner Management (Flexible workflow based) – Content Rating – Content Transcoding – Content Protection Device & Media Management Merchandising – Dynamic segmentation and profiling – Content Promotions and Notifications – Recommendations Personalized discovery and reliable delivery (Device, Locale, LOB,BU etc) Subscription Management Subscriber Management & Customer Care
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Content Gateway - Functions
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Functional Architecture
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Portal Architecture – Context Diagram
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore High Level Security Architecture of Portal
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Functional Architecture
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Core SDP Engine – WPS/WESB
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore The Logical Topolgy – Gold Standard IHS Plug-in IHS DMGR WebSphere Process Server 3WebSphere Process Server 4 Support Cluster WebSphere Process Server 1 WebSphere Process Server 2 Messaging Engine Cluster Active Messaging Engine SCA Queues BPC Queues Active Messaging Engine SCA Queues BPC Queues CEI BRM CEI BRM •Modules are run in Active–Active config •Queues in Active-Active Config •This is a node and not a physical machine WebSphere Process Server 5WebSphere Process Server 6 WebSphere Process Server Cluster BPC Container HTM Container BPC Container HTM Container
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WebSphere Process Server – V1 WPS LPAR 1 WPS LPAR 2 WPS LPAR 3 DB2 LPAR 1 WPS LPAR 4 WPS LPAR 5 WPS LPAR 6 DB2 LPAR 2 Web Servers WebSphere MQ FileSystem • Active- Active Cluster • Two instances per LPAR • HA Configuration • HACMP based with shared disk
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WPS Deployment Diagram WPS CEI AME WPS WPS AME WPS WPS AME WPS WPS WPS WPS WPS WPS Active Messaging Cluster Arbor Mediation Workload Cluster All Other HTTP and MQ based Workloads Cluster • Each Green Box is a physical LPAR • Each box within the green box is a WAS node • Groups of WAS nodes are marked for function and the cluster is indicated • Summary : 4 Clusters + 1 CEI; Single Cell Service creation Subscription Mgmt Selfcare ADC Portal PACS/BI
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Functional Architecture
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TWSS - Architecture Enable 3rd Party Framework The third-party framework allows: – Rapidly creating and – Offering new services – Extends the various levels of functionality such as MB, CMS, DRM, Identity and Access Management, Service Creation and Composition, Security, etc. – The Third Party Enabling Framework enables: – The interaction with third party applications. – The Third Party framework also: – Authenticates and Authorizes 3rd parties’ applications and – Provides access to the services offered by Bharti Airtel. – This framework also provide the customize interfaces to the components existing both in IT and network. TWSS will address A2P – Actual Messaging from 3rd Party / CP applications – Phase - 1
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TWSS HA Deployment (w/ SIP)
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Overview of TWSS Internal Components CMS/Third party Applications Transaction Recorder Policy and Subscription Network statistics Service Authorization SLA Enforcement Service Invocation SMS Service Implementation MMS Service Implementation WAP Service Implementation MM7 ConnectorSMPP Connector SPM Partner Portal TWSS CORE
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP Functional Architecture
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Network Directory Service Architect, Design, Implement and Deploy Directory for: Providing one view for Bharti Subscribers for SDP services. Flexible and extensible schema for future to accommodate other network grade services and functions Flow through provisioning of Bharti Subscribers for SDP and its service platforms (CMS, MB) Prepaid & Postpaid Open access to data directory protocols such as LDAP executable software procedures with in NDS architecture Simplified integration of SDP components: For read operations over direct LDAP i/f from individual component For all write operation over Process Server (single point for writes) Profiles to be supported Subscriber • Partners Service • Device Packages, Add-ons, Bundles, Plans References like Subscriber to device Subscriber to language, Subscriber to geo etc.. Visibility of all subscription and service data in the directory to all other hosted applications for query and update on a controlled and secure basis
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SDP View of Using NDS As Common Data Repository NDS CMS WPS First Hop Portal WEB/Mobile Integration Hub TWSS SDP A SDP D CRM Billing SC IN PACS Provisioning & Profiling Mgmt Subscriber Profile Service Profile Partners Profile Device Profiles SDP Service Packages & Bundles CRBT Voice Portal Broadband Subs Tele voting ADC
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore High – Level SDP Deployment Architecture View OSS/BSS Nodes CRM Rating Billing IN PACS RA BI SMSC Node Core Network Nodes STP HLR GGSN SGSN MMSC Node Streaming Server Node WAP G/W Node IVR CRBT D M Z F I R E W A L L N O D E External Environment DMZ I N T R A N E T F I R E W A L L N O D E Internal Network E N T E R P R I S E F I R E W A L L N O D E Enterprise/ Core Network Content Management System Node Message Broker Node TWSS Node Parlay Gateway Node Portal Server Node I n t e r n e t DRM Node Transcoder Node TAM Node W P S N O D E Web Server Content Partner Client Node Subscribers PC Node Inranet Network Admin Node Content Partner Host Node Subscribers Mobile Client Node CMS DB Node MB DB Node Portal DB Node NDS Node
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Conclusion Business Value Reduced ‘Concept to Market to Revenue’ timeframe for new innovative products & Services Enables Innovative business models reaching out to 3rd party, developers Standardized and repeatable IT cost across service deployments IT Value Optimized and converged workflow and business processes Shared operating infrastructure and resources Scalability for over 200 million subscribers and 300+ devices High performance/ availability platforms Provides key reconciliation infrastructure between OSS/BSS systems and down stream network application and media environments Network Value Creates a Service Control Layer to the Network This layer operates independently from the access network (wire line, wireless, broadband, IP, 3G…) and OSS-BSS systems.
  • © 2008 IBM Corporation Page 36 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore A few key points to take away A SDP solution is a service implementation layer across a diverse set of capabilities Drastically reduces time to market of new services SDP is a complex System Integration exercise No two SDP implementations are the same
  • © 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore THANK YOU QUESTIONS ? Reach me at +9198184870626 ybhatia1@in.ibm.com
  • Randy Tan, Rational IT Specialist tanrkk@my.ibm.com Session D05: Using Rational Performance Tester v7.0.1
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Create realistic user scenarios that exercise key business transactions Identify the root cause of problems Validate Performance with IBM Rational Performance Tester Powerful workload scheduler simplifies the creation of realistic user workloads Advanced Data Correlation features significantly reduce test development effort Easily identify parameters and enable scenarios for data-driven testing Quickly identify poor- performing transactions Breakdown response times across application tiers and drill down to source Correlate response times with system resource measurements Test Authoring and Control System Under Test Load Generators Is your application ready for production deployment?
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What’s new in RPT v7.0.2?
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore New Platforms and Protocols Socket (telnet, ftp, etc) Siebel 8.0 Citrix Performance Server 4.5 Windows 2003 SP2 Red Hat Enterprise Linux Version 5
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What type of environments are supported? Web-based applications HTTP/HTTPS with client-side digital certificates Web 2.0/Ajax Entrust TruePass security SOA SOAP over HTTP/HHTPS SOAP over MQ SOAP over JMS ERP environments mySAP, SAP GUI and SAP-JCO Siebel-web (7.7 and 8.0) Lawson System Foundations Session Initiation Protocol (SIP) Citrix Presentation Server (Citrix ICA) Partner Integrations BSD Oracle eBusiness Suite
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Line Speed Control Emulate WAN connection speeds for HTTP to determine the effect on response times and throughput. Control by User Group. Specify upload or download speed. Wide range of speed options available 56KB Modem to 1GB LAN Custom values
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Example Enabling Line Speed Control
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Time-based Loops New options for how loops behave in schedules and tests Execute specified number of iterations Infinite loops controlled by schedule stage time or manual stop Time-based loops execute for the time you specify
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Staged User Scheduling Ramp Up Virtual Users Ramp Down Virtual Users Virtual User Load Preview Graph Automatic Time Ranges Automatic Compare Report
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Greater Performance Comment Actions Disable Secondary Requests Custom Percentile Reports Random Selector in Test Test Log Data Scalability Playback Engine CPU Efficiency
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Comment Actions Possible to comment out actions Temporarily disables action execution Right-click and choose Disable Grays out and marks action with //
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Disable Secondary Requests HTTP At playback GETs only primary request Skips gifs, jpegs, etc.
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Custom Percentile Reports Defaults are 85, 90, 95 Possible to change or enlarge this set Windows > Preferences > Test > Performance Test Reports > Percentile Reports
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Random Selector in Test Already available for schedule elements Run test elements in random order
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Test Log Scalability Disk Paging System Collect Test Log data up to disk size Just because you can… don’t ☺ Many users logging at a high level can take a lot of disk space Data must be transferred and loaded after the run which takes time Not much user feedback during the transfer process When should I “turn the knob to 11”? Data needed for debugging incorrect test operation Very small number of users running
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Playback Engine Efficiency (Internal Benchmark) HTTP Performance Metrics by Release 0 2 4 6 8 10 12 6.1.1 6.1.2 7.0 7.0.1 7.0.2 Release %CPU/MB 0 200 400 600 800 1000 1200 KB/user CPU (%/MB) Memory (KB/user)
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore IBM Rational Tester for SOA Quality Code free testing of GUI-less services Simplify testing of GUI-less services and ensure interoperability Automated generation of web service test client Create, comprehend, modify, and execute a functional test Graphical test editor enables both high-level and deeper detail views Support Web services standards Simplify service integration testing Automated test creation from Web Services Business Process Execution Language ( WS-BPEL) resources Accelerate test creation and execution Automated data correlation and data driven testing through a code-free test environment Supporting a wide variety of users on broad range of platforms Test creation WS- BPEL Test log details
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore New & Noteworthy in 7.0.2 Support for the latest Web services standards SOAP 1.2, MTOM attachments, and XML type derivation Support for asynchronous communication models Support for WS-Addressing, WS-Notification Enhanced functional test report HTML-based comprehensive functional report 5 pre-defined report design templates with different level of details Highly customizable through the creation of user-defined design template Improved usability New Web services test creation wizard Automatic import of multiple WSDL/XSD files SOAP 1.2 7.0.17.0.17.0.1XML 7.0.17.07.0SOAP 1.1 MQJMSHTTP
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore HTML-based comprehensive functional report
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Functional Report Generation 1. In the Test Navigator, select the runs 2. Right-click « Generate…. » 3. Select the design
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction Platforms and Protocols Workload Scheduling Greater Performance & Flexibility SOA Capabilities Debugging Tests
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Debugging Tests – Data Correlation Enhanced support for WebSphere Portal Test Data Sources View Separate Custom Code, Datapool, references, and candidates Help find missed correlations
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Real-time Browser View Observe test request and response behavior in real time Capture and compare request and responses between record and playback Pause and navigate live rendering to review browser and events during playback Note: implemented for stand-alone test run only. Future update will integrate at the schedule level Real-time Browser results for test validation and debugging
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore RPT 7.0.2 – Documentation Improvements Posted external Help system (information center) on http://publib.boulder.ibm.com/infocenter/rpthelp/v7r0m0/index.jsp Addressed customer comments in 7.0.1.1 beta. New topics include Changing look of a report Deleting a file Section on error messages New SAP and Citrix tutorials—entire walkthroughs Schedule UI has changed. Help topics will include not only how to use the new UI, but how to do things the old way. More context sensitive help – just press F1
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Plus More! Change order of Custom Code arguments New counters for bytes sent and bytes received Automated correlation of XML data Add or Remove users button Automatic compare report Remove users incrementally
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Lab Agenda Six lab exercises to build expertise with Rational Performance Tester: Introduction to RPT Lab 1: Basic Capture/Replay Lab 2: Input Data Variation Lab 3: Data Validation Lab 4: Custom Coding & Data Correlation Lab 5: Workload Generation Lab 6: Performance Analysis
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Let’s Begin!!!
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • Session F02: XML Solutions withSession F02: XML Solutions with WebSphere Transformation ExtenderWebSphere Transformation Extender
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 22 Conference SpeakerConference Speaker XML Solutions with WebSphereXML Solutions with WebSphere Transformation ExtenderTransformation Extender Session Number: F02Session Number: F02 ErneseErnese NorelusNorelus ernese@sg.ibm.comernese@sg.ibm.com Senior Managing ConsultantSenior Managing Consultant IBM Software Group | ASEAN Software Lab ServicesIBM Software Group | ASEAN Software Lab Services
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 33 WebSphere Transformation Extender SessionsWebSphere Transformation Extender Sessions • F08 – WebSphere Transformation Extender 8.2 Design Tools Tuesday 14:30 – Elmer Senson and Dinesh G. Shetty • F02 - XML Solutions with WebSphere Transformation Extender Tuesday 16:45 – Ernese Norelus • F02 - XML Solutions with WebSphere Transformation Extender Thursday 10:45 – Ernese Norelus • F01 – Building Financial Solutions with WebSphere Transformation Extender Friday 9:00 - Elmer Senson • B04 – WPS/WESB 6.1 Integration with WTX, Support and Best Practices Friday 14:45 - Ernese Norelus
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 44 ObjectivesObjectives • Gain a basic understanding of WebSphere Transformation Extender (WTX) concepts What is WTX? Key Concepts Methods of Execution Connectivity Packs WTX Components Demos
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 55 WebSphere TX VisionWebSphere TX Vision Collaborative, extensible and ever-simplifying environment The seamless conversion of all information Ubiquitous, plug and play engine
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 66 WebSphere TX VisionWebSphere TX Vision -- SimplifiedSimplified Everyone Transforms Transform Anything Transform Anywhere
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 77 What Kinds of Problems DoesWhat Kinds of Problems Does Transformation Extender Solve?Transformation Extender Solve? A powerful, portable, transformation engine Simplified ComplexitySimplifiedSimplified ComplexityComplexity No coding Consistent approach to multiple types of integration Reusable objects AdaptabilityAdaptabilityAdaptability Leverages existing IT infrastructure Fully support existing IT investments such as databases and messaging middleware Non-invasive integration Application and industry packs SpeedSpeedSpeed Easy design and implementation Fast maintenance without coding Rapid adaptation to change InteroperabilityInteroperabilityInteroperability Connect applications, databases, processes within the enterprise Partners, processes across enterprises Powerful transformation and routing
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 88 WebSphere TX Product StrategyWebSphere TX Product Strategy • The best-in-class transformation engine that is seamlessly leveraged across the WebSphere platform • The best-of-breed transformation engine that is easily deployed in any enterprise infrastructure
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 99 Transform, validate, and enrich any document, message or complex data Deliver trustworthy information for critical business initiatives Meet regulatory compliance requirements Codeless development; universal reuse and deployment WebSphere TX: Universal TransformationWebSphere TX: Universal Transformation & Reuse& Reuse “Make Advanced ESB More Advanced” Embedded Standalone Event driven MAP TRANSFORM Advanced ESBESB J2EE App. Server Process Server Standalone System z (native) Windows, UNIX, Linux Multi-In Multi-Out
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1010 For Batch Environments (as a standalone engine) For WebSphere Portfolio For ETL Solutions WebSphere TX WebSphere TX MAP TRANSFORM Unifying Enterprise Data Transformation Unifying Enterprise Data Transformation WebSphere Transformation Extender: A UniversalWebSphere Transformation Extender: A Universal Transformation EnvironmentTransformation Environment –– Manage The Complexity!Manage The Complexity! • ONE Transformation Engine for the Enterprise! Powerful transform capabilities without coding Natively handles any data type Solves really hard transformation problems • Complements existing portfolio of products For Event Driven Environments For 3rd Party Integration Products (as an embeddable engine)
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1111 WebSphere Transformation ExtenderWebSphere Transformation Extender and the IBM Platformsand the IBM Platforms WebSphere Application Server WebSphere ESB WebSphere Process Server WebSphere Message Broker z/OS (CICS, MVS, IMS Batch, USS) Leveraging common tooling and runtime, for multiple, interoperable, deployment options WebSphere DataPower
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1212 WTX Industry Accelerator PacksWTX Industry Accelerator Packs –– Adding Value to the WebSphere PortfolioAdding Value to the WebSphere Portfolio X12 EDIFACT TRADACOMS ODETTE EANCOM SWIFT Net FIN SWIFT Net Funds *SEPA *ACORD *NACHA *FIX HIPAA HL7 NCPDP WTX (any deployment option) WTX Industry Pack’s A Pack is not an adapter - a Pack contains meta data, data validation, usage rules, transformation maps and additional functionality
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1313 WTX (any of the server types) WTX Enterprise Application PACK’s SAP R/3 SAP XI Siebel PeopleSoft WTX Enterprise Application AcceleratorWTX Enterprise Application Accelerator PacksPacks Includes BAPI, ALE and IDOC integration capabilities Runs inside SAP App Server Includes BAPI, ALE and IDOC integration capabilities A Pack is not an adapter - a Pack contains meta data, data validation, usage rules , transformation maps and additional functionality (you can sell an adapter and a Pack)
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1414 * $4.6T is based on available, tangible measurements. • On average, over US $4.6T* in transactions flows through WebSphere Transformation Extender on a daily basis 4 Major Stock exchanges 9 of 10 Major US Financial Firms 6 of 6 Top Global Banks 6 of 10 Top US Banks, 4 of 5 Top UK Banks 75% of Blue Cross / Blue Shield claims 6 of 10 Major Insurance Firms 3 of 3 Top US Pharmacy Benefits Providers Major Global and National Telecoms Over 95% of all US Department of Defense logistic transactions pass through TX and on and on and on…. WebSphere Transformation Extender is aWebSphere Transformation Extender is a strategic solution for our customersstrategic solution for our customers
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1515 Unique Values of WebSphereUnique Values of WebSphere Transformation ExtenderTransformation Extender • Handles all data natively Adapters are communication vehicles only No “internal format” – all work happens in the engine, under the control of the designer Significantly reduces complexity of product, and opportunity to leverage customer technology investment • Can process interdependent inputs simultaneously High performance data validation and processing Handles Complex IT processing / transformation challenges without code • Can be leveraged in all “styles of integration” Batch, Asynchronous, Synchronous, Real-Time and Web Service environments
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1616 WebSphere Transformation ExtenderWebSphere Transformation Extender across WebSphereacross WebSphere • WTX is the WebSphere “Universal Transformer” Any kind of data, any kind of transform Is uniquely capable of MANY to MANY transforms • WTX is a value add on top of the WebSphere Platform Products • WTX runs natively in CICS, MVS and IMS Batch Unique offering across IBM Data Processing Capability without Code • Same Maps can be reused across Mainframe, App Server, Windows, Unix and ESB, Message Broker “One Tool and One Engine, able to be leveraged in Any Architecture, for Any Transformation Demand, in Any Transformation Style… a true Universal Transformation Engine”
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1717 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- ToolingTooling • Design Tools Map Designer and Type Designer have become Eclipse plug-ins. They may be hosted by a stand-alone Design Studio, or they may be hosted in other Eclipse design environments, such as the WebSphere Message Broker Toolkit, or WebSphere Integration Developer. No significant change in functionality or design Integration Flow Designer and its Management Tools have not been ported to Eclipse. Maps are still MMS files and most Type Trees are still MTT files Native support added for XML Schemas (details to follow)
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1818 WebSphere TX: Key ConstructsWebSphere TX: Key Constructs • Type trees •Graphical “data” dictionary • Maps •Complete definitions of data objects and rules for their transformation •A subroutine implemented as a map is called a Functional Map • Cards •Inputs to or outputs from maps • Systems •Collections of Maps configuration to run inside the Launcher
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 1919 WebSphere TX: Working with Type DesignerWebSphere TX: Working with Type Designer • Type Designer •Used to create and edit type trees • Importer Collection •Launched from the Type Designer. •Used to generate type trees from external meta data sources • Database Interface Designer •Used to build type tress from Database tables •Generates Database Query (MDQ) files in additions to type trees
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2020 WebSphere TX: Working with MapsWebSphere TX: Working with Maps • Map Designer •Used to create, edit and compile maps. •Host the Profiler and Debugger •Can launch the Type Designer. • Resource Registry •Defines runtime parameters which may be referenced • Command Server •Used to configure and execute maps
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2121 Map DesignerMap Designer
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2222 Type DesignerType Designer
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2323 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- ToolingTooling • ‘Classic’ Importers The classic importers are now Eclipse importers, which may be invoked from the new Type Designer. No change in behavior from v8.1. They will be joined by a new set of EMD importers common to many IBM Eclipse-based design tools. • Text File Importer improvements Group members no longer must be all on the same line Types organized by Category Improved usability • New PL/1 importer coming with 8.2.0.2
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2424 Importing a Type TreeImporting a Type Tree -- ExampleExample
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2525 XML SchemasXML Schemas vsvs Type TreesType Trees • Schemas appear as regular type trees • In addition to .mtt files, the new Map Designer supports .xsd files • There is actually no need to import schemas anymore, but the XML importers are retained in V8.2 • Schemas may be edited using any schema editor – IBM or other. You do not need to use Type Designer to edit them • Expect performance improvements compared to previous versions
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2626 Working with XML SchemaWorking with XML Schema -- ExampleExample Create a map Create a card Choose a schema Choose a root Output cards work too Map away ! It looks like a Type Tree
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2727 WTX demoWTX demo • Create a WTX Project • Create a Type Tree • Create a Map • Load a WTX example • WTX Importers • Any-2-XML
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2828 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- XMLXML • Native support added for XML Schemas Restart and reject logic is added for XML Type Trees With restart enabled, the XML validation log file now reports all validation errors, it doesn’t stop at the first error. • Schema location setting for Cards Card Metadata Location parameter Allows you to specify the location of your schema (as a file or url) Prior to v8.2 the information was only in XML Type Trees (Intent/Validate As/Location on the ‘Doc’ group) • Schema validation can be disabled on a per Card basis via a new Card property option Document Verification / Well Formed • Support has been added for the XSD Union Type and xsi:type
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 2929 XML document as map sourceXML document as map source 1. Generate type tree from schema or DTD 2. Modify Doc component list to identify root element when possible 3. Create map with schema type tree on input card 4. Select document validation option on input card 5. Expand components 6. Enter map rules based on input and output type properties 7. Create functional maps or use INDEX to map objects that occur more than once 8. Build and run the map
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3030 AnyAny--toto--XML utilityXML utility • Creates an XML schema from a type tree • Generates a Xerces-based type tree from the XML schema • Optionally creates a new map source file and map that transforms original input to XML output Input card references the reference type tree and input file Output card references new XML type tree and schema to create output file
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3131 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- ConnectivityConnectivity • A subset of IBM WebSphere Adapters are offered with WTX v8.2 WebSphere Adapters are JCA based Their legacy counterparts are still included, but have been deprecated. The specific adapters included are: Email Includes support for multiple attachments, and headers Adds a ‘listener’ interface FTP Allows remote files to be renamed once they have been read Adds a ‘listener’ interface SAP * Adds support for the Query Interface for SAP Software (QISS) PeopleSoft * Adds a ‘listener’ interface for Component Interfaces Siebel * Adds a ‘listener’ interface for Business Objects • IMS TM (coming in 8.2.0.2) *Note: SAP, PeopleSoft and Siebel are available only in the Enterprise Packs
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3232 ImportersImporters • Each JCA adapter includes a new ‘EMD’ * importer, which may be used to generate: XML schemas – in lieu of Type Trees Service Definitions – similar to MDQ files Adapter Command Lines At the point where the adapter is bound to the card, a ‘Command Line Wizard’ is available. * Enterprise Metadata Discovery
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3333 WebSphereWebSphere TXTX v8.2 Release Featuresv8.2 Release Features -- EditionsEditions • WTX the edition for imbedding is renamed for clarity of purpose. WebSphere Transformation Extender for Application Programming (for z/OS) • The Extender for Message Broker is now packaged with a new Extender in a single Edition *. WebSphere Transformation Extender for Integration Servers • Integration Servers name chosen as an inclusive term for ESB and BPM solutions: Extender for WebSphere Message Broker Extender for WebSphere Process Server / WebSphere ESB A single License instance permits use of one or the other. Note: The Extender for Message Broker remains at V8.1 in terms of functionality but will be significantly enhanced in 1H08.
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3434 WebSphere TX for Integration ServersWebSphere TX for Integration Servers -- extender for WPS/WESBextender for WPS/WESB • WebSphere TX can be selected as a Data Binding on JMS, MQ, HTTP or EIS Imports or Exports • Maps typically convert between external data formats and Business Objects • Map Generator is a new Eclipse based Wizard simplifies development of maps Business Objects are converted to ‘Type Trees’ (technically ‘schemas’) Template Maps are created Projects (including their maps) are easily deployable using standard WPS and WID mechanisms • The Wizard integrates WTX Design Studio with WebSphere Integration Developer
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3535 Map Generator WizardMap Generator Wizard –– ExampleExample
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3636 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- Industry PacksIndustry Packs WTX Packs for EDI v2.7.1.1 • WTX Pack for X12 Released for WTX V8.2 in 1Q08 • WTX Pack for EDIFACT Released for WTX V8.2 in 1Q08 • WTX Pack for EANCOM No change • WTX Pack for TRADACOMS No change • WTX Pack for ODETTE Bug Fixes only
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3737 EDIEDI –– Earlier ReleasesEarlier Releases Earlier Releases • Released Sept. 2007 to work with WTX v8.1 • WTX Pack for X12 v2.7.1.0 Support for ANSI X12 5040 Majority of changes in Transportation/Shipping Also includes a new transaction for Customs and Border Protection • WTX Pack for EDIFACT v2.7.1.0 Support for UN/EDIFACT D.06A, D.06B No new messages added; only a few changed.
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3838 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- PlatformsPlatforms • Development Windows • Run time 32 bit Windows IBM AIX® Red Hat Enterprise Linux™ for x86 platforms SUSE Linux Enterprise HP-UX 11i (PA-RISC) or 11i V2, V3 (Itanium) Sun Solaris z/OS 1.4 or later (batch, USS, IMS, CICS) • Run time 64 bit tolerance ”new” IBM AIX® Red Hat Enterprise Linux™ for x86 platforms SUSE Linux Enterprise HP-UX 11i (PA-RISC) or 11i V2, V3 (Itanium) Sun Solaris Linux on System z “new”
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 3939 WebSphere TX v8.2 Release FeaturesWebSphere TX v8.2 Release Features -- UpgradingUpgrading • Design Studio All existing maps and type trees will automatically open in v8.2 No changes to Database Interface Designer & Integration Flow Designer • WebSphere TX for Message Broker Existing flows that use the v8.1 Node will seamlessly migrate to the v8.2 node Existing flows that use the v8.1 Parser will need to be recreated • WebSphere TX Runtimes All compiled maps from WebSphere TX v5.0 and higher will run without recompiling If a map will be recompiled with Version 8.2, all type trees must be reanalyzed with Version 8.2 Use of the new Native XML features instead of type trees will require maps to be rewritten. Existing XML type trees and maps will remain valid. Like previous versions, TX can be installed side by side on the same machine
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 4040 SummarySummary • What is WebSphere TX ? WebSphere TX is IBM’s Universal Transformation Engine • Key Concepts Maps, Cards, Type Trees, Functional Maps • Methods of Execution Command Server, Launcher, Extenders for WebSphere Portfolio, SDK • Connectivity Adapters, Importers • Demos Importers, XML output
  • 2008 WebSphere Technical Conference © 2008 IBM Corporation 4141 QuestionsQuestions • Please complete your session evaluation • Thank you!!
  • Dinesh G Shetty IBM Certified IT Specialist F03 - Driving WebSphere Message Broker using WSRR
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Brief Review of WSRR WMB usage patterns Implementing patterns
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Brief Review of WSRR What is it? How WMB interacts WMB usage patterns Implementing patterns
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What is WSRR? Repository?Registry? Stores Artifacts / entities WSDL XSD Other XML documents SCDL… etc. Hold Meta-data Describes the service artifacts Allows you to search for artifacts in the repository
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore How WMB Interacts with WSRR Two nodes that query an instance of WSRR Result placed into LocalEnvironment Payload is not modified Results of query propagated to Out terminal Out Failure NoMatch In Out Failure NoMatch In EndpointLookup Node Focused on the retrieval of endpoints defined within WSDL Retrieves EPR and Metadata, not whole artifact Able to search port type names within WSDL imported into WSRR Use endpoint for dynamic routing at runtime RegistryLookup Node Generic Node for retrieval of any WSRR entity Able to search entity names Retrieve entire contents of entity into the Broker Use entity information at runtime
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Brief Review of WSRR WMB usage patterns General patterns Sample patterns where WSRR fits Implementing patterns
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WMB Usage patterns Service Enablement Service Virtualization OR OR OR Message Enablement Message Brokering File Processing Event Processing A10
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes We have identified six key usage patterns for Message Broker. You may find that all or only some of these patterns apply to your environment. The extent to which all patterns apply will probably depend on the number, age and maturity of your IT systems. It is also important to say these classifications are not rigid. As with any classification system there will be anomalies and overlaps. Indeed some implementations of Message Broker will sit in more than one usage pattern and this happens in a couple of cases that we will look at. Service enablement. This is essentially taking a varied collection of applications and presenting aspects of their functionality through a service orientated interface. This pattern represents the transition from more traditional notions of EAI into service orientated architectures. Service virtualization. This pattern is concerned with providing true loose coupling between services by providing additional levels of indirection through an ESB. This addresses the requirements of mediation between services in a services orientated model. Message enablement. This is concerned with applying a message based architecture to applications, which were not necessarily designed to use one by using an Enterprise Service Bus (ESB). Message brokering. This covers extending an existing messaging infrastructure by providing an environment for building and deploying "infrastructure level" message-based "applications”. Examples of such “applications” would include routing, logging and transformation services. File processing. This usage pattern is about facilitating the integration of file based applications and providing a managed execution environment for the processing of files. It is also about providing a bridge between batch-orientated applications and messaging models of execution including transactional processing. Event processing. This is about providing a managed execution environment for the processing of events including distribution of events in real time, processing of events from physical devices, simple information distribution and detection of temporally orientated event patterns.
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Enablement Message Enablement Message Brokering WMB Usage patterns OR OR OR File Processing Event Processing Service Virtualization A10
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy request message contains Service Name or rather a Service ID ESB retrieves endpoint information for that particular service from WSRR hiding the actual service providers true endpoint ESBESB Provider Service B Provider Service B RequestorRequestor Provider Service A Provider Service A Service A WSRRWSRR
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Multiple implementations of a service interface. Example: request is sent to a certain ESB environment (e.g., TEST / PROD) depending on which environment the ESB is running on, the message is sent to the corresponding service provider’s environment ESB (PROD)ESB (PROD) Provider Service A (PROD) Provider Service A (PROD) RequestorRequestor Provider Service A (TEST) Provider Service A (TEST) Service A WSRRWSRR
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Fault Handling) retrieval of a collection of service endpoints providing the same function message is sent to one of the services taking into contrast the services providers’ availability, workload, performance etc. ESBESB Provider Service A Provider Service A RequestorRequestor Provider Service A Provider Service A Service A Fault WSRRWSRR
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Normalizer mapping information for the mapping between the service requestor’s interface and the service provider’s interface is retrieved from WSRR message content is adapted before the message is sent to the provider ESBESB ProviderProviderRequestorRequestor A B C C AB D WSRRWSRR
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Translator (Binding) ESB retrieves the binding information from WSRR message is then sent to the Service Provider using the adequate protocol and transport for that service implementation ESBESB ProviderProviderRequestorRequestor WSRRWSRR
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Policy Enforcement request message is signed and / or encrypted ESB checks against WSRR if the service policy is met for a particular service ESBESB ProviderProviderRequestorRequestor WSRRWSRR
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Brief Review of WSRR WMB usage patterns Implementing patterns Service Virtualization – Service Selector (Environment) – Service Proxy & Translator (Binding)
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) What is it? What problem does it solve? How to define in WSRR? How to build WMB solution? Demo Service Virtualization OR OR OR ESB Mediation: Service selector Service requestor Service provider Service proxy Service Registry Service provider Service provider Content Based Router endpoint reference ESB Mediation: Service selector Service requestor Service provider Service proxy Service Registry Service provider Service provider Content Based Router endpoint reference Service Selector
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) What is it? Enables invocation of endpoint based on runtime environment For example: – Development – Test – Production ESB Mediation: Service selector Service requestor Service provider Service proxy Service Registry Service provider Service provider Content Based Router endpoint reference DevDev TestTest ProdProd
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) What problem does it solve? Empowers customers to manage runtime behavior via central SOA repository Minimizes preparing deployment artifacts per environment Minimizes revisiting development artifacts per environment
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) How to define in WSRR? 1. Follow recommended practice for Endpoint Management 2. Follow recommended practice for multi-file wsdl – Separate service port from the interface – (May have separate binding or include binding with service port) 3. Load WSDL and Schema to WSRR 4. Classify ports by “Environment”
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore <?xml version="1.0" encoding="UTF-8"?> <schema targetNamespace="http://demo.sr.eis.ibm.com” <element name="updateCustomer"> <complexType> <sequence> <element name="str" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerResponse"> <complexType> <sequence> <element name="updateCustomerReturn" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerAlternateInfo"> <complexType> <sequence> <element name="customer" nillable="true" type="impl:CustomerInfo"/> </sequence> </complexType> </element> <complexType name="CustomerInfo"> <sequence> <element name="firstName" nillable="true" type="xsd:string"/> <element name="lastName" nillable="true" type="xsd:string"/> <element name="middleName" nillable="true" type="xsd:string"/> <element name="addresses" nillable="true" type="impl:ArrayOfAddressInfo"/> </sequence> </complexType> <complexType name="ArrayOfAddressInfo"> <sequence> <element maxOccurs="unbounded" minOccurs="0" name="AddressInfo" </sequence> </complexType> </schema> – schema (xsd) – – wsdl – <?xml version="1.0" encoding="UTF-8"?> <schema targetNamespace="http://demo.sr.eis.ibm.com” <element name="updateCustomer"> <complexType> <sequence> <element name="str" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerResponse"> <complexType> <sequence> <element name="updateCustomerReturn" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerAlternateInfo"> <complexType> <sequence> <element name="customer" nillable="true" type="impl:CustomerInfo"/> </sequence> </complexType> </element> <complexType name="CustomerInfo"> <sequence> <element name="firstName" nillable="true" type="xsd:string"/> <element name="lastName" nillable="true" type="xsd:string"/> <element name="middleName" nillable="true" type="xsd:string"/> <element name="addresses" nillable="true" type="impl:ArrayOfAddressInfo"/> </sequence> </complexType> <complexType name="ArrayOfAddressInfo"> <sequence> <element maxOccurs="unbounded" minOccurs="0" name="AddressInfo" </sequence> </complexType> </schema> <?xml version="1.0" encoding="UTF-8"?> <schema targetNamespace="http://demo.sr.eis.ibm.com” <element name="updateCustomer"> <complexType> <sequence> <element name="str" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerResponse"> <complexType> <sequence> <element name="updateCustomerReturn" nillable="true" type="xsd:string"/> </sequence> </complexType> </element> <element name="updateCustomerAlternateInfo"> <complexType> <sequence> <element name="customer" nillable="true" type="impl:CustomerInfo"/> </sequence> </complexType> </element> <complexType name="CustomerInfo"> <sequence> <element name="firstName" nillable="true" type="xsd:string"/> <element name="lastName" nillable="true" type="xsd:string"/> <element name="middleName" nillable="true" type="xsd:string"/> <element name="addresses" nillable="true" type="impl:ArrayOfAddressInfo"/> </sequence> </complexType> <complexType name="ArrayOfAddressInfo"> <sequence> <element maxOccurs="unbounded" minOccurs="0" name="AddressInfo" </sequence> </complexType> </schema> 2. Follow recommended practice for multi-file wsdl Service Virtualization – Service Selector (Environment)
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment)3. Load WSDL files and Schema files to WSRR
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment)4. Classify ports by “Environment”
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment)4. Classify ports by “Environment”
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) How to build WMB solution?
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) In Set Query Endpoint Lookup Map Map ReplyRequest
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Set Query Endpoint Lookup
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) NOTE: For demo purposes the above code uses Execution Group to represent the runtime “Environment”. DECLARE ThisBroker CHAR BrokerName; DECLARE ThisEG CHAR ExecutionGroupLabel; DECLARE ClassificationString CHAR 'http://www.ibm.com/xmlns/prod/serviceregistry/6/1/GovernanceProfileTaxonomy#'; -- Copy HTTP Request SET OutputLocalEnvironment.Destination.HTTP.RequestIdentifier = InputLocalEnvironment.Destination.HTTP.RequestIdentifier; -- Set Service Registry Lookup elements SET OutputLocalEnvironment.ServiceRegistryLookupProperties.Classification = ClassificationString || ThisEG;
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment)
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo 1. Scenario Overview 2. Define WSDL 3. Configure WSRR 4. Construct Solution 5. Test
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo scenario – begin Service Requester Service Provider Sub-flow
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo scenario Reply message is string: “Updated customer via WMB env: <Broker Name>.<EG Name>” deployed to Development using URL: http://localhost:7080/DemoCustomerMB/services/Dev/DemoCustomer deployed to Test using URL: http://localhost:7080/DemoCustomerMB/services/Test/DemoCustomer Service Provider
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo scenario Begin challenge: static URL Moving from Development to Test env requires modifying bar file property for the HTTP Request node url – from Development URL: http://localhost:7080/DemoCustomerMB/services/Dev/DemoCustomer – to Test URL: http://localhost:7080/DemoCustomerMB/services/Test/DemoCustomer Service Requester
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo scenario Using VMWare WMB v6.1 fp1, WSRR v6.1 fp2 Single Broker: WBRK61_DEFAULT_BROKER ServiceRegistry pre-configured to broker Two EGs: – Development – Test
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo scenario – end Service Requester Service Provider Sub-flow
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Selector (Environment) Demo link
  • © 2008 IBM Corporation Page 36 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) What is it? What problems does it solve? How to define in WSRR? How to build WMB solution? Demo Service Virtualization OR OR OR ESB Mediation: Service Normalizer Service requestor Service provider Service proxy Service Registry Service provider Service provider Content Based Router Service proxy Service proxy Message translator Message translator Message translator Service Translator
  • © 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) What is it? decoupling between service requestors and service providers by introducing an additional level of indirection to the service invocation technical details about the services’ implementation are not exposed to the service requestors Service Normalizer Service requestor Service provider Service proxy Service Registry Service provider Service provider Content Based Router Service proxy Service proxy Message translator Message translator Message translator
  • © 2008 IBM Corporation Page 38 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) What problems does it solve? single point of access service implementation details can change without the need to inform the clients further advantages with the implementation of policy enforcement, autiting etc.
  • © 2008 IBM Corporation Page 39 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) Assumptions / Constraints service requestor knows the service names service interfaces / message content not validated in the flows definition of a SOAP binding for a service suggests that the service endpoint must be defined as a SOAP address (regardless of the transport used for that endpoint) simple functional error-handling no technical error handling – communication “ESB WSRR” and “ESB Services” has been established – service information is present and can be retrieved from WSRR service responses (processing of requests) are “hard-coded” ServiceRegistry pre-configured to broker
  • © 2008 IBM Corporation Page 40 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) How to define in WSRR? followed recommended practice for multi-file WSDL (Top-Down Approach) problems with the recommended practice for WSDL service versioning – SOAP Nodes do not support different target namespaces within the single files internal service provider: “AccountBalanceService” (SOAP / WMQ) external service provider: “StockQuoteService” (SOAP / HTTP)
  • © 2008 IBM Corporation Page 41 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore StockQuoteService WSDL (Binding and Port) Binding Port
  • © 2008 IBM Corporation Page 42 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AccountBalanceService WSDL (Binding and Port) Binding Port
  • © 2008 IBM Corporation Page 43 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Endpoint representation in WSRR SOAP protocol can be used with different transports (e.g. HTTP, WMQ) for the WMQ service refer to the IRI specification which defines the addresses and additional messaging properties of WMQ resources (SupportPack MA93)
  • © 2008 IBM Corporation Page 44 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Binding representation in WSRR the different transports are specified on the SOAP binding native MQ binding as defined in supportpack MA93 is not supported yet refer to REDP-4350-00 “Cataloging WebSphere MQ applications in WSDL for reuse in SOA” for a workaround on that
  • © 2008 IBM Corporation Page 45 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) How to build WMB solution?
  • © 2008 IBM Corporation Page 46 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Considering the use of the new SOAP Nodes LocalEnvironment overrides are available for a couple of runtime properties, e.g. endpoints other information about the service interface and binding are set with a particular WSDL during buildtime these information cannot be changed during runtime for a Service Proxy you would need the same number of SOAP Request Nodes defined at buildtime as the number of services you want to access at runtime
  • © 2008 IBM Corporation Page 47 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Example Services (Broker Message Flows) StockQuoteService AccountBalanceService 1. get the request message 2. construct the response message 3. send back the response message
  • © 2008 IBM Corporation Page 48 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) Request to ESB Request to ESB WSRR Lookup WSRR Lookup Transport Selection Transport Selection Response from ESB Response from ESB Protocol Conversion Protocol Conversion Service Request Service Request
  • © 2008 IBM Corporation Page 49 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Enterprise Service Bus – Message Flow Request to ESB / WSRR-LookupRequest to ESB / WSRR-Lookup Protocol Conversion / Transport RoutingProtocol Conversion / Transport Routing Service Request / Protocol ConversionService Request / Protocol Conversion Response from ESBResponse from ESB Functional Error HandlingFunctional Error Handling
  • © 2008 IBM Corporation Page 50 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Trade-Off between WSRR Nodes and Java Compute Node WSRR Nodes supports WSRR API v6.0.x cache feature WSDL extension elements are not supported custom relationships are not supported whole WSDL can be retrieved and parsed in broker Java Compute Node any WSRR API (e.g. v6.1) can be used cache has to be build on someone’s own take into contrast the complexity of cache updates (relationships between artifacts)
  • © 2008 IBM Corporation Page 51 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WSRR Lookup and Response Processing dummy value on the ServiceRegistryLookUp Node modifying the lookup properties at runtime SET OutputLocalEnvironment.ServiceRegistryLookupProperties.Name = InputRoot.XMLNSC.RequestData.Service;
  • © 2008 IBM Corporation Page 52 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Evaluating the results from WSRR “SOAPAdress” element suggests the service uses a SOAP binding determines the further route in the flow “location” element contains the service endpoint
  • © 2008 IBM Corporation Page 53 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Accessing fields in the LocalEnvironment results of a WSRR lookup are written to the LocalEnvironment initial problems with accessing particular fields in the tree InputLocalEnvironment.ServiceRegistry.Entity.ports.SOAPAddress trace output of the ServiceRegsitryLookupNode showed that namespaces were put into the LocalEnvironment for some of the fields you have to use the correct namespace DECLARE sdoNS NAMESPACE 'http://www.ibm.com/xmlns/prod/serviceregistry/6/0/sdo'; DECLARE soapRef REFERENCE TO InputLocalEnvironment.ServiceRegistry.Entity.sdoNS:ports.sdoNS:SOAPAddress;
  • © 2008 IBM Corporation Page 54 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Routing to the appropriate service request part continue with routing the message to the corresponding service request part depending on the transport information that are available from the environment for the HTTP request the endpoint can be taken as is http://192.168.19.20:7801/banking/StockQuote for the WMQ request the endpoint has to be decomposed into the request queue and the reply to queue part wmq:/msg/queue/ACC.BALANCE.REQUEST?ReplyTo=msg/queue/ACC.BALANCE.RESPONSE create a standard SOAP Envelope before the message is sent further
  • © 2008 IBM Corporation Page 55 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Requests request is send out with either a HTTP Request or WMQ Output node for the HTTP service request the SOAP Request Node should be avoided in a generic Service Proxy Pattern SOAP envelope is removed before the message is sent back to the requestor
  • © 2008 IBM Corporation Page 56 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Future Considerations? SOAP Node support for multi-file / multi-namespace WSDLs SOAP Node support for JMS / WMQ Support for dynamic changing interface / binding information at runtime Support for dynamic transformation resolution Support in WSRR nodes to dynamically set query (avoid need for another compute node to set query) WSRR Node support for WSDL extension elements WSRR Node support for custom relationships IBM Confidential
  • © 2008 IBM Corporation Page 57 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Service Virtualization – Service Proxy & Translator (Binding) Demo Link
  • © 2008 IBM Corporation Page 58 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WSRR and WMB References IA9Q: WebSphere MB v6 Client for WebSphere Service Registry and Repository http://www.ibm.com/support/docview.wss?uid=swg24014652 Publishing a Web Service to WebSphere Service Registry and Repository from CICS Transaction Server v.3.1: http://www.ibm.com/developerworks/websphere/library/techarticles/0610_millwood/0610_millwood.html Introducing WebSphere Service Registry and Repository - Part 1: http://ltsbwass001.sby.ibm.com/cms/developerworks/websphere/library/techarticles/0609_mckee/0609_mckee.html Introducing WebSphere Service Registry and Repository - Part 2: http://www.ibm.com/developerworks/websphere/library/techarticles/0609_mckee2/0609_mckee2.html Introducing WebSphere Service Registry and Repository APIs: http://www.ibm.com/developerworks/websphere/library/techarticles/0611_baldwin/0611_baldwin.html Build flexible ESB mediations with MessageBroker and WebSphere Service Registry and Repository: http://www.ibm.com/developerworks/websphere/library/techarticles/0610_patten/0610_patten.html WSRR Info Center http://publib.boulder.ibm.com/infocenter/sr/v6r0/index.jsp WSRR Product Page http://www.ibm.com/software/integration/wsrr/index.html
  • © 2008 IBM Corporation Page 59 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Questions Questions? Please complete the session evaluation form Thank you!!
  • © 2008 IBM Corporation Page 60 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
  • Dinesh G Shetty IBM Certified IT Specialist F04 - Best Practices in Modeling Complex Messages with WMB 6.1
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Agenda Introduction to message modeling A reminder of the key concepts Key tasks when modeling non-XML messages Understanding the physical format Understanding the logical structure Relating the physical format to the logical structure Text message formats A look at text messages in more detail Scenario COBOL application Comma Separated Values (CSV) application Best practices Diagnosis of modeling problems XML parsing
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes - Agenda This session assumes some prior knowledge of message modeling and parsing using WebSphere Message Broker. However, don’t panic if you aren’t that familiar as we’ll start with an introduction of the main concepts. We will introduce the key steps involved when modeling messages using the MRM domain, looking at binary and text message formats. We will concentrate on text message formats as they are usually the hardest to model, but the techniques introduced are applicable to all message styles you are likely to encounter. We will conclude with a scenario to illustrate these techniques.
  • Introduction to message modeling
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore <Person Age=“32” Height=“172.5”> <Name>Joe Bloggs</Name> </Person> What is message parsing? Input message bit stream Output logical message tree Input logical message tree Output message bit stream Transformation Parser Parser 032 1725 Joe Bloggs Model Model
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – What is message parsing? Typically a WebSphere Message Broker message flow will receive messages in a defined format, transform them, and output them in a different format. The example shows a COBOL Person data structure being transformed into an XML Person document. You can see that each of the Person messages contains Name, Age and Height fields. The COBOL Person message arrives in the form of an input bit stream. Before it can be processed by the message flow, it must be converted into something called an input logical message tree for transformation by the message flow. This is a broker data structure that reflects the logical structure of the message. All of the broker’s processing nodes work with the logical message tree, regardless of whether your transformation logic is expressed in Java or ESQL or Mapping. In the example, some logic is applied which transforms the message and creates a new, output logical message tree. Finally this is converted into an output bit stream that represents the message as an XML document. The component of the broker responsible for converting a bit stream into a logical message tree (parsing), and vice versa (serializing or writing), is called a parser. The parser must understand both the physical format of the bit stream and its logical structure in order to create the logical message tree. In the example you can see two parsers, one that understands COBOL data structures and another that understands XML. Note that it is only the parser that needs to understand the physical format of the message. The logical message tree is independent of the physical format of the message bit stream. This decoupling of physical format from transformation logic is a key architectural feature of broker message flows. A parser may, or may not, use a model when parsing. You will see the advantages provided by having such a model, later on.
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Parsers and domains Each message is associated with a domain A domain is suited to a particular class of messages and has a parser Broker supplies several domains Some domains use a model-driven parser – XMLNSC domain for XML documents (6.1) – MRM domain for binary and text data (also handles XML) – SOAP domain for SOAP-based web services (6.1) – DataObject domain for WebSphere Adapters business objects (6.1) Others do not use a model – BLOB domain for opaque data – XMLNSC (pre-6.1) or XMLNS domain for XML documents – MIME domain for MIME messages You can also write your own
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Parsers and domains Each message that is to be processed by a message flow must be associated with a domain. A domain determines the parser that is used when parsing and serializing the message. Each domain is suited to a particular class of messages. Some domains even support several different classes of message. The domain to use for an input message is typically specified on the input node of the message flow, but can also be specified by an MQRFH2 header or JMS header. The domain to use for an output message is specified when the message logical tree is created by a transformation node within the message flow. To keep things simple, the parser for a domain has the same name as the domain – effectively domain and parser mean the same thing. The recommended domain for XML documents is XMLNSC. This has high performance and creates a compact message tree. From 6.1 it is optionally model-driven and so can validate XML documents according to XML Schema 1.0 rules. The recommended domain for non-XML messages is MRM which has a general purpose parser capable of parsing a wide range of binary and text formats. It is model-driven and so can validate messages. It also understands XML documents, so it is truly general purpose! The recommended domain for SOAP based web services from 6.1 is SOAP. This also handles SOAP with Attachments (SwA) and MTOM encodings and understands WS-Security and WS-Addressing headers. When communicating with the new JCA-based WebSphere Adapters from 6.1, use the DataObject domain to parse and write the business objects used by the adapters. The BLOB domain can be used when the message is being treated as opaque, and just being routed. Effectively the message is not parsed at all. If you are parsing XML documents and need the message tree to conform exactly to the XML infoset (and so be fully XPath compliant) you can use the XMLNS domain. This does not use a model and so can not validate. Use the MIME domain for MIME messages other than SwA (eg, RosettaNet). Alternatively you can write your own parser for use by your own user-defined domain.
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 1. Most messages are not self-defining Often a model is needed to parse the message bit stream 2. Validation A model is needed to check message correctness when parsing 3. Speeds development of transformations Provides source and target for graphical mappings Provides assistance when editing transformation programs 4. Version control By storing in a central shared repository 5. Instant documentation For programmers, analysts and integration specialists Runtime use of model by parser Design time use of model by toolkit Why model messages?
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Why model messages? There are several reasons why you might need to model your messages. The first two are uses of the model by a parser at runtime. The others are uses of the model at design time, and are independent of whether the model is actually used by a parser. 1. The majority of messages are not what is termed ‘self-defining’. Think of a message created by a C or COBOL program – it’s just a stream of binary data. Without some intelligence to interpret it, it is meaningless, and that’s where the model comes in. XML is the opposite – it is so verbose that an XML parser can parse any XML document without using a model at all. 2. If you want to validate that your messages are correctly structured you need a model. An XML parser can parse any XML document but it can only check it is correct if it has a model to guide it. 3. A model can speed up the development of transformations enormously. For example, graphical mapping from a source message to a target message is not possible without a model. If you are transforming XML documents in this manner, you need a model even if the XML parser chooses not to use a model at runtime. 4. Models provide a good way of tracking different versions of your messages. A COBOL programmer would typically create a new version of a copybook each time he made a change, using a configuration control system. The same principle applies to message models. 5. A model provides documentation for a message that can be shared between programmers, business analysts and integration specialists.
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Creating & using message sets Generators Message set WSDL MessageMessage BrokersBrokers ToolkitToolkit MessageMessage BrokerBroker RuntimeRuntime Mapper Model Model MRM Parser XMLNSC Parser Domain: MRM Set: Person Defs Type: Person Format: COBOL1 Domain: XMLNSC Set: Person Defs Type: Format: Importers COBOL C XSD WSDL Adapter ESQL Message dictionary WSDLs Zip of XSDs and WSDLs Message dictionary XSDs
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Creating & using message sets Message models live in a container called a message set. A message set is a special folder inside a message set project within the Message Brokers Toolkit workspace. The typical sequence of events when creating and using a message set is as follows. First you create a message set project and message set using the wizard. This will ask you what kinds of message format you are working with, and set the message domain of the message set for you (which you can change later if you want). Next you import application message formats described by XML DTD, XML Schema, WSDL types, C structures or COBOL structures, which create and populate message definition files (.mxsd). These are annotated XML Schema files – more on this later. Alternatively, you can create an empty message definition file and create your message model using just the editor. If the message models in the message set are to be referenced by the Mapping editor or the ESQL editor, you must set the message flow project so that it references the message set project. This is done using the File -> Properties menu. If the message models in the message set are to be used by other tools to create Web Services, you can generate WSDL from the message set using a wizard. If the message models in the message set are to be used by a parser at runtime, you must deploy them to the broker. It is the message set that is the unit of deployment. You do this by adding the message set to a broker archive (.bar) file. If the message set is to be used by the MRM parser, it is generated into a compact form called a message dictionary. If the message set is to be used by the XMLNSC, SOAP or DataObject parsers, it is generated into pure XML Schema (xsd) files which are zipped. If the message set is to be used by the SOAP parser, WSDLs are added to the zip.
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore ElementElement Substitution Group Message1 Element content n n Attribute Group n Attribute 1 n Attribute content Wildcard Attribute Wildcard n 1 1 Value Constraint n Simple Type Complex Type Group n n 1 Element 1 Reference Contains Logical Message Model
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Logical Message Model #1 It was noted earlier that a message definition file is actually an annotated XML Schema file. That’s because the WebSphere Message Brokers message model is XML Schema, with a few extensions and restrictions. It turns out that XML Schema is great at expressing the logical structure of all kinds of data, not just XML, and of course it’s an industry standard. The following is a quick summary of the key features of the logical message model: Each data value in a message must be modeled by a Simple Type. XML Schema supplies many built-in Simple Types and you can create your own by restricting a built-in Simple Type using Value Constraints. To refer to a data value in the message tree, you need to give it a name. That’s what an Element provides, a way of naming a data value. An Element therefore refers to a Simple Type. As well as data values, a message might have named structures. Elements therefore can alternatively refer to a Complex Type that defines the structure. The actual content of a Complex Type is described by a Group. A Group typically contains several Elements. A Group can also contain other Groups directly. In WebSphere Message Broker, we need to add an additional concept that is not in XML Schema, that of a Message. A Message is nothing more than an Element which describes an entire message. Simple Types, Complex Types, Elements and Groups can be local or global. A global object can be reused from many places. A local object can only be used at the point it is defined. Elements always have names, whether they are local or global. Simple Types, Complex Types and Groups only have names if they are global. To use a global Element or global Group within a Group, an Element Reference or Group Reference is used. These refer to the Element or Group by name. A local Element, a local Group, an Element Reference and a Group Reference can be defined as mandatory, optional or repeating, using minOccurs and maxOccurs properties. The content of a Group can either be an ordered list of objects (sequence), an unordered list of objects (all or unordered), or a choice of one of a list of objects (choice), as described by the Group’s composition property.
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Logical Message Model #2 If you are modeling XML data, there are some additional objects and concepts that you might require. An XML attribute is modeled by an Attribute which must refer to a Simple Type. Attributes can also be gathered together into Attribute Groups Both Attributes and Attribute Groups are used directly by a Complex Type, that is, they are not part of the Group that defines the main content of the Complex Type. A Group can also contain Element Wildcards. These are used when the content of the Group is not known at design time. A Complex Type can also contain Attribute Wildcards. These are used when the attributes are not known at design time. Elements can belong to something called a Substitution Group. This is an advanced XML Schema concept and will not be described further here. Simple Types of variety list and union are supported when modeling XML. XML namespaces are supported. XML Schema Unique, Key and KeyRef objects are imported but ignored. Objects in one mxsd file can refer to objects in another mxsd file in the same message set using XML Schema Include or Import. Redefine is not supported.
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore MRM Domain Flexible, model-driven parser Supports binary messages C, COBOL, PL/1, etc, structures Supports text messages Industry standards such as SWIFT, X12, HL7, FIX Common text formats such as CSV Supports XML messages Use to generate equivalent XML document MRM needs both logical and physical information Add a physical format to a message set Provides a rich set of properties capable of representing most kinds of data – Held as annotations to logical model objects <Person> <Age>32</Age> <Height>172.5</Height> <Name>Joe Bloggs</Name> </Person> 032 1725 Joe Bloggs 32,172.5,Joe Bloggs {A:32;H:172.5;N:Joe Bloggs} CWF XML TDS
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes - MRM domain The most flexible domain is called MRM which has a general purpose model-driven parser. For example, the MRM parser can perform runtime validation of messages against the model. The MRM domain supports modeling binary messages from applications written in C, COBOL, PL/1 and other languages. This support includes the ability to create a message model directly from a C header file or COBOL copybook. The MRM domain supports modeling text messages, perhaps with field content identified by tags or separated by specific delimiters or both. This includes industry standards such as SWIFT, EDIFACT, X12, HL7 and FIX, and commonly used text messages such as Comma Separated Values (CSV). The MRM domain supports modeling XML messages, including those that exploit XML namespaces. From 6.1, the improvements to the XMLNSC domain in terms of performance and XML Schema 1.0 validation mean the primary use case for MRM XML is to generate an XML equivalent of a CWF or TDS message. The MRM parser needs to know not only the logical structure of a message, as defined by the logical model, but also how to create that logical structure from a physical stream of bytes. You do this by adding a physical format to a message set. A physical format provides a rich set of properties that can be set to represent most kinds of non-XML and XML data. Binary messages are modeled using a physical format called Custom Wire Format (CWF). Text messages are modeled using a physical format called Tagged/Delimited String Format (TDS). XML messages are modeled using a physical format called XML Wire Format (XML). Physical format properties are held as XML Schema annotations to logical model objects. In the remainder of this session, we will explore how to model non-XML messages using MRM domain. We will ignore XML, as these days the majority of XML modelers will create their model in XML Schema (xsd) form and simply import into a message set.
  • Modeling MRM messages
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Approaching Message Modeling Message modeling is like programming You can read up on the theory You can learn how to use the message editor The hard part is knowing how to structure your message model KnowledgeKnowledge WisdomWisdom X
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Approaching Message Modeling Message modeling has a strong analogy with programming. Let’s say you want to learn a new programming language and write a program to solve a business problem. Take Java as an example. You buy a Java book and read up on the language theory. You get hold of a good Java editor and learn how to use it. But the hardest part is taking your business problem and working out how to structure the program that will solve it. Very often you will look at examples created by other programmers in order to get a head start. With message modeling it’s the same. You can learn the theory and you can learn how to use the message editor. But the hardest part is looking at an actual message and working out how to go about modeling it. If you are lucky the problem is solved in whole or in part by already having a model of the message in one format or another (metadata). If you are even luckier that format will be one for which an importer exists, which will create the message model for you automatically. We’ll cover importers in this session. The real fun starts when you have no model to reference. All you have is one or more example messages. This most frequently occurs for formatted text messages, such as Comma Separated Values (CSV) messages. We’ll show how to analyse messages in order to understand the message structure and create the corresponding message model, using CSV as our example.
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Three Key Tasks 2. Understand the logical structure Enables creation of the logical model 3. Relate the physical format to the logical structure Enables setting of the model physical properties 01 PERSON. 05 AGE PIC 999; 05 HEIGHT PIC 999V9; 05 NAME PIC X(48); 032 1725 Joe Bloggs 1. Understand the physical format COBOL, C, CSV, SWIFT etc CWF or TDS Msg Person Type Person Element Age Element Height Element Name Type INTEGER Type DECIMAL Type STRING 32,172.5,Joe Bloggs {A:32;H:172.5;N:Joe Bloggs}
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Three Key Tasks The first task is to understand the physical format of the message. Where does this message come from? Does it conform to an industry standard, or perhaps a well-known format like CSV? Is it from a C or COBOL application? Which of the MRM’s physical formats should be used to describe it – CWF or TDS? The second task is to understand the logical structure of your message. This determines the shape of your logical model, which in turn dictates the shape of the broker’s logical message tree. The third task is the hardest one; relating the physical format to the logical model. It is this task that determines the physical model and provides values for all the CWF, TDS or XML physical properties.
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 1. Understanding the Physical Format For messages with just binary data use the CWF format Examples are COBOL, C, PL/1 Importers are available for COBOL and C structures These automatically populate CWF format properties For messages with text markup use the TDS format Examples are SWIFT, HL7, CSV IBM sells message sets for several industry standard formats – SWIFT, EDIFACT, X12, FIX, TLOG, HL7 Samples provided for CSV messages For mixed binary and text messages use the TDS format From 6.1, TDS also handles binary data and binary markup
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 2. Understanding the Logical Structure 1. Identify complex structures Provides your – Complex Types – Groups – Complex Elements 2. Identify simple items Provides your – Simple Types – Simple Elements – Attributes 3. Identify structure ordering Provides your – composition (Sequence, Choice, All, Unordered) 4. Identify structures and items that repeat Provides your – Element minOccurs & maxOccurs
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Understanding the Logical Structure There are four stages to understanding the logical structure of your message 1. Identify the complex structures. These correspond to the Complex Types, and the Groups used within these Complex Types. There will be an overall Complex Type for the message itself. If the message contains sub-structures, there will be a Complex Type per sub-structure. For example, each parent level in a COBOL copybook, or each different row in a CSV message, corresponds to an Element of Complex Type. 2. Identify the simple items within each Complex Type and Group and their logical data type. These correspond to simple Elements and Attributes. For example each field in a COBOL copybook with a PIC clause, or each comma separated value in a CSV message, corresponds to an Element of Simple Type. 3. Identify the ordering rules of the items within each complex structure. This provides the value of the composition property for the Complex Type or Group. Typically the composition is just a sequence of one or more items (sequence). But sometimes the rule is that only one of several items can occur (choice), examples being C unions and COBOL REDEFINES. Perhaps the composition is a sequence of items but on closer examination the order of the items is not fixed (all or unorderedSet)? 4. Identify which complex structures and which simple items repeat. This provides the cardinality rules for your Elements, that is, the values for the minOccurs and maxOccurs logical properties. Is an Element mandatory (minOccurs != 0) or optional (minOccurs = 0). Are there a fixed number of occurrences of the Element (minOccurs = maxOccurs) or can the number vary (minOccurs != maxOccurs)? If the number can vary can there be an unlimited number (maxOccurs = -1)?
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 3a. Relating CWF data to the Logical Model For simple Elements Identify the physical characteristics of each data item Physical type is the key property – Strings: Fixed length, Null terminated, Length encoded – Numbers: Binary integer, Packed decimal, External (zoned) decimal, IEEE Float Length count and Length units Justification and Padding character Signed, Sign orientation, Virtual decimal point For simple and complex Elements Byte alignment Repeat reference For COBOL and C the importers create both logical and physical models For PL/1 consider mapping to equivalent C structure then fixing up
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Relating CWF data to the Logical Model For a CWF message there is nothing in the message data itself that identifies the individual fields or even the structures to which these fields belong. Fortunately most CWF messages you will encounter in the real world will be described by external metadata, for example a C header file, a COBOL copybook or PL/1 include file. For each Element of simple type in the logical model there is an allowable set of CWF Physical types. The key task is to identify this physical type as it also determines what other CWF properties need to be set. With COBOL for example, : PIC X(20) is a fixed length string, left-justified with a length of 20 bytes. PIC S9(9) is an extended (zoned) decimal, right-justified with a length of 9 bytes and signed. Because compilers align fields of certain data types on specified memory boundaries, it is important to also set correctly the CWF Byte alignment rules for each Element and Attribute. This applies to both simple and complex Elements. For CWF the number of repeats must be known in advance, the only exception being when the number of repeats is specified by a field earlier in the message. This is called a Repeat reference and can be specified instead of the logical maxOccurs property. The good news is that for the most commonly encountered CWF messages, in practice from COBOL and C applications, importers are provided that automatically create the logical and physical models, so you don’t have to do any of this ☺. For PL/1, there is no importer (yet), so consider creating the equivalent C or COBOL structure, importing that, and fixing it up post-import using the editor. Specifically, check Byte alignment and change Physical type of variable length strings to ‘Length encoded string 2’.
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore 3b. Relating TDS to the Logical Model For Complex Types and Groups Data element separation of each structure is key property – Determine this by looking at the markup in the message Does each data item have a Tag? – If so is the Tag fixed length or is there a Tag data separator? Are the data items fixed length or separated by a Delimiter? Does the structure itself have a Group indicator (GI) or terminator (GT)? For simple Elements Identify the physical characteristics of each data item Length if used Tag if used Justification and Padding character Sign orientation, Virtual decimal point, Precision For simple and complex Elements Repeat reference, Repeating element delimiter (RED)
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Relating TDS to the Logical Model TDS messages vary widely. There may, or may not, be something in the message data itself that identifies the individual fields. There may, or may not, be external metadata that describes the structure of the message (such as a specification). We’ll work on the assumption that metadata is not available, and that all we have to go on are example instances of the messages we want to model. The key step is to identify the TDS Data element separation property. To answer this question you have to look at the markup in the message. Markup is extra message content that assists with the identification or parsing of the data. Does each data item have a Tag ? If so is the Tag fixed length or is there a Tag data separator ? Are the data items fixed length or separated by a Delimiter ? Does the structure itself have a Group indicator (GI) or Group terminator (GT) ? For TDS each Element and Attribute of simple type in the logical model is physically a string in the message. So there is not the same concept of physical type like there is with CWF. But there are still TDS properties to set that describe how these physical strings appear in the message data. Certain Data element separations need to know the number of repeats in advance, while others do not. Those that do need to know use maxOccurs and minOccurs directly or a Repeat reference if the number of repeats is specified by a field earlier in the message. Additionally, where Delimiters are involved without Tags, an additional Repeating element delimiter property is needed in order to parse repeating items. The good news is that for the some of the commonly encountered industry text standards, IBM supplies pre-packaged message sets which contain the logical and physical models, so you don’t have to do any of this ☺. There’s also a set of samples for Comma-Separated Value messages (CSV) which you can use as a starting point for your own CSV messages. For non-standard formats you have to create the model yourself using either the message definition editor or SupportPac IA7D “Importer for formatted text message models”.
  • Text message formats
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS Messages - Example #1 {A:32,H:172.5,N:Joe Bloggs} A N172.5 Joe Bloggs{ }: , ::H32 , Group Terminator Group Indicator DataTag Tag Data Separator Delimiter ?Tagged Delimited Tagged Delimited
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS Messages - Example #2 32,172,Joe Bloggs CSVCSV Data Delimiter Joe Bloggs172.5 ,32 , ? All Elements Delimited All Elements Delimited
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS Messages - Example #3 32,172.5,Joe Bloggs<CRLF> 111,104.0,Bilbo Baggins CSVCSV 111 Bilbo Baggins104.0 ,, Joe Bloggs172.5 ,32 , CRLF Repeating Element Delimiter All Elements Delimited All Elements DelimitedHow many different complex types? How many different complex types? 2 2
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes - TDS Messages - Examples The previous foils show two different example TDS ‘Person’ Complex Types. The first example contains data, Tags and Delimiters. The Tags in this example are separated from the data by a Tag data separator. We would expect to use a Data element separation of Tagged Delimited to model such a Complex Type. Note that the Person child data is wrapped by a GI and a GT. The use of these is optional. In fact GI and GT can be used with all Data element separations. The second example contains just data and Delimiters. It is an example of a Comma Separated Values (CSV) message. This is a widely used format, for example it is an export format from databases and spreadsheets. The data is considered to be a ‘row’ and each simple value is a ‘column’. We would expect to use a Data element separation of All Elements Delimited to model such a Complex Type. The third example shows the most typical example of a CSV message, where the ‘row’ structure repeats multiple times, each instance separated by <CRLF>. This separator is called a Repeating element delimiter (RED). It is important to notice that there are two complex structures in the third example! The first is the ‘Person’ Complex Type itself, the second is the ‘People’ Complex Type for the overall message, containing two occurrences of the ‘Person’ Complex Type. We would expect to use a Data element separation of All Elements Delimited to model the overall message Complex Type as well. Also notice that <CRLF> has been explicitly shown, but you would not see this quite so obviously in a text editor! Always make sure that you understand the white space in your text messages! We will look at several different CSV messages in more detail later and see how they can be modeled using TDS.
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS Algorithm Tags?Unique? Y All Elements Delimited All Elements Delimited YAll fields? Y Tagged Delimited Tagged Delimited Y Tagged Fixed Length Tagged Fixed Length Y Tagged Encoded Length Tagged Encoded Length Y Var Len Elements Delimited Var Len Elements Delimited Y Fixed Length Fixed Length Y Delimited? Y Fixed lengths? N Encoded lengths? N Use Data Pattern Use Data Pattern N N N Variable fields? N Fixed lengths? NN GI Delimited? N A:32,H:172.5,N:Joe Bloggs A:03032H:05172.5N:10Joe Bloggs A:032H:172.5N:Joe Bloggs 32,172.5,Joe Bloggs 032172.5Joe Bloggs 032172.5,Joe Bloggs
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes - TDS Algorithm This foil provides an algorithm which can be applied to the physical modeling of TDS complex structures in order to deduce the Data element separation. 1. Start off looking for Tags. If the data contains Tags and the Tags are unique for each field, then the chances are it can be modeled using one of the ‘Tagged’ Data element separations. If the data is separated by Delimiters use Tagged Delimited, if the data is fixed length use Tagged Fixed Length or if the data contains an embedded length use Tagged Encoded Length. 2. If the Tags are not unique, then consider whether they could be treated as Group indicators. Remember that GIs can be used with any Data element separation. 3. If the data does not contain unique Tags, then see whether the data is separated by Delimiters. If not and the data is fixed length use Fixed Length. 4. If all the fields are delimited use All Elements Delimited, if just varying length fields are delimited and the remainder are fixed length use Variable Length Elements Delimited. 5. If none of the above quite manages to model the structure, then you should fall back to Use Data Pattern. A data pattern is a regular expression which is dynamically matched against the data in the message to determine what it is. It is often the only method of modeling certain types of structure. For example, a numeric field might be variable length, but the end is only known because the next field is alphabetic. A regular expression can be created that stops when all digits have been consumed. We’ll also see data patterns when we look at CSV messages in more detail later on. Tip: If you encounter a stray piece of markup at the end of a structure, you may be able to treat this as a Group terminator (GT). One common application of GTs is in a delimited structure when the delimiters are post-fix as opposed to the more normal in-fix. The final delimiter can often be modeled as a GT. Similarly, you can often treat a pre-fix delimiter as a Group indicator (GI).
  • Scenario
  • © 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Scenario COBOL appCOBOL app 01 CUSTOMERS. 05 CUSTOMER-DETAILS OCCURS 5 TIMES. 10 CUS-FIRST-NAME PIC X(10). 10 CUS-LAST-NAME PIC X(10). 10 CUS-STREET-ADDR PIC X(25). 10 CUS-CITY-NAME PIC X(20). 10 CUS-STATE-CD PIC X(2). 10 CUS-ZIP-CD PIC X(5). 10 CUS-POLICY-KEY PIC X(10). 10 CUS-LUPD-CNT PIC S9(4). 10 CUS-MR-FROM PIC S9(9). 10 CUS-MR-TO PIC S9(9). 01 CUSTOMERS. 05 CUSTOMER-DETAILS OCCURS 5 TIMES. 10 CUS-FIRST-NAME PIC X(10). 10 CUS-LAST-NAME PIC X(10). 10 CUS-STREET-ADDR PIC X(25). 10 CUS-CITY-NAME PIC X(20). 10 CUS-STATE-CD PIC X(2). 10 CUS-ZIP-CD PIC X(5). 10 CUS-POLICY-KEY PIC X(10). 10 CUS-LUPD-CNT PIC S9(4). 10 CUS-MR-FROM PIC S9(9). 10 CUS-MR-TO PIC S9(9). CSV appCSV app Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686 Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432 Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456 Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045 Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686 Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686 Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432 Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456 Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045 Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686 Message Broker Message Broker CWF TDS
  • © 2008 IBM Corporation Page 38 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – Scenario To provide some examples of message modeling we are going to consider a simple integration scenario. We have a COBOL application which emits a CUSTOMERS message consisting of a number of customer details records. We will model this message using CWF. There is a requirement to take these messages and forward them to an application that understands CSV messages, perhaps to display them in a spreadsheet. We will model this message using TDS. This scenario enables us to demonstrate: Creation of logical and CWF models using the COBOL importer Creation of TDS models for CSV messages using the editor Several variations of CSV message and how to go about modeling them using TDS facilities
  • © 2008 IBM Corporation Page 39 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore COBOL message Create a model using the COBOL copybook importer 01 CUSTOMERS. 05 CUSTOMER-DETAILS OCCURS 5 TIMES. 10 CUS-FIRST-NAME PIC X(10). 10 CUS-LAST-NAME PIC X(10). 10 CUS-STREET-ADDR PIC X(25). 10 CUS-CITY-NAME PIC X(20). 10 CUS-STATE-CD PIC X(2). 10 CUS-ZIP-CD PIC X(5). 10 CUS-POLICY-KEY PIC X(20). 10 CUS-LUPD-CNT PIC S9(4). 10 CUS-MR-FROM PIC S9(9). 10 CUS-MR-TO PIC S9(9). 01 CUSTOMERS. 05 CUSTOMER-DETAILS OCCURS 5 TIMES. 10 CUS-FIRST-NAME PIC X(10). 10 CUS-LAST-NAME PIC X(10). 10 CUS-STREET-ADDR PIC X(25). 10 CUS-CITY-NAME PIC X(20). 10 CUS-STATE-CD PIC X(2). 10 CUS-ZIP-CD PIC X(5). 10 CUS-POLICY-KEY PIC X(20). 10 CUS-LUPD-CNT PIC S9(4). 10 CUS-MR-FROM PIC S9(9). 10 CUS-MR-TO PIC S9(9). Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical
  • © 2008 IBM Corporation Page 40 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – COBOL message We will use the COBOL importer to create a message model from the COBOL copybook shown. We can see from the metadata that the logical structure of the CUSTOMERS message is a Complex Type that contains a single child Element called CUSTOMER-DETAILS which repeats exactly five times. CUSTOMER-DETAILS is itself complex, and its Complex Type contains a sequence of ten simple Elements, being a mixture of strings and integers, none of which repeat. Firstly we create a message set project and message set to hold the model, using the New Message Set wizard. This asks you for the kind of data you will be modeling, so select binary data. This automatically creates a CWF physical format. This ensures that the importer will create a physical model as well as a logical model. The COBOL importer is invoked via the New Message Definition File From COBOL wizard. Once the COBOL copybook and target message set have been selected, a pre-import check takes place to ensure that the copybook is syntactically correct and to analyze the copybook for level 01 structures. You then select the structures you want to import. By default a level 01 structure is imported as a Complex Type. You can choose which of your selected structures to make into actual Messages. The wizard also lets you choose some extra options, such as which compiler originally compiled the copybook, to ensure that your created model exactly matches the real- world message.
  • © 2008 IBM Corporation Page 41 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – COBOL message Once the import is complete, a new message definition file called Customers will have been created. Opening this shows the message model. You can see how the logical structure mirrors the original copybook, and you can examine the Element CWF property values and relate them back to the COBOL field definition PIC clauses. A command line version of the importer is also supplied, called mqsicreatemsgdefs. This is capable of bulk importing several copybooks at once. The C importer is very similar in operation. You will notice that the wizard lets you choose a target namespace for your message definition file. Why might we want to specify a target namespace for a COBOL message? The MRM parser, when parsing messages that are defined in a message definition file that has a target namespace, produces a logical message tree containing both name and namespace information. It does this regardless of the physical format of the message. For CWF and TDS messages, the namespace is obtained from the message definition file. If you will be transforming a COBOL message into a namespace-aware XML message, the transform must map the logical message tree created for the COBOL message to a logical message tree that matches the XML message. If the COBOL message tree does not contain namespace information then each mapping from a COBOL field to an XML element must set the namespace for the XML element. However, if the COBOL message tree already contains the same namespace information then this mapping is greatly simplified.
  • © 2008 IBM Corporation Page 42 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CSV message Create an equivalent CSV physical model Re-use the same logical structure Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686 Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686 Delimiter Repeating Element Delimiter All Elements Delimited All Elements Delimited Occurs=5 Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical
  • © 2008 IBM Corporation Page 43 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message To exchange data with the CSV application we need an equivalent CSV message. We don’t have any metadata to describe this, but we do have example instances of what the messages must look like. We know that we will model CSV using TDS. We can also see that the logical structure of the CSV message is the same as the COBOL message. This simplifies our task as we can simply add a TDS physical format to our existing message model and then populate its properties. Being a CSV message we can see the expected Delimiters and also Repeating element delimiters. Note that if we were to view this data in an editor we would not see <CRLF>, so it is easy to ignore this markup accidentally! Because only data, Delimiters and Repeating element delimiters are involved, the appropriate TDS Data element separation for a CSV row is All Elements Delimited so we can set the CUSTOMER-DETAILS Complex Type TDS property accordingly. We can also set the Complex Type TDS Delimiter property to be comma, and we can set the CUSTOMER-DETAILS Element TDS Repeating Element Delimiter property to <CR><LF> (TDS supplies mnemonics for special characters). Similarly the appropriate Data element separation for the overall CSV message is All Elements Delimited so we can set the CUSTOMERS Complex Type TDS property accordingly. We can also set the CUSTOMERS Complex Type TDS Delimiter property to be <CR><LF>. Interestingly, because the CUSTOMERS Complex Type has only one child Element, and Delimiters in CSV are in-fix (meaning they only occur between Elements) there is no example of the CUSTOMERS Delimiter in the message data. But because this is a CSV message we know that the Delimiter should be <CR><LF> (we’ll see this on the next foil). In 6.1, setting the TDS message set property Messaging Standard to ‘CSV’ will automatically set many of the properties mentioned above.
  • © 2008 IBM Corporation Page 44 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CSV message #2 Add Header and Trailer CSV rows #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF> 5,349634 #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF> 5,349634 Delimiter Delimiter All Elements Delimited Occurs=5 CLASH Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical Delimiter Delimiter All Elements Delimited All Elements Delimited All Elements Delimited Why is the delimiter clash not a problem? Why is the delimiter clash not a problem?
  • © 2008 IBM Corporation Page 45 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message #2 The CSV message we have been looking at consists of a single row that repeats five times. Commonly CSV messages start with a Header row and finish with a Trailer row so let’s add these to our message. Clearly adding a Header and a Trailer changes the logical model. We need to create a Complex Type for the Header and a Complex Type for the Trailer, then create Header and Trailer Elements that use those Complex Types and insert them into the appropriate place in the overall CUSTOMER Complex Type. In fact, rather than disturb the original CUSTOMERS Complex Type, we will create a new Complex Type called CUSTOMERS2, to consist of the Header Element, the CUSTOMER_DETAILS Element (repeating five times) and the Trailer Element. Because the Header and Trailer Complex Types are just examples of CSV rows we will model them in TDS in exactly the same way we did for CUSTOMER_DETAILS, namely All Elements Delimited with a Delimiter of comma. The overall CUSTOMERS2 Complex Type is modeled in TDS as All Elements Delimited with a Delimiter of <CR><LF>, exactly like CUSTOMERS. In the message data we can now see <CRLF> in use as both a Delimiter and a Repeating element delimiter. We call this a delimiter clash. Most text based messaging standards deliberately avoid clashes like this for clarity reasons, but with CSV it is unavoidable. In the message shown the clash is not a problem, because the MRM parser knows to expect exactly one Header, exactly five CUSTOMER_DETAILS, and exactly one Trailer. In other words, the logical Max occurs property enables the MRM parser to unambiguously interpret each <CRLF> it encounters.
  • © 2008 IBM Corporation Page 46 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CSV message #3 Number of CUSTOMER-DETAILS records can vary Clash is now a problem – use Tags Header,#CUST1,2005-06-01<CRLF> Cust,Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Cust,Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Cust,Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Cust,Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Cust,Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF> Trailer,5,349634 Header,#CUST1,2005-06-01<CRLF> Cust,Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Cust,Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Cust,Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Cust,Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Cust,Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF> Trailer,5,349634 Unbounded Tag Tag Data Separator Tagged Delimited CLASH Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical All Elements Delimited All Elements Delimited All Elements Delimited
  • © 2008 IBM Corporation Page 47 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message #3 The Delimiter clash becomes a problem when the value of minOccurs and maxOccurs for CUSTOMER_DETAILS are no longer the same. If we change the message so the repeats of CUSTOMER_DETAILS become unbounded, the MRM parser is no longer able to tell whether <CRLF> means the end of an individual repeat or the end of the entire Complex Type . One way to solve this is to see whether the CSV row itself gives us a clue as to what type it is. In the message we can see that each row has now been appended with a new first column value, which appears to be fixed for a given type of row. If we choose to interpret this fixed value as a TDS Tag, instead of as data, then the MRM parser can use this information to know which Complex Type it has encountered. Note also that we interpret the comma after the fixed value as a Tag data separator and not as a Delimiter. We model this in TDS by giving each of the Header, CUSTOMER_DETAILS and Trailer Elements an appropriate Tag property, and by changing the CUSTOMERS2 Complex Type to have a Data element separation of Tagged Delimited and a Tag data separator property of comma. The Header, CUSTOMER_DETAILS and Trailer Complex Types do not change; their Data element separations remain All Elements Delimited, and no new Element is needed for the first column value because it is being treated as markup and not as data.
  • © 2008 IBM Corporation Page 48 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CSV message #4 Number of CUSTOMER-DETAILS records can vary Clash is now a problem – use Data Patterns #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF> 5,349634 #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF> 5,349634 Unbounded CLASH [0-9].* [^0-9].*rn #.*rn No Tags! Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical All Elements Delimited All Elements Delimited All Elements Delimited Use Data Pattern Group Terminator Group Terminator
  • © 2008 IBM Corporation Page 49 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message #4 If we cannot use Tags in this manner, because all the CSV rows do not contain a fixed value in their first column, then we have to use data patterns. To use this technique, we must look at the message data in the Header row, the CUSTOMER_DETAILS row and the Trailer row and work out a regular expression for each that a) distinguishes the row apart, and b) covers all the data and markup in the row. For Header we can use a regular expression of #.*rn which means “any sequence of characters that starts with # and ends with <CRLF>”. For Trailer we can use a regular expression of [0-9].* which means “any sequence of characters that starts with a digit”. For CUSTOMER_DETAILS we can use a regular expression of [^0-9].*rn which means “any sequence of characters that does not start with a digit and end with <CRLF>”. This works because we know the CUSTOMER_DETAILS data can never start with # and the Trailer data can never start with something other than a digit. We model this in TDS by giving each of the Header, CUSTOMER_DETAILS and Trailer Elements the appropriate TDS Data pattern property, and by changing the CUSTOMERS2 Complex Type to have a Data element separation of Use Data Pattern. Note also that a side-effect is that the <CR><LF> is longer treated as a delimiter, and must therefore be modeled as a Group terminator. So while the Header, CUSTOMER_DETAILS and Trailer Complex Types do not change their Data element separations (they remain All Elements Delimited), the Header and CUSTOMER_DETAILS Complex Types must have Group terminator set to <CR><LF>. It should be noted that when using data patterns to parse a message, the MRM parser does not use maxOccurs, it continues to consume the message until the regular expression fails to match.
  • © 2008 IBM Corporation Page 50 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message #4 - Variation If we look again at our example message, the data pattern technique is only actually needed to distinguish Trailer from CUSTOMER_DETAILS. An alternative logical model is shown below. Here, the overall CUSTOMERS2 Complex Type remains with Data element separation All Elements Delimited, but we have changed the logical structure by inserting a local anonymous sequence with a Data element separation of Use Data Pattern purely to parse CUSTOMER_DETAILS and Trailer. This is a good idea if there are several other child Elements prior to CUSTOMER_DETAILS in the CUSTOMERS Complex Type, as it avoids having to think up unique regular expressions for all the other child Elements (which might not be possible). It’s also a good idea from a performance point of view, because Use Data Pattern is the slowest Data element separation in TDS. This is just one example of inserting a Group to scope and solve a TDS parsing problem. It’s a very useful and powerful technique, can be used with any TDS Data element separation, has no effect on the broker’s logical message tree, and is sometimes the only way to model the message…as we shall see on the next foil. Use Data Pattern All Elements Delimited All Elements Delimited All Elements Delimited All Elements Delimited
  • © 2008 IBM Corporation Page 51 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore CSV message #5 What if a field contains the Delimiter as part of the data? 6.1: Handled automatically if TDS Quote Character selected Pre 6.1: Use Data Pattern to parse the field #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,”461, Ocean Boulevard”,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF> 5,349634 #CUST1,2005-06-01<CRLF> Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF> Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF> Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF> Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF> Eric,Clapton,”461, Ocean Boulevard”,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF> 5,349634 (”[^”]*”) | ([^,rn]*) Use Data Pattern Understand physical Understand logical Relate physical to logical Understand physical Understand logical Relate physical to logical
  • © 2008 IBM Corporation Page 52 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Notes – CSV message #5 Let’s look at one final variation on the CSV theme. It can arise that a column value validly contains a comma. When this happens the usual action is for the value to be placed in double quotes, which acts as an escape mechanism to prevent the interpretation of the embedded comma as a Delimiter. In our message, this has happened with the CUS_STREET_ADDR Element in the final repeat of CUSTOMER_DETAILS. TDS provides an escape scheme for escaping individual characters. In 6.1 this allows quotes to be designated as the escape scheme, so in 6.1 the fact that CSV data is quoted is not a problem and the parser handles it seamlessly. However, prior to 6.1, escape schemes did not handle quotes. To solve this problem we once again use a Data pattern. We need to provide a regular expression that accepts either an unquoted value that contains no comma, or a quoted value that can contain a comma. Such a regular expression would be (”[^”]*”) | ([^,rn]*) which means “either any sequence of characters except quote preceded and followed by a quote, or any sequence of characters that does not include the comma or <CRLF>”. The CUSTOMER_DETAILS Complex Type remains with Data element separation All Elements Delimited, but we change the logical structure by inserting a local anonymous sequence with a Data element separation of Use Data Pattern purely to parse CUS_STREET_ADDR, which is given the Data pattern above. The model shown enables the parsing of quoted and unquoted values, but it does not cope with extra quotes that validly occur in data values nor does it strip off the quotes before inserting the value in the message tree. Both these can be achieved by using a further embedded group and more complex regular expressions combined with a GI and a GT. Finally, the Samples Gallery from 6.0 onwards contains a sample message set that illustrates many of the CSV variations described here.
  • Best Practices : Diagnosing modeling problems
  • © 2008 IBM Corporation Page 54 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Diagnosis tools Debugger Allows single-stepping through nodes and ESQL Displays the message tree at a breakpoint Warning: Message tree may not be an exact representation of the real tree Trace node Provides a 100% accurate representation of the message tree Can write to file or user trace User Trace Shows the full text of all error messages Logs TDS parser activity Can contain Trace node output
  • © 2008 IBM Corporation Page 55 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS user trace output Debugging TDS parsing problems Can be quite difficult to debug a complex TDS model ‘What was the parser doing before it failed?’ In v6.1 you can find out User Trace ( debug level ) TDS parser issues user trace messages at important stages of the parse UserTrace BIP5612I: ‘ELEM1:', ending at byte ‘5', was matched as the tag for '/myStructure/myElement'. UserTrace BIP5609I: '#' has been matched as a delimiter for '/myStructure' at byte '17'.
  • © 2008 IBM Corporation Page 56 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Diagnosis Best Practices Use the debugger as a ‘first line of defence’ only Debugger only tells you what happened User Trace should be a standard diagnosis tool User trace tells you why it happened Be wary of the message tree displayed by the debugger Use a Trace node if you want to be sure of an accurate picture Trace nodes write to User Trace, so they make a good pair
  • © 2008 IBM Corporation Page 57 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Best Practices : XML Parsing XMLNSC XMLNSC is the preferred XML parser for new flows Validation is now available in the XMLNSC domain Performance of XMLNSC is ~100% better than any other XML parser ( and sometimes better than that ) XMLNSC validation is far less costly in CPU than MRM validation Opaque parsing Performance feature now available in XMLNSC Configurable via the input node properties XML domain XML domain is now ‘deprecated’ All existing message flows will continue to work in future versions. ( No function is actually being withdrawn )
  • © 2008 IBM Corporation Page 58 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore TDS enhancements in v6.1 New ‘CSV’ Messaging Standard Includes support for CSV-style quoting of strings Tags / delimiters can contain raw byte values Support for a wide range of CWF physical types Repeat reference Exactly like the CWF support Automatic trimming of over-size Fixed Length data Also added to CWF Support for SAP text Idocs IBM-supplied message for CC/DD message C importer populates TDS physical format(!) Note : v6.1 TDS is not a replacement for CWF no support for alignment/padding of structures Not all physical types are supported
  • © 2008 IBM Corporation Page 59 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Summary Understand the physical format CWF or TDS When you already have metadata Use an importer if possible When you don’t or can’t use an importer Understand the logical model – Structures, simple items, composition, repeats Relate the physical format to the logical model – For CWF, Physical type property is key – For TDS, markup is key to deciding Data element separation property Be smart about diagnosis Use User Trace / Trace node as well as debugger Don't forget to complete your evaluation ! Session xxx
  • SundarGopal Venkatraman sundaven@in.ibm.com Session Number: Q30Session Number: Q30 WebSphere Message Broker Administration Best Practices
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Objectives This session discusses the common administration tasks and best practices for WebSphere Message Broker. Section at end suitable for advanced users General Knowledge of WebSphere Message Broker is assumed • All statements will apply to V6 and V6.1 unless otherwise stated
  • © 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Acknowledgements Matt Lucas (WMB Development)
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Administration Architecture Common administrative actions and how best to accomplish them Power users section Agenda
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Administration Architecture Message Broker Toolkit (Admin Perspective) Message Broker Explorer (IS02) Command line Configuration Manager Proxy Configuration Manager Broker
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Message Broker Explorer (IS02) Alternative MQ Explorer based administration – Simplifies administration of MQ,MB networks in single Explorer console – Uses MQ Eclipse ‘extension points’ to provide seamless experience Comprehensive administration facilities – All features in Broker Administration (excluding publish/subscribe) – Also includes new features such as multi Execution Group deploy – Offload WS-Security to Data Power appliances IS02 Cat 3 SupportPac – Fully supported in production Performance Monitor – Easily view CPU, IO and other metrics – Requires APAR IZ07803
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Configuration Manager ProxyComprehensive Java API for administering broker domains: import com.ibm.broker.config.proxy.*; public class CreateBrokerReference { public static void main(String[] args) { ConfigManagerProxy cmp = ConfigManagerProxy.getInstance(…); TopologyProxy topology = cmp.getTopology(); topology.createBroker(“MYBROKER”, “QMGR”); } } Used for administering brokers, execution groups, message flows, dictionaries (and other deployed message flow dependencies), subscriptions, the publish/subscribe topology, collectives, the event log, topics hierarchy and the Configuration Manager The Message Broker Toolkit Administration perspective, IS02 and various commandline utilities are visualizations of this API. Powerful samples available (e.g. Configuration Manager Proxy Exerciser)
  • © 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Configuration Manager Proxy API Exerciser
  • © 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Commands – three types Commands that perform “direct” administration Tend to contain an unflagged component name parameter (e.g. mqsilist BROKER) Run locally on the machine on which the component exists Examples: mqsilist, mqsicreatebroker Commands run in Eclipse “headless” mode Parameters to specify workspace or project location Examples: mqsicreatebar, mqsicreatemsgdefs Commands based on the Configuration Manager Proxy Parameters to specify location of Configuration Manager Examples: mqsideploy, mqsicreateexecutiongroup
  • © 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Common Administrative Actions Create/Delete components Controlling access to resources Versioning of resources Configuring broker archive files Deploying broker archive files Operational Tweaks
  • © 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Naming Standards Configuration Manager • Functional Domain name • Prefix/Suffix of QMGR Name ? • Prefix/Suffix ‘CMGR’ ? Broker • Functional Project/Application Name • Prefix/Suffix of QMGR Name ? • Prefix/Suffix ‘BROKER’ ? Execution Group • Application/Project Name • Prefix/Suffix of ‘EG’ ? Message Flow • Mediation/Application Name • Prefix/Suffix of ‘FLOW’ ? Create/Delete components Ensure consistent naming convention
  • © 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore How many Configuration Managers do you need ? Create/Delete components Configuration Manager How many brokers/environments and their complexity? Are there different classes/types of applications deployed? Security and Isolation Ease of administration If each environment has multiple brokers, it is recommended to have separate Configmgr’s. e.g. DEV, TEST, etc ..
  • © 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore http://www.ibm.com/developerworks/websphere/library/techarticles/0706_lucas/0706_lucas.html …..…Or Google “Config Manager Secrets” Create/Delete components
  • © 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore How many Execution Groups per Broker can you have ? Create/Delete components Complexity of message flows Message sizes that are expected to be processed Environment limitations e.g. hardware, operating system Are there different classes/types of applications deployed ? Ease of administration Broker/Execution group/Message flow How many Message flows per execution group can you have ?
  • © 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore http://www-128.ibm.com/developerworks/websphere/library/techarticles/0311_dunn/dunn.html …..…Or Google “Message Flow copies” Create/Delete components Applicable to all Broker versions
  • © 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Controlling access to resources MQ Configuration • Control access to the Configuration Manager and broker’s queues and queue manager: – Who should have access to the Configuration Manager’s queues and queue manager? – Who should have access to the broker’s queue manager? And broker system queues? • Use SSL between MBT and CM (and between Config Manager and brokers if necessary) • Consider using the same queue manager for both broker and Config Manager – This will minimize MQ administration, but having multiple CMs may increase administration cost Configuration Manager ACLs Runtime authentication / authorization / identity mapping with LDAP or TFIM (MB v6.1 only) Message Brokers Toolkit Configuration Manager Broker EG
  • © 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Runtime versioning Various pieces of information gathered at deployment time • The time the object was deployed • The time the object was last saved in the broker archive • The name of the broker archive file that deployed the object • Other user-defined keywords and their values – For instance, “Author” -> “Prasad Imandi”, “Release name” -> “S000” Viewable in Message Broker Toolkit, mqsireadbar and CMP applications e.g:
  • © 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Runtime versioning – Embedding keywords The Configuration Manager scans through each file looking for a predefined eye catcher that describes each keyword and its value Each time one of these strings appears in the file being deployed, it is associated with the Configuration Manager’s record for that file However, message flows and dictionaries have a freeform ‘Version’ property, which means that for simple versioning scenarios you don’t need to remember the $MQSI..MQSI$ syntax: $MQSI keyword = value MQSI$
  • © 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Embedding Version Information – “How to” examples Message flows: • Use the string field of a passthrough node • Use ESQL strings: Set target = ‘$MQSI target = production MQSI$’ • When using subflows, use a different version keyword in each subflow; this allows you to see version information for each constituent part Dictionaries • Use ‘Version’ property only Stylesheet • In a comment or unused field <?xml version=”1.0” encoding=”UTF-8”> <!-- $MQSI author = Matt MQSI$ $MQSI Version = v1.0 MQSI$> <xsl:stylesheet version=”1.0”> <xsl:output method=”text” encoding=”UTF-8”/> <xsl:template match=”/”> <xsl:value-of select=”message”/> </xsl:template> </xsl:stylesheet>
  • © 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Embedding Version Information – ZIP format files ZIP based-files can be deployed inside BAR files • For example, JARs for Java Compute nodes Because the zip format used by these file types is binary and therefore not easy to embed eye catchers, the Configuration Manager will not scan the zipped envelope for $MQSI..MQSI$ occurrences. Instead, use the file META-INF/keywords.txt inside the zip if you want to associate keywords with these files: META-INF/broker.xml Flow1.cmf Flow2.cmf JavaCompute.jar META-INF/Manifest.mf META-INF/keywords.txt com/mycorp/Compute.class com/mycorp/Helper.class $MQSI Version = v1.2 MQSI$ $MQSI Author = Matt MQSI$ Deployed BAR Embedded JAR BAR JAR keywords.txt
  • © 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Linking with version control systems Many version control systems (e.g. CVS) use keyword systems that allow information to be expanded when files are checked out. For example: – CVS (RCS): – $Revision$ expands to $Revision: 1.3 $ – SCCS: – %I% expands to 1.3 Use these tags inside $MQSI..MQSI$ eye catchers to conveniently embed source code repository information: – $MQSI MyVersion = %I% MQSI$ – $MQSI MyRevision = $Revision$ MQSI$ When files containing these eye catchers are checked out, the version control system will resolve their values accordingly.
  • © 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Synchronous vs. Asynchronous Deployment IS02 supports both deploy types (via “Run in Background”) All deployment methods use the CMP • CMP supports both synchronous and asynchronous deployment models Drag BAR File onto EG CM extracts BAR and sends to broker MBT reports “deploy initiated” New Flows start. Event Log + Domains view updated MBT Admin Perspective … mqsideploy Enter deploy command CM extracts BAR and sends to broker New Flows Start Results displayed and finish CM and broker respond to command
  • © 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Configuration Tweaks Two problems: • Broker Application Developers knowledge of runtime environment, vs. Broker Administrators knowledge of message flows • Deployment can be an operationally expensive operation, even if not technically expensive Use Deployment Descriptors before deployment • Message Broker Toolkit (BAR file editor) • mqsiapplybaroverride • Configuration Manager Proxy (V6.1 only) An increasing number of properties can be changed after deployment • Configurable Services Information – FTP, SMTP, JMS, JNDI, LDAP etc. • Additional Instances • Registry Properties etc. • Use mqsichangeproperties or Configuration Manager Proxy
  • © 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Advanced Administration Topics How the various components interact What to do if deployment goes wrong Diagnosing the state of your domain Recovering your domain Broker / Config Manager Synchronization
  • © 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Inter-component Communication CMP S.B.CONFIG.REPLY SYSTEM.BROKER.CONFIG.QUEUE Config Manager Broker S.B.ADMIN.QUEUE S.B.ADMIN.REPLY Now consider the scenario of deployment not working…
  • © 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deployment – Problem Determination Any error messages? • Event log viewer/command output/DeployResult • System log Are the components running? • Alerts view • System log • ps/Task Manager etc. Check queue depths Use supplied tools
  • © 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Tools to Determine the State of your Domain Configuration Manager Proxy API Exerciser mqsideploy –cancel (only if there is no chance that the broker could have picked up the deployment message) Only at the direction of IBM Service (or the foolhardy): • mqsicmdt • mqsicmrv Also: Documentation, mqseries.net, Tech docs, IBM Education Assistant, IBM Service etc.
  • © 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Advanced Tracing of the Configuration Manager mqsistop [CMNAME] set MQSI_CONSOLEMODE=1 bipconfigmgr [CMNAME] mqsistop [CMNAME] export MQSI_CONSOLEMODE=1 export MQSIJVERBOSE=-DMQSI_CONSOLEMODE=1 bipconfigmgr [CMNAME] mqsichangetrace [CMNAME] -t -b -l debug Win UNIX
  • © 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Configuration Manager in Foreground Mode
  • © 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Recovering a broker A broker is made up of four resources: Holds message flow, dictionary, pub/sub and other configuration Holds XSL stylesheets, JARs, Adapter Configuration files, XSDZIPs etc. ($MQSI_WORKPATH)Holds some configuration information (mqsicreatebroker) Holds in-flight aggregations and collections Database File System Registry Queues …plus any resources your message flows require
  • © 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Recovering your Configuration Manager A Configuration Manager is made up of three resources: Holds configuration information set by mqsicreateconfigmgr Holds all other configuration information! Holds unprocessed messages from the broker and CMP applications Queues Registry Repository Safeguarding the repository is therefore key to the Configuration Manager’s survival! • Use mqsibackupconfigmgr on a regular basis
  • © 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Recovering a Configuration Manager Repository Restore from backups, or Recreate your Configuration Manager, and then: • V5 or V6: Delete your brokers and redeploy everything • V6: Add each broker back to the domain, set the UUID of each broker using the CMP API Exerciser. Existing flows cannot be managed. Use Remove Deployed Children to reset the broker. Then redeploy everything. • V6.1 only: Have the Configuration Manager “adopt” each broker in your domain V6.1 adoption will cause the Configuration Manager to query the broker for its execution groups, message flows and other configured properties
  • © 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Adopting a broker
  • © 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Synchronization (V6.1 only) A solution to a Configuration Manager that is “out-of-sync” The Configuration Manager is able to synchronise itself with every broker in the domain • After the first successful deployment • During the “adoption” process • When the Configuration Manager starts • Every sixty minutes (configurable) • If you call the CMP method BrokerProxy.discoverConfiguration() The Configuration Manager compares its definition with what the broker is actually running, and updates the definition accordingly • Execution Groups • Message Flows • Properties set using mqsichangeproperties • Registry Properties • Configurable Services
  • © 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Administration Architecture • Message Brokers Toolkit • Message Broker Explorer (IS02) • Command line • Configuration Manager Proxy Common administrative actions and how best to accomplish them • Components • Controlling access to resources • Versioning of resources • Deploying broker archive files Power users section • How the various components interact • What to do if deployment goes wrong • Diagnosing the state of your domain • Recovering your domain • Broker / Config Manager Synchronisation Summary
  • © 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore References Managing WebSphere Message Broker Resources in a Production Environment: http://www.redbooks.ibm.com/abstracts/sg247283.html Coordinated deployment in WebSphere Message Broker V6 using the Configuration Manager Proxy API http://www.ibm.com/developerworks/websphere/library/techarticles/0611_lucas/0611_lucas.html WebSphere Message Broker deployment scripting using Ant: http://www.ibm.com/developerworks/websphere/library/techarticles/0706_spriet/0706_spriet.html
  • © 2008 IBM Corporation Page 36 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Questions Please complete the session evaluation form. Thank you!!
  • © 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore AIM Technical Support and Customer Service Providing innovative support IBM Web Support: Provides product support pages with searchable in-depth technical information, interim fix downloads, technical product training and Webcasts, subscriptions to stay current on product news, troubleshooting guidance documents, online problem management tools, product lifecycle links, and related links to reference material and support options. These timesaving resources make the support process more efficient and the experience more successful. Visit the IBM Software Support poster session and for future reference: ibm.com/software/websphere/support/d2w.html ibm.com/software/support/ IBM Education Assistant (IEA): Provides short, focused, multimedia modules designed to help customers solve problems independently. Modules consist of Flash presentations and demonstrations, tutorials, and links to external resources that help customers gain a better understanding of IBM products and use them more effectively to meet their business requirements. Content is presented in an organized navigational hierarchy and hosted on a highly available IBM Eclipse Help System framework. ibm.com/software/info/education/assistant/ IBM Support Assistant: Provides a free application that simplifies support and helps customers research and resolve issues with IBM software. The Support Assistant includes an extensible search capability, automated data gathering, troubleshooting tools, and other features to help customers resolve problems on local and remote systems. For more information, visit the Support Assistant Poster Session or our Web site. ibm.com/software/support/isa/ Premium Support: Provides the ability for customers to purchase a level of proactive technical support and customized services to help them reach their business goals. This service provides personalized support services including a consistent, dedicated contact for project management, migration planning, rapid escalation of critical issues, improvement of customer system reliability, and availability and enhancement of technical expertise of customer personnel. ibm.com/software/support/premium/ To learn more about the AIM Technical Support and Customer Service organization, visit our Web site at aimcustomersupport.raleigh.ibm.com. Archive Explorer (AE): Provides L2 support engineers with a front end to the customer data repository (ECuRep). AE helps reduce time to resolution for customer problems by allowing efficient access to customer files, facilitating communication with customers, and providing tools for diagnostics and problem analysis. ecurep.mainz.de.ibm.com/aex Serviceability Tools: Provides diagnostic tools in Archive Explorer and IBM Support Assistant which help customers troubleshoot problems they might experience in a variety of IBM software products and runtime environments such as WebSphere Application Server and Java Virtual Machines. Problems that the tools can help solve include: memory leaks, deadlocks, crashes, and performance problems. Examples include: the Memory Dump Diagnostic Tool for Java, Thread Analyzer, and the Visual Configuration Explorer.
  • WebSphere MQ and the Web Session Number: F06Session Number: F06 Yak Yew Cheng yakyc@sg.ibm.com
  • 2008 IBM Corporation Page 2 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore © IBM Corporation 2008. All Rights Reserved. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete list of IBM trademarks, see www.ibm.com/legal/copytrade.shtml AIX, CICS, CICSPlex, DB2, DB2 Universal Database, i5/OS, IBM, the IBM logo, IMS, iSeries, Lotus, OMEGAMON, OS/390, Parallel Sysplex, pureXML, Rational, RCAF, Redbooks, Sametime, Smart SOA, System i, System i5, System z , Tivoli, WebSphere, and z/OS. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
  • 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WMQ and WS-* - Standards-Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction: Extending Application Reach Using MQ New distributed programming models are becoming widely adopted SOA Web services – W3C / OASIS style Web services – standards based, interoperable e.g. WS-* Programmable web – Ad-hoc web services – simple, lightweight, pragmatic e.g. REST IBM is extending MQ to take advantage of these new models… To Embrace Standards Based (“Enterprise”) Web Services SOAP/JMS standard Windows Communication Foundation channel WMQ Service Definition To Allow Access From “Zero Footprint Clients” and the Web Goal: Easy access to MQ from HTTP and the World Wide Web – HTTP ‘REST’ Gateway for direct HTTP client access (0FC and Web) – PHP language support for PHP server apps
  • 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Extending the Reach of Applications Using MQ Recent years have seen a growth of interest and adoption of distributed programming styles, led primarily by the success of standards based Web services. The success of Web services standards based on SOAP (WS-*) has been based on the promise of greatly improved code re-use and component interoperability through the development of composite applications, built from services; using a variety of qualities of service; and accessible over a range of communication transports. The reliable and asynchronous qualities of MQ provide an excellent compliment to the HTTP’s less reliable synchronous protocol. To make MQ easier to use in service oriented architectures, IBM is engaged in defining standards for transporting SOAP over MQ (and JMS), and in defining service definitions so that native MQ applications can be catalogued as a first-class part of an SOA, and consumed in composite applications. The success of Web services has been dramatic, but they are not seen as the best solution to all problems. The specifications are geared towards recreating many of the traditional Enterprise qualities of service (transactionality etc) over interoperable XML protocols. This makes the specifications complicated, and the XML-based protocols are relatively heavyweight. Web services implementations rely heavily on tools to hide this complexity which pushes up the cost-of-entry in terms of skills and software product requirements. For simpler scenarios, a section of the software development community is advocating more basic APIS and protocols – typically based on HTTP so that they can be invoked directly from Rich Internet (Web 2.0) Applications. These APIs are geared towards keeping simple things simple, and so deliberately steer clear of the complexities of Enterprise qualities of service, leading to a lower cost-of-entry than SOAP and WS-*,. Many of these APIs follow the REST best practices discussed later in this presentation, and use plain old XML (POX) or JavaScript Object Notation (JSON) message formats. IBM is embracing the lightweight HTTP programming style by adding an HTTP gateway to WebSphere MQ. This will allow Web based clients to access data flowing through an MQ network, and to invoke MQ applications. It also allows low QoS “Zero footprint” client applications to access MQ over HTTP, eliminating the need to deploy MQ client libraries when client applications are deployed. N O T E S
  • 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WS-* - Standards Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 7 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Part 1a - HTTP-MQ Goals: 1. Simplifies access to MQ from Rich Internet Applications Gives AJAX and Web 2.0 applications access to the Enterprise – Submit data direct to queues & topics from Browser – Web Pub/Sub – Stock price update, Sports scoreboard, Airport and Rail Departures / Arrivals Notification 2. “Zero Footprint Client” Enable MQ Application Connectivity from any Platform or Language with HTTP capabilities Significantly increase range of supported platforms – e.g. – Linux distros, POS terminal running Windows Services for Unix environment, RFID reader, Mobile devices Reduce cost of MQ Client deployment – No client library installation required Lightweight (low qualities of service) messaging
  • 2008 IBM Corporation Page 8 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore HTTP-MQ Goals Web Browsers provide a near ubiquitous platform for deploying client applications HTTP is the pervading transport protocol The first goal of the MQ HTTP feature is to extend the reach of MQ applications to these environments over HTTP This will give Rich Internet Applications simplifed access to the Enterprise For example web-based clients will be able to submit data (e.g. forms) as MQ messages directly to enterprise batch applications Req/resp MQ applications can be fronted with web-based clients Pub sub – data flowing through network Another goal is to provide access to MQ from many more environments Eliminating the MQ client will reduce the cost of application deployment Though of course this is not a complete replacement for the MQ client It is missing many MQI features and does not offer transactionality, assured delivery etc. But in many cases where applications have resend logic and check for duplicates it will be good enough N O T E S
  • 2008 IBM Corporation Page 9 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Two Implementation Options Available 1. SupportPac MA0Y: HTTP Bridge implemented as a Servlet Suitable for deployment to existing J2EE app. servers (WAS, Tomcat, Jetty) Back end connectivity uses the WMQ JCA resource adapter and WMQ JMS JMS ConnectionFactory configuration determines if MQ connection uses client mode or bindings 2. SupportPac MA94: Standalone HTTP Listener Available for Windows, AIX, and Linux Client or bindings mode, Security exits http://mq.com:1415/msg/ queues/myQ topics/stocks/IBM libwww Web Browsers Javascript / AJAX HTTP clients Java SVRCONN CHANNEL Client or Bindings Connection Queue Manager MA0Y Servlet (MQ HTTP Bridge) MA0Y Servlet (MQ HTTP Bridge) MQ JCA + JMS MQ JCA + JMS J2EE Application Server MA94 Standalone HTTP listener (httplistener.exe) MA94 Standalone HTTP listener (httplistener.exe) REST style HTTP: POST / GET / DELETE
  • 2008 IBM Corporation Page 10 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Architectural Overview The functionality is in two SupportPacs – MA0Y which is a servlet based implementation which can be deployed in any J2EE app server, or MA94 which is a standalone executable HTTP listener which runs on Windows, AIX, and Linux. Front end connectivity is over HTTP(S) into the Application Server, or HTTP into the Listener, enabling any HTTP capable device or scripting language to send and receive messages. This could include browser based web 2.0 applications using AJAX to dynamically refresh data, server side applications, or mobile/point of sale devices. For the MA0Y, backend connectivity is supported by using the WMQ JCA adapter or WMQ JMS, the choice of which depends on the application server. Either way, the connection mode that’s used to connect to MQ can be configured using a JMS ConnectionFactory, both Bindings and clients modes are supported. Authentication and access control to URI’s must be configured in the application server. MA94 also supports Client or Bindings connections to the queue manager, is secured by a security exit, and includes some additional features like a “memory” of messages delivered to topics The HTTP interface is modelled on REST N O T E S
  • 2008 IBM Corporation Page 11 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Background – HTTP-MQ Loosely Modelled on REST REpresentational State Transfer (Roy Fielding) o Everything is modeled as a Resource o Every resource is identified by an address (URI) o Resources have state (representation) o HTTP is used to transfer state to networked application o HTTP verbs operate on the resource GET retrieves a resource’s state representation POST Updates resource (or other processing) DELETE deletes resource PUT Creates / updates resource state
  • 2008 IBM Corporation Page 12 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Background – HTTP-MQ Loosely Modelled on REST REST was originally described by Roy Fielding (principle author of HTTP spec) in his 1999 dissertation. http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm Has been described as the “Architecture of the WEB” – essentially a set of architectural best practices for highly scalable scalable distributerd applications. HTTP and the Web is the de-facto standard implementation of REST. It represents a Client-Server Protocol with a uniform interface across resources (HTTP Methods POST, GET, DELETE, etc). Resources are identified by URIs, connections are stateless etc. So for example, a REST interface for a online catalogue would have a separate URI for each item in the catalogue. Items could then be added by using a PUT, removed by using a DELETE and enquired by using a GET. REST therefore offers a different integration style to WS-* standards based web services. Qualities of service are sacrificed for simplicity and scalability to keep barriers-to-entry low. REST API’s are typically simple and so don’t require an SOA expert, or heavyweight tools to implement. They can be used spontaneously and incrementally – for example in Web 2.0 mash-ups. “Good Enough” is often Good Enough N O T E S
  • 2008 IBM Corporation Page 13 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore HTTP-MQ API: Verb / Resource Mapping HTTP verb mapping MQGET-MQPUT MQGET w. browse http://host/msg/queue/qname/ http://host/msg/topic/topic_path/ Messages DELETEPUTPOSTGETSample URIsResource Message header fields (MQMD) are conveyed in HTTP headers All HTTP headers prefixed with “x-msg-” Message Body is passed in HTTP entity body Message type is conveyed in HTTP Content-Type HTTP media-types are used : “text/plain” or “text/html” equate to MQ string messages (MQFMT_STRING) All other media types map to MQ Binary messages (MQFMT_NONE) No plan to supply client-side libraries – apps code direct to HTTP Defines an IRI to identify the address of the queue (or topic) API is a simple mapping of HTTP verbs to the MQI…
  • 2008 IBM Corporation Page 14 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore HTTP-MQ Verb / Resource Mapping The HTTP/MQ API is largely based on REST, though it has some quirks. For example MQ/HTTP transfers message representations, but messages are not ideal REST resources They do not necessarily have a unique identifier, and so cannot be addressed individually Not generally amenable to caching etc. because they must be delivered only once They are very transient The API defines a URI to address messages on queues and topics It is a stateless / connectionless API with one HTTP verb -> one MQ operation HTTP headers = Message headers request headers (get and put options) – wait, requires-headers entity headers (MQMD options) – priority, expiry, timestamp, persistence, msgId, correlId, replyTo, encoding HTTP request payload = Message body Message Body Passed as the HTTP entity body - can be text or binary The MQ message type is conveyed in the HTTP Content-Type header We don’t supply client libraries – applications code directly to HTTP verbs The API is not finalised !!! - e.g. some firewalls may block DELETE and extended HTTP headers, so In the future we may allow other resource types in the URI For instance ‘status’ resources to allow Queries on the status of MQ channels, ‘admin’ resources to allow administration alternatives may be required (possibly passing additional data as IRI query parameters) N O T E S
  • 2008 IBM Corporation Page 15 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deconstructed HTTP Flow - POST ( = MQPUT) Request: POST /msg/queue/requestQ/ HTTP/1.1 Host: www.mqhttpsample.com Content-Type: text/plain Content-Length: 60 x-msg-replyTo: /msg/queue/replyQ/ x-msg-requiresHeaders: msgID, priority, timestamp Message body which will appear on the queue as an MQSTR Response: HTTP/1.1 200 OK x-msg-msgID: 1234567890 x-msg-timestamp: Thu, 22 Mar 2007 08:49:37 GMT x-msg-priority: 4 Put to destination Type and length of message (60 char string) Headers to include on reply Message Data Response code Required Headers returned reply Queue
  • 2008 IBM Corporation Page 16 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deconstructed HTTP Flow - POST ( = MQPUT) The previous slide shows the HTTP flow on the wire for putting an MQ request message to a queue. Request The first line is the request line which identifies the HTTP verb (i.e. POST - mapped to an MQPUT) The request line is followed by the HTTP headers are passed as colon separated name-value pairs The Content-Type header = text/plain tells the MQ HTTP gateway to put the message to the queue using MQMD.Format = MQFMT_STRING. The Content-Length header tells the MQ HTTP gateway that the message is 60 bytes long “x-msg-replyTo” is an extension header which specifies the reply queue and queue manager to go in the MQMD.ReplyToQ and ReplyToQmgr fields “x-msg-requiresHeaders” is an extension header which specifies the header fields which the MQ HTTP header should return in the HTTP response. The HTTP headers are followed by the HTTP entity body which contains the message data Response The response to a POST shows whether the message was put successfully to the queue or not. The first line of the HTTP response is the status line – which includes the response code (in this case it is 200 - “OK”) The headers which follow the status line are the headers requested by “x-msg-requiresHeaders” i.e. msgID, priority, and timestamp N O T E S
  • 2008 IBM Corporation Page 17 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Deconstructed HTTP Flow - DELETE ( = MQGET) Request: DELETE /msg/queue/replyQ/ HTTP/1.1 Host: www.mqhttpsample.com x-msg-wait: 10 x-msg-correlID: 1234567890 x-msg-require-headers: correlID Response: HTTP/1.1 200 OK Content-Length: 60 Content-Type: text/plain x-msg-correlID: 1234567890 Message body which will appear on the queue as an MQSTR Get from destination Get wait time (ms) Headers to include on reply Response code Message length Message data type Retrieved Message Data Retrieved Message Correlation ID Get using this correlID
  • 2008 IBM Corporation Page 18 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore <head> <script language="javascript"> function doPOST() { xmlDoc = new XMLHttpRequest(); xmlDoc.onreadystatechange = processPOSTResponse; xmlDoc.open("POST", "http://localhost:50667/msg/queue/myQueue/", true); xmlDoc.setRequestHeader("Content-Length", 11); xmlDoc.setRequestHeader(" x-msg-requiresHeaders ", “msgID, timestamp “); xmlDoc.send("Hello World"); } function processPOSTResponse() { if ( xmlDoc.readyState == 4 ) { if (xmlDoc.status == 200) { window.status = "POST succeeded"; } else { window.status = "POST failed: " + xmlDoc.status; } } </script> </head> <body onload="doPOST() " > This page sent a message when it was loaded. </body> JavaScript XHR Example: Web Page to put a message
  • 2008 IBM Corporation Page 19 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Java Example: Method to PUT a message over HTTP public void doPOST() { try { // Set URI to point to queue URL postURL = new URL("http://localhost/msg/queue/myQueue/" ); // Open HTTP connection to URI HttpURLConnection connection = (HttpURLConnection)postURL.openConnection(); connection.setRequestMethod("POST"); // Set HTTP method connection.setDoOutput(true); // Send message PrintWriter out = new PrintWriter(connection.getOutputStream()); out.println("Hello from a Java app"); out.flush(); } catch (Exception e) { e.printStackTrace(); } }
  • 2008 IBM Corporation Page 20 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Current Status & Summary MA94 – Standalone listener – no JEE pre-requisites http://www-1.ibm.com/support/docview.wss?uid=swg24017593 Runs on Windows, Linux, AIX Ajax Samples, and Sample security exit MA0Y – Servlet-based HTTP Bridge http://www-1.ibm.com/support/docview.wss?uid=swg24016142 Windows SupportPac includes a bundled copy of WAS CE – Pre-configured with samples – Pre-configured with JCA Benefits No client libraries required – use stand-alone or in a Browser Simple API mapping HTTP to MQ – But highly extensible API Low cost of entry – develop in notepad, ad-hoc service descriptions / message formats Limitations As (un)reliable as HTTP No transactional capabilities Subset of MQI FEEDBACK WELCOME – TELL US YOUR REQUIREMENTS!
  • 2008 IBM Corporation Page 21 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Current Status The feature is delivered in the (unsupported) Category 2 SupportPacs mentioned Plan is to ship the Servlet-based listener as an integral (supported) part of the next WMQ release. Longer term we intend to add a tightly integrated HTTP listener and channels as a native part of WMQ Limitations REST is not good for everything – for instance event-based low latency systems do not work well in the REST model (a separate ‘server-push’ API over HTTP is under investigation to address that problem). REST is also silent on many of the quality of service standards typified by WS-* like reliable messaging, duplicate elimination, end-to-end security etc. (but this contributes to it’s simplicity so can be considered a plus point). N O T E S
  • 2008 IBM Corporation Page 22 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Summary - HTTP-MQ Interface Longer term, could become a new listener type integrated within the Qmgr - e.g. – No application server pre-requisite – HTTP clients connect direct to MQ – A new HTTP channel type and an HTTP listener – No access by default, secured by exits N O T E S
  • 2008 IBM Corporation Page 23 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WS-* - Standards Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 24 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SAM – Simple Asynchronous Messaging Languages like PHP simplify creating dynamic web pages They are not over engineered but good enough The languages and runtimes are largely open-source They are lightweight and require no tooling There have been no easy ways to get at our Messaging products from these scripting languages Aims of SAM: Develop a simplified messaging API for PHP Can be used uniformly across the WebSphere messaging family Deliver as an open-source extension
  • 2008 IBM Corporation Page 25 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Why PHP? “LAMP” is a very popular server-side environment (Linux, Apache, MySQL, PHP)
  • 2008 IBM Corporation Page 26 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore PHP PHP: Hypertext Processor From the php.net website… “PHP is a widely-used general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. “ <html> <head> <title>A real simple page with PHP</title> </head> <body> <h1>A section title</h1> This is some text with a PHP code embedded in it. Today is <?php echo date("F j, Y”); ?> </body> </html>
  • 2008 IBM Corporation Page 27 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Usage scenarios (1) Simple “send” Build message from user (HTML form / AJAX) Submit to back-end processes Transaction scope limited to single operation Uses: Brochure request, bank statement request, etc. Brochure request Messaging InfrastructurePHP script 1 SEND Webserver POST Web browser
  • 2008 IBM Corporation Page 28 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Usage scenarios (2) Simple “send” and “receive” Build a message from user input (HTML forms / AJAX) Submit to back-end processes Retrieve response from back-end Transaction scope may span more than one operation Uses: Insurance quote request, bank balance request, etc. Quote Form Messaging Infrastructure PHP script 1 SEND Webserver POST Web browser RECEIVE
  • 2008 IBM Corporation Page 29 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore API style – send and receive example // Create a connection to a broker $conn = new SAMConnection(); $conn->connect(SAM_WMQ, array (SAM_BROKER => ‘MyQueueManager’)); //Create a message $msg = new SAMMessage(‘Simple text message’); // set a reply queue identity in the message header $msg->header->SAM_REPLY_TO = ‘queue://receive/test’; //Send it and capture the correlation id that is returned… $correlid = $conn->send(‘queue://send/test’, $msg); // receive the answer but only wait for 5 seconds $answer = $conn->receive(‘queue://receive/test’, array(SAM_WAIT => 5000, SAM_CORRELID => $correlId) );
  • 2008 IBM Corporation Page 30 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Architecture overview web server Brokers MQ, WAS, MicroBroker, etc. XMS C api, client SAM Messaging extension PHP enginePHP scriptsHTTP This is the SAM deliverable
  • 2008 IBM Corporation Page 31 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Interested in taking a look? Downloads Available for PHP 4 and 5 Source available from PECL: http://pecl.php.net/package/SAM Binaries (limited binaries available for Windows from the SAM home page) Latest Version: 1.1.0 Released Feb 2007 Further info SAM Home Page: http://project-sam.awardspace.com Google Group: http://groups.google.co.uk/group/phpsam ….Feedback welcome
  • 2008 IBM Corporation Page 32 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WS-* - Standards Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 33 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WS-* - Standards Based Web Services WS-* web services can be used in conjunction with WebSphere MQ in two main ways 1. WebSphere MQ provides a first class reliable asynchronous transport for Web services Since WMQ v5.3 it has been possible to host Java and .NET services on WMQ for Windows and Linux It is possible to use WMQ as a ‘SOAP/JMS’ transport to access these services and to access services from CICS and WAS We are working on a prototype to integrate WMQ seamlessly into Windows Communication Foundation (Indigo) as a custom channel transport 2. Web Services Description techniques can be used to describe new and existing WMQ applications We are defining a Native MQ description specification which allows Native MQ applications to be described, as URI’s or WSDL This will allow WMQ applications to be catalogued in the registry …and allow tools to be created to simplify the invocation of WMQ applications N O T E S
  • 2008 IBM Corporation Page 34 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WMQ SOAP Transport Feature MQ is an ideal transport for WS-* Web services Offers enhanced Qualities of service when compared with Web services over HTTP, e.g.: Queuing offers more reliable scaling characteristics Better manageability – more tangible messages – Use MQ infrastructure, track messages, find bottlenecks etc. Asynchronous invocation – decoupled in time, multi-hop, reroute responses WMQ SOAP Transport enables use of WMQ for WS requests Made available in WMQ v5.3 as SupportPac MA0R - “MQ Transport for SOAP” Allows deployment of Java and .NET classes as SOAP Web services Uses WMQ as transport Uses typical WS deployment model - Client Proxy generation etc.
  • 2008 IBM Corporation Page 35 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Standards Based Web Services SOAP / MQ / JMS WMQ offers a number of advantages over HTTP as a web services transport The first advantage is in improved scalabilty for birectional data exchanges. HTTP Caches offer amazing scalability when data is static, but are no use for scaling unique req/resp. and HTTP infrastructures need to be designed to be able to handle the peak number of concurrent requests. Queues are inherently better at variable workloads. The second advantage is manageability – WMQ / JMS messages are more tangible and traceable than HTTP requests The third advantage is asynchrony – Limited (short term) async has existed in .NET for some time, and it has been added to Java JAX-WS. This does not offer any guidance on how to persist and correlate requests over a long time. True queued async adds resilience – (both parties don’t need to be running at the same time), and requests can be longer running than typical HTTP request. To allow exploitation of WMQ as a SOAP transport, WMQ has been offering SOAP/JMS support since v5.3 with SupportPac MA0R. See SupportPacs site for download… http://www-1.ibm.com/support/docview.wss?rs=171&q1=mA1J&uid=swg24006280&loc=en_US&cs=utf- 8&lang=en This SupportPac allows services to be created and deployed to MQ – so MQ can be used as a service hosting environment, but we expect most services in production to be hosted in other – more functional managed environments like CICS or WAS. The most important thing about this SupportPac is that it defined a standard way for carrying SOAP service invocations over MQ – thus ensuring interoperability between CICS, MQ, and WAS. Any message format could have been chosen, but JMS fits in best for compatibility with WAS. N O T E S
  • 2008 IBM Corporation Page 36 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP Transport for WMQ – Status SOAP Transport Feature available in WMQ V6 and V7 user SOAP WMQ Key to provider client app SOAP Layer (Axis / .NET) WMQ sender Typical Deployment of Web Service using HTTP WMQ SOAP listener SOAP Layer (Axis / .NET) target object WMQ Web Services Client (Java / .NET) Service (Java /.NET) client app SOAP HTTP SOAP HTTP server target object HTTP Deployment of Web Service using WMQ WMQ SOAP listener can host Java or .NET services in WMQ (Windows and Linux) Uses JMS message formats (RFH2 etc.) – Deployed services have a target uri of form ‘jms:queueName…’ – Options to control other WMQ details such as queue manager, persistence etc Interoperable with WAS and CICS can use WMQ as a SOAP/JMS transport
  • 2008 IBM Corporation Page 37 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WMQ and SOAP/JMS – Status SOAP/JMS shipped in WMQ v6 Scripts allow Java or .NET objects to be deployed as a SOAP/JMS web service over MQ Utilities generate WSDL, client proxies, Request & Response Queues etc. A WMQ Web service is identified by URI prefix “jms:” E.g. – jms:/queue?destination=myQ&connectionFactory=()&targetService=myService &initialContextFactory=com.ibm.mq.jms.Nojndi Client applications call SOAP/JMS services as if they are HTTP services - using generated proxy code One additional line of code is required to register the “jms” extension – Java : com.ibm.mq.soap.Register.extension (); – C# : IBM.WMQSOAP.Register.Extension(); – Visual Basic : IBM.WMQSOAP.Register.Extension() On the Server – a new MQ SOAP/Listener process waits for messages to arrive on the Q Hands off the request to the appropriate SOAP stack (.NET or Axis 1) The SOAP stack de-martials the SOAP, invokes the service, and queues the response Interoperates with WAS and CICS SOAP/JMS implementations allowing MQ services to be called from WAS, or CICS WAS or CICS services to be called using MQ as the transport N O T E S
  • 2008 IBM Corporation Page 38 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS – Vendor Interoperability Other vendors have also implemented SOAP/JMS Sonic, TIBCO, BEA, Axis Interoperability is required Vendor solutions can not interoperate on the wire (JMS is a standard API but not a standard wire format) …but should be able to interoperate at the API level – e.g. WAS -> WebLogic IBM JMS BEA JMS IBM WebSphere Application Server IBM SOAP Engine & Transport Handler JMS API WebSphere MQ JMS classes Service Consumer BEA WebLogic Server BEA SOAP Engine & Transport Handler Web Service JMS API WebSphere MQ JMS classes (Only *if* IBM JMS libraries at both ends, and IBM and BEA SOAP/JMS Transport Handlers agree on message formats etc.) E.g. Both expect a BytesMessage, etc.
  • 2008 IBM Corporation Page 39 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS – Vendor Interoperability A number of other vendors have also implemented SOAP/JMS in their Applications Servers, ESB’s, etc. Customers have heterogeneous environments For example using MQ as the JMS provider for BEA Weblogic Using a mixture of WAS and other application servers …and would like interoperability when invoking services using SOAP over JMS E.g. Consuming a service deployed in in WAS or CICS from a client running in BEA WebLogic Different vendor JMS solutions can not interoperate on the wire (because JMS is a standard API but not a standard wire format), but if Ideally it should be possible for different SOAP/JMS implementations to interoperate. Sonic, TIBCO, BEA, Axis …but should be able to interoperate at the API level – e.g. WAS -> WebLogic N O T E S
  • 2008 IBM Corporation Page 40 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS Standardisation A Standardisation effort is underway Initial participants: IBM, BEA, Sonic, and TIBCO In Oct ’06 the specification reached v1.0 RC1 – Five more vendors were invited to provide feedback (WebMethods and WSO2subsequently joined the core team) Specifications have been submitted to W3C and IETF See http://www.w3.org/2007/08/soap-jms-charter.html Standardisation will make it possible to have: JMS provider-neutral deployment of SOAP/JMS services Bidirectional HTTP - SOAP Bridging – Maintaining integrity of SOAP message and protocol metadata SOAP/JMS was one of the most requested features by participants at W3C Enterprise Web Services Workshop in Feb. 2007
  • 2008 IBM Corporation Page 41 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS Standardisation A SOAP/JMS standardisation effort between IBM, BEA, Sonic, and TIBCO has been underway for some time. In Oct ’06 the specifications reached v1.0 RC1 and were distributed to five more vendors and various customers for feedback. WebMethods subsequently joined the core team. http://mail-archives.apache.org/mod_mbox/ws-axis- dev/200701.mbox/%3c80A43FC052CE3949A327527DCD5D6B27020FB65C@MAIL01.bedford.progress.com% 3e Standardisation will make it possible to mix and match JMS providers and SOAP transport handlers, and will allow HTTP SOAP requests to be bridged to Message oriented middleware transports without losing metadata like SOAPAction There is growing interest in this standard – for instance at a recent W3C workshop (http://www.w3.org/2006/10/wos-ec-cfp.html) standardisation of SOAP/JMS was the feature most requested by the participants Standardisation will be a step towards WS transport independence – something which so far has been lacking in WS-* - an issue identified by Paul Downey in his - BT Position Paper “Services on the Web and Web Services” “Description of Transports In spite of Web services advocating transport independence, there are currently no well supported means to describe the WSDL endpoint for many commonly used paths such as IBM MQSeries or Sonic MQ, which are often abstracted using the Java Messaging Service (JMS). This is quite baffling, given how many of these transports are under the direct control of individual companies who advocate Web services.” http://www.w3.org/2007/01/wos-papers/bt N O T E S
  • 2008 IBM Corporation Page 42 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS Specification Details Based on IBM SOAP/JMS but some minor differences Consists of 2 specification documents 1. SOAP Binding Specification Message Format (BytesMessage) Properties that MUST / SHOULD flow in JMS messages SOAP/JMS WSDL bindings Request / Response & One-Way MEP State Machines Member submission Submitted to W3C - http://www.w3.org/Submission/2007/05/Comment 2. URI Scheme Specification – jms: “Internationalized Resource Identifier” (updated version of URI - RFC3987) e.g. jms:news?targetService=current-affairs &connectionFactory=SOAPJMSFactory &deliveryMode=2 &priority=8 replyToName=interested The scheme is under review on the IETF review list: http://www.ietf.org/internet-drafts/draft-merrick-jms-uri-01.txt SOAP/JMS enables interoperable access to services over MQ It isn’t restricted to Java (e.g. .NET support in MQ v6) Non-Java Applications can construct JMS messages to invoke services – MQ JMS message formats are all published, so e.g. COBOL apps can build JMS messages
  • 2008 IBM Corporation Page 43 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore SOAP/JMS Specification Details The SOAP/JMS standard is being delivered in 2 specifications. 1. The SOAP Binding Specification which defines how JMS messages should be constructed. This specifies the message type which should be used (BytesMessage), and the properties that MUST or SHOULD flow in JMS service invocation messages. It also specifies how to describe a SOAP/JMS service in WSDL bindings, and defines State Machines for the supported Message Exchange Patterns (Request / Response and One-Way). • Now submitted to W3C who have proposed a Working Group be chartered 2. The SOAP/JMS IRI Scheme defines the syntax of the jms: IRI (an “Internationalized Resource Identifier” is a address (URI) which can support internationalised character sets as defined in RFC3987) • Submitted to IETF URI review list: http://www1.ietf.org/mail-archive/web/uri-review/current/index.html Inevitably there are some slight differences between the IBM SOAP/JMS support implemented in WMQ v6, and the standardised version of SOAP/JMS. These are diffences in the IRI and the message formats Once the SOAP/JMS standard has been published, IBM products will need enhancements to implement the standard It’s important to note that, despite its name, the SOAP/JMS specification is not restricted to Java and JMS. It enables interoperable access to services over MQ from and to any languages Because MQ JMS message formats are all published, non-Java Applications can construct JMS messages to invoke services by building the correct RFH2 headers – e.g. COBOL apps building JMS messages, .NET support in v6 N O T E S
  • 2008 IBM Corporation Page 44 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WS-* - Standards Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 45 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WCF (Indigo) Custom channel for MQ Windows Communication Foundation underpins .NET Web services and Messaging Built-in Transports e.g. MSMQ, HTTP(S), Named Pipes, TCP/IP, etc. Transports can be extended with ‘custom channels’ Allows alternative transports (like MQ) to be slotted into WCF seamlessly IBM has built a prototype custom channel for MQ Released on AlphaWorks June 2007: – http://www.alphaworks.ibm.com/tech/mqwcf Using SOAP/JMS message formats – For interoperability with WAS, CICS SOAP/JMS services We are looking for feedback to assess demand for this feature Candidate for future SupportPac (with possible interim step of fee-based support) Long term goal is to ship in MQ as a supported programming interface (subject to demand)
  • 2008 IBM Corporation Page 46 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WCF Custom channel The work IBM is doing on using MQ for the Windows Communication Foundation is an example of how SOAP/JMS can be used outside Java The Windows Communication Foundation (aka Indigo) provides the core for all .NET connectivity – Available for Windows XP – An integral part of Vista Microsoft supply a number of built-in transports – MSMQ, HTTP(S), Named Pipes, TCP/IP, etc. – Can be used with .NET bindings to invoke .NET services – Or with SOAP bindings for interoperability with other services – The communication framework is extensible We have requirements to provide a WMQ transport for WCF Allowing seamless WMQ integration in Windows A prototype WCF custom channel has been released on AlphaWorks and will help gather requirements. This will allow .NET to .NET communication using WMQ as a Web services transport, and, because it uses the SOAP/JMS message formats and IRI, it will also allow WCF applications to communicate with SOAP/JMS services deployed to CICS, WAS etc. In the longer term we would like to ship this as a supported part of MQ (if there proves to be sufficient demand) N O T E S
  • 2008 IBM Corporation Page 47 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WCF Custom channel Features & Status Features : Can call a service using One-Way (Fire and forget), Request-Reply, and Callback MEPS SOAP/JMS or POX messages Download includes samples for: Calling Request-Response, and One-way WCF services Calling a sample Axis service hosted by WMQ v6 Calling a sample .NET service hosted by WMQ v6 Dependencies XMS .NET and WMQ .NET clients .NET Framework v3 runtime & SDK Limitations: No advanced features such as transactions or security Properties can only be specified in URI Updated March 2008 Adding Callback channel shape and fixing some concurrency issues
  • 2008 IBM Corporation Page 48 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Introduction Part 1: Web 2.0 - Easy access from HTTP and WWW WMQ/HTTP Gateway SupportPac – Client side access to WMQ from the Web and HTTP-capable languages “SAM” PHP API – Server side access to WMQ from PHP Part 2: WS-* - Standards Based Web Services SOAP and Standards Based Web services – WMQ and SOAP/JMS WMQ Channel for Windows Communication Foundation – Native WMQ interface for WCF WMQ Service Definitions – Describing WMQ assets in a Service Registry Summary Contents
  • 2008 IBM Corporation Page 49 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WMQ Service Descriptions Requirement for a standard to describe MQ apps as SOA assets Allowing stand-alone WMQ applications to be described as services: – To be inventoried, and catalogued in Service Registry – To be re-used as services in composite SOA applications – To be managed and traced with SOA tools IBM has specified the MQ Service Definition and SOAP binding IRI for MQ Queues and Topics (“wmq:”) WSDL bindings Defines applications properties, and where and how to send messages – E.g. the Message Exchange Pattern; Request queue; Response queue; Correlation style; Message format; Message persistence, priority etc. This is a published specification WMQ SupportPac MA93 - http://www-1.ibm.com/support/docview.wss?uid=swg24017518 It defines the standard to be implemented by IBM and other vendor tools
  • 2008 IBM Corporation Page 50 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore WMQ Service Descriptions MQ users have requested guidance from IBM on how they should describe their MQ applications as services for use in service oriented architectures. There has been particular interest in applying this to unmanaged native WMQ applications (i.e. those coded to the MQI - not JMS - and those running outside of an application server / CICS etc.) This will allow applications to: Be inventoried, and catalogued in Service Registry. For example, the WSDL description of an application can be stored in WSRR Be managed and traced with SOA tools. which - for example – will be able to monitor the queues associated with a service Be re-used in composite applications. For example, once the MQ service definition has been implemented by web services tools, it will be possible to drop an MQ application into a composite Web services application, and the tools will generate the code required to invoke the MQ application IBM has creating the MQ service definition specification to address this This consists of two documents. An IRI specification, which defines : – The address of a WMQ message destinations i.e. Queues or Topics for use by messaging applications – The address of other WMQ resources i.e. Qmgrs, Queues, channels, channel status etc. for use by admin tools A Bindings Specification, which defines : – Properties which may be used to describe and connect to a WMQ app. – The mapping of properties to message headers for the construction and interpretation of SOAP and non-SOAP messages – Supported message exchange patterns – A WSDL binding for SOAP/WMQ and non-SOAP/WMQ – Examples of IRIs, Messages, and WSDL documents This is published in SupportPac MA94 No immediate plans to add tooling support in the base MQ product, but it will provides a standard to be implemented by IBM and vendor tooling products Expect this to be picked up first by Message Broker and WebSphere Service Registry and Repository N O T E S
  • 2008 IBM Corporation Page 51 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Example – Native WMQ Request-Response MEP (WSDL) <definitions ... xmlns:wmqservice="http://www.ibm.com/2007/02/service/bindings/WMQ/"> <!–- Message and portType definitions are all standard WSDL --> <!– Binding - specifies persistence, message formats, and correlation style --> <binding name="mq_insurance_bindings" type=“tns:portType1"> <wmqservice:binding/> <wmqservice:persistence>MQPER_NON_PERSISTENT</wmqservice:persistence> <wmqservice:format>MQSTR</wmqservice:format> <wmqservice:reportOptions>MQRO_PASS_MSG_ID</wmqservice:reportOptions> <operation name=“getQuote"> <input> <wmqservice:body/> </input> <output> <wmqservice:body/> </output> </operation> </binding> <service name=“InsuranceServices"> <port name=“mq_ports" binding=“tns:mq_insurance_bindings"> <wmqservice:address location=" wmq:/msg/queue/INS.QUOTE.REQUEST"/> <wmqservice:replyTo>msg/queue/INS.QUOTE.REPLY</wmqservice:replyTo> </port> </service> </definitions> 2 1 3
  • 2008 IBM Corporation Page 52 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Example – Native WMQ Request-Response MEP (WSDL) This example shows a native MQ request/response application which processes an insurance quote request and place a response on a reply queue The service Interface (portType) and Message definitions are not shown in this example because the MQ Service Definition specification does not extend or change WSDL's intrinsic XML Schema-based type system. Note, this means that WMQ messages encoded in non-XML data – e.g. COBOL copybooks – must be described in WSDL using XML schema. A standard set of schema annotations and rules - called Data Format Description Language (DFDL) - is being developed to extend XML schema to non-XML formats and should be used once ratified, but until it is ratified, proprietary schema annotation can be used – (e.g. the ones used by WebSphere Message Broker). Looking at the WSDL in more detail: The service definition (1) contains the addresses of the request queue (wmq:/msg/queue/INS.QUOTE.REQUEST) and response queue (msg/queue/INS.QUOTE.REPLY) and defines the bindings to use (mq_insurance_bindings). The bindings section (2) is identified as an MQ transport binding by the presence of the <wmqservice:binding/> element. It specifies that the application must be accessed using the qualities of service defined by the elements shown in (3) – i.e. Persistent messages, with an MQ String format. It also specifies that the responding MQ Service Application correlates the response message to the request message by copying the request MsgId into the Response MsgId (MQRO_PASS_MSG_ID) The Application can either be described using the WSDL shown on the previous page, or alternatively it can be completely described using the following IRI. wmq:/msg/queue/INS.QUOTE.REQUEST? &replyTo=msg/queue/INS.QUOTE.REPLY &format=MQSTR &persistence=MQPER_NON_PERSISTENT &reportOptions=MQRO_PASS_MSG_ID The IRI definition is a more compact version of the WSDL definition, but they are equivalent. The presence of the replyTo parameter tells the invoking implementation to expect a response (In the IRI this MEP could also be indicated by a “msgType=MQMT_REQUEST” parameter, in which case the replyTo parameter could be omitted and the implementation would be expected to provide information about the replyTo destination – perhaps by using a temporary queue) The format and persistence parameters specify that the messages are non-persistent text messages. The reportOptions parameter specifies that the responding application should set the messageID of the response to the messageID of the request message (in addition to the default behaviour of copying the request messageId to the correlId of the response). N O T E S
  • 2008 IBM Corporation Page 53 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Summary – MQ, the Web, and Web Services MQ can be used as a Web 2.0 enabler Extending the reach of Enterprise applications to the Web MQ HTTP can be used for “Zero Footprint Clients” Server, Client , and Device applications can communicate over HTTP No need to deploy client libraries Extends MQ to many more client environments MQ can be used as a Web services transport Using SOAP over JMS standard, or by sending SOAP messages in raw MQ messages As a integrated custom channel in the Windows Communication Foundation WMQ based applications can be described and re-used in Service Oriented Architectures Describe WMQ applications in a standard way like other Web service assets using WSDL and IRI’s Store in Service Registry, and use tools to access
  • Elmer Senson (sensonel@ph.ibm.com) Original Slides by: Nancy Rosett (nrosett@us.ibm.com) Session F08: WebSphere Transformation Extender 8.2 Design Tools Lab
  • © 2008 IBM Corporation Page 1 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Objectives Overview of Websphere Transformation Extender 8.2 Setting up the WTX 8.2 environment Defining the data Creating a map
  • Objective 1: Overview of Websphere Transformation Extender 8.2
  • © 2008 IBM Corporation Page 3 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What is WebSphere Transformation Extender? WebSphere Transformation Extender (WTX)is a unique data integration technology for use in Application to Application and Business to Business enablement. What makes WTX different from all other types of integration products on the market is its ability to speak directly to native interfaces and APIs without the need for coding or re-tooling.
  • © 2008 IBM Corporation Page 4 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore What does WebSphere TX do? It’s a complex transformation engine that extends the value of WebSphere Enterprise It takes any kind of data from its native form Processes them together, natively, with no code And outputs them into their native target formats
  • © 2008 IBM Corporation Page 5 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Complex Integration Challenges Addressed by WTX Data Enhancement High Throughput Lookups Data Logic and Routing Data Validation Transaction Repair Dependent Data and Usage Rules Many to Many Transformation Single Pass of Data Sources – conversions and logic Mixed Data and Source/Target Types Dependent Result Sets, Nested Structure Dependencies Complex Data Transformation Nested, Semi-structured And Hierarchical Data Types Dependent Inputs And Outputs Type Conversions, Auto-Format Conversions Binary, Packed, EBCDIC, ASCII, Mixed Character Data
  • © 2008 IBM Corporation Page 6 WebSphere Technical Conference 2008 Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Successful Transactional Data Integration Requires: WebSphere Transformation Extender™™™™ Provides: • Quick-response, high- performance data integration • Event-driven, real-time data synchronization • Transformation of complex data from multiple sources to multiple targets • Automated transformation of complex, hierarchical nested data across many interfaces • Automatic routing of data based on complex business rules • Routing based on content anywhere in multiple data sources • Interoperability across operational data systems • Codeless synchronization of legacy, custom and off-the- shelf applications • Real-time screening of key transactions • Data content validation against business rules • New integration power without replacement of existing IT infrastructures • Loosely coupled components for rapid embedding (SOA) and seamless interoperability • Compli