Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

InteragencyCollaboration 1-27FINAL


Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

InteragencyCollaboration 1-27FINAL

  1. 1. Collaborative Application Development Among New York State Agencies Strategy for collaborative, cost-effective development of computer applications supporting government business processes January 11, 2008 State of New York Department of Health Department of Labor Department of Taxation and Finance Office of Children and Family Services Office of Temporary and Disability Assistance
  2. 2. Collaborative Application Development Among New York State Agencies Table of Contents Introduction ................................................................................................................................................................... 3 State OCIO Principles.................................................................................................................................................. 5 State Enterprise Architecture Principles.................................................................................................................. 10 Approaches Specific to Application Development....................................................................................................12 Open New York (OpenNY)......................................................................................................................................... 19 OpenNY Governance and Quality Control .............................................................................................................. 21 Service Oriented Architecture.................................................................................................................................... 28 Service Oriented Architecture – the Business Case..................................................................................................34 Service Oriented Architecture Formalism................................................................................................................ 37 Service Oriented Architecture Governance and Quality Control.......................................................................... 43 Conclusion....................................................................................................................................................................47 Appendix A - Open Standards Organizations .......................................................................................................... 49 Appendix B – NYS Enterprise Architecture Principles........................................................................................... 53 Appendix C – Manifesto for Agile Software Development...................................................................................... 69 2 / 70
  3. 3. Collaborative Application Development Among New York State Agencies A. Introduction In his 2007 State of the State message, Governor Eliot Spitzer called for changes that “...transform a government that is structurally oriented to resist change into one that is oriented to embrace it.” He went on to say that “we must work together to implement structural reform at every level of government to make it more flexible and adaptive to change.” For technology, change in government policies and procedures translates into responsiveness and agility: a requirement that technology-driven processes are aligned with the business of government. As the business of government changes to better support the needs of the people, so must the information technologies which support those business processes change to better align with governmental processes. A government of innovations—a government which embraces an innovation economy—requires technology which innovates. In that spirit, we offer an approach that provides for new and effective ways to support existing procedures as well as methods for collaborating on cost-effective solutions for the future, while still addressing the diversity of need in New York State communities. Each of the New York State agencies which comprise the study group for this white paper have a number of governmental business partners, organizations with which we conduct the public’s business. In addition to each other, these partners include local governments, other state governments, inter-state authorities, federal agencies, private not-for-profit corporations, employers, agents of employers, commercial interests, and of course other New York State agencies not in the study group. Moreover, our agencies have a wide variety of individual clients and customers, including individual taxpayers, unemployment insurance claimants, Medicare claimants, and citizens who live beneath the poverty line. We cannot serve all of these organizations and individuals through a single, one-size-fits-all, monolithic application. 1 Human-to-computer interfaces with governmental business processes (whether through an on-line tax filing mechanism, an interactive telephone claims system, caseworker-assisted data entry, or a business-to-business bulk data transfer) must be shaped to speak to individual communities of need. And of course the business processes that are internal to government agencies, and that are used to serve these communities, vary widely with governmental program designs: a “case management system” for mental 1 There are many different cateories of computer software. The distinctions are important in the context of this paper especially because different categories of software use development methodologies unique to them. Application software is a computer program (or suite of programs) that addresses some task that ordinary humans need to have done. It contrasts with system software (such as an operating system like Windows or UNIX), which provides underlying support for applications, but does not provide a direct service to users. Business application software refers to applications which support the flow of office work through an organization. Engineering application software is software supporting systems like the Space Shuttle. Office automation applications, like Microsoft Word™ or email systems, provide generalized utility to users. 3 / 70
  4. 4. Collaborative Application Development Among New York State Agencies health patients is nothing like a legal “case management system” such as that used at the Attorney General’s Office. Diversity in application architectures translates to choice and accommodation in how services are delivered through automated solutions. But over-diversification can sometimes become wasteful. Waste arises in a number of ways in information systems. One way is when individual technology units go off in different directions, adopting their own standards and reducing opportunities for aggregate acquisitions, code-sharing, and reuse of solutions. This is an especially large problem in New York State where governmental business processes grew up without any overarching strategy. But another way that waste arises is when aggressive consolidation—often as a reaction to too much diversity—ignores the complexity of government programs and individual business needs in favor of homogenization and standards for the sake of standards. Approaches to code consolidation can often be more costly than accommodating some diversity. 2 3 Happily, newer approaches—supported by some innovative technologies—can provide the needed flexibility in solution development, without waste. 2Natis, Yefim 2005 “Applied SOA: Conquering IT Complexity Through Software Architecture,” Gartner Research paper no. G00128127, May 25. 3 Strassmann, Paul A. 1995 The Politics of Information Management: Policy Guidelinesp. 57 ff. 4 / 70
  5. 5. Collaborative Application Development Among New York State Agencies B. State OCIO Principles At the behest of Melodie Mayberry-Stewart, the New York State CIO (State CIO), the agencies which are partnering in this study presented a series of “guiding principles” that can shape how State agencies work together going forward. These principles were articulated by the State CIO at the September 2007 Government Technology Conference in Albany and were presented on her behalf by William Travis, CIO at the Office of Children and Family Services, before a meeting of the Governor’s Economic Security Subcabinet in early October 2007. They were also presented by the State CIO herself at a meeting in Connecticut of state welfare agencies. In this section of the report we provide further clarification of these principles, defining terms with which non-technology managers might not be familiar. Throughout this document we provide additional clarification of these concepts. Provide for Interoperability using Open Standards and seamless data sharing through common enterprise platforms. Interoperability refers to the ability of applications to exchange data, although they might have been written in different computer languages, operate on different makes and models of computer, or use data internally that is organized in a unique way.4 Open Standards are those protocols and methods which are defined by such standards-setting bodies as OASIS, NIST, ISO, W3C, IEEE.5 By favoring Open Standards over closed systems, the State avoids lock-in, a situation in which an organization’s information assets become dependent upon a particular company’s proprietary approach. Such a situation leads to the inability of organizations to control costs since they have made themselves subject to the whim of companies whose technology cannot be easily replaced, except at great cost. Through use of standards that are openly defined in the industry all vendors have the opportunity to participate in contract opportunities. Moreover, products which adhere to Open Standards can be used together as components in a larger system, regardless of manufacturer. This latter quality is called Interoperability. 4 The ability to exchange data, although it might be represented differently in each application, is the hallmark of interoperability. Using the XML markup language, computer applications can interoperate, even though one might represent the names of clients as one 65-character text item, while another uses separate items of varying length to represent first name, middle name, last name, etc. Nonetheless, the disparity between data organizations adds effort. Later in this document, we propose a collaborative body—along the lines of international standards bodies—to arrive at standard data definitions and structures. 5 See Appendix A for a description of these organizations. 5 / 70
  6. 6. Collaborative Application Development Among New York State Agencies Seamless data sharing is the ability to send data between two applications without any special intervention. The classic counter example is transfer of data using a method where a printed copy of reports are generated out of one computer system, so that a clerk can manually enter data off the report into another computer system. In contrast, seamless sharing occurs through totally automatic transfers. Modern data transfers occur through a mechanism called “messaging.” A message is package of data that is sent over a network that includes information about security, routing, and priority. Such messages are typically encoded in a mark-up language called XML. In effect, XML is today’s lingua franca of data exchange. Common enterprise platforms means that the many computers (IBM, Sun, Unisys, Dell, Apple) and computer operating systems (AIX, Unix, Linux, Z/OS, Windows, OS X) which are components of New York State’s hardware infrastructure are to be considered a shared resource. Sharing can occur in a number of ways. One way is through use of the same computer by two agencies. Another form of sharing occurs when a computer operated by one agency provides a service for another agency. 6 Deploy an “Open New York” community approach to facilitate peer review and enhance quality control. Open New York is our branding of the Open Source model of collaborative development. While formal, joint development projects and contracts for individual development for sharing have their place from time to time, Open Source development does not require those formalities, yet can lead to cost-saving and very nimble responses to program requirements.7 8 Leverage prior IT investments with software reuse when feasible to achieve greater cost efficiencies. A fundamental principle of modern application development, leveraging prior investments, is often described with the phrase, “standing on the shoulders of those who came before us,” or more simply as “reuse.” With the development of the Object Oriented model of programming, reuse was effected, 6 Common platforms does not mean, however, that all applications must use the same make and model computer. Most technology managers would like that to be the case because it is easier to administer one kind of thing, and there is the presumption (not always borne out in empirical studies) that platform consolidation reduces cost. The reality, however, is that different computer platforms can have different capability and better service specific business needs. That, combined with the fact that shifting costs, competitive bidding requirements, and need to maintain vast libraries of legacy (i.e. historical) applications, draws us all to the conclusion that we have to be good at managing heterogeneity. 7Riehle, Dirk n.d. “The Economic Motivation of Open Source Software: Stakeholder Perspectives,” Computing Practices. SAP Research. 8 Gallagher, Scott, “Patterns of Open Innovation in Open Source Software,” in Henry Chesbrough, Wim Vanhaverbeke, and Joel West, eds., Open Innovation; Researching a New Paradigm. Oxford Press, Cambridge, p. 82 ff. 6 / 70
  7. 7. Collaborative Application Development Among New York State Agencies at least in principle, through the sharing of class objects: not so dissimilar from code libraries associated with earlier structured languages, like C. Today, however, reuse has a better chance of widespread realization through sharing of Services within a Service Oriented Architecture. Credite Suisse reports that 75 to 80 percent of required Services for any new application are already available in its repository of Services.9 A Service Oriented approach to reuse is superior to earlier attempts because it is not necessary for all participants in the sharing to use identical programming languages and platforms (something which is frequently undesirable). 10 As is described in more detail later, Service Oriented Architecture allows programs to interoperate, regardless of the development platforms on which they were built or in which they execute. And by making possible for the Services themselves (separately compiled, externally orchestratable programs) to be reconfigured into applications on the fly, the strictures of sharing programming code gives way to sharing what a business process is actually after. Implement agile systems development approaches to improve speed to market. Speed to Market. Such phrases as “speed to market,” “time to market,” and “delivery interval” speak to the requirement that IT deliver useful products (computer applications) to governmental programs on short order. Changes to business processes occur regularly and persistently throughout the history of any program. Regulations or procedural changes or outreach strategies in a program like Food Stamps can occur many, many times throughout a fiscal year. If IT cannot respond to such changes rapidly, with effective solutions, opportunity is lost and waste rises. Agile systems development.11 A modern method of development, called, “Agile, Evolutionary Development,” contrasts with an historical method called “Waterfall.” Waterfall method requires long periods for comprehensive requirements gathering, followed in sequence by a design phase, then a programming phase, then a testing phase. (There can be more phases than this in some approaches.) Waterfall works well with static systems, such as work processing programs like Word, or large engineering projects like the software which supports the Space Shuttle. It has been found largely ineffective in producing business applications. Agile development, on the other hand, divides application development work into short-term intervals (as short as two weeks) at the end of which some deliverable results. Further “iterations” of this result in adding more components. Along the 9Krafzig, Dirk, Karl Banke, and Dirk Slama. 2005 Enterprise SOA; Service-Oriented Architecture Best Practices. Prentice Hall, New York, p. 356. 10 It is generally acknowledged today that languages and platforms should be chosen to meet the need of the business objective. Domain specific languages (DSL) have been developed to address specific business problem areas, making reuse of lines of program code difficult if not impossible. 11 Larman, Craig 2004 Agile & Iterative Development; A Manager’s Guide. Addison Wesley, Boston. 7 / 70
  8. 8. Collaborative Application Development Among New York State Agencies way, component sub-assemblies are placed into production so that business units get early use, and so that changes to business plans can be accommodated immediately, rather than five years into a project. Establish strong enterprise governance to ensure alignment of technology plans with business goals. Governance is a detoxified term of art for authority and control. Governance can proceed from many quarters and via many methods. Where Information Technology is responsive to business needs, it flows first from the requirements that are articulated by policy makers and governmental program managers. The mechanics of governance assures that there is faithful adherence to contracts and service level agreements between executive leaders and technology managers on the one hand, and among technology managers themselves. Alignment is a quality of IT organizations as well as IT products such that IT activities conform to the needs of program areas, as opposed to either expecting programs to conform to computer products, or allowing IT to steer a course that is of interest to engineers and technical managers regardless of whether those interests support business requirements. Seek innovative collaborations to leverage State enterprise IT resources and assets. Innovation is the result of taking risks to move outside a generally accepted standard with the result of obtaining new efficiencies. If the solution is worth the risk, we declare a success and modify the standard to accommodate the approach as an accepted pattern 12. If the solution carried high tolls in cost, user dissatisfaction, and inefficiency, we have a failure and strengthen change controls to prevent experimentation along paths proven to end poorly. Codicils There are some supplementary considerations that further amplify these principles: COTS. One codicil is a caution against significant modifications to Commercial Off-the-Shelf (COTS) products. Use of COTS products is unavoidable; they provide pre-built components that can hasten the delivery of key functionality in application suites. Fortunately, many COTS products are based on Open Standards and easily interoperate. But all COTS applications also have a proprietary bent. When 12Pattern is a term of art in Information Technology. A pattern can be thought of as a formula for handling a certain set of recurring problems. Unlike a standard, which covers wide ranges of behavior and which are authoritative if adopted, patterns are specific to parochial problems and only suggest a possible solution. Think of a litigator who might have several pre-written paragraphs to insert into a brief if the case has certain characteristics. The paragraphs don’t work in every situation, and they frequently have to be tweaked for the particulars of a specific case. But without patterns, work would take longer and require that, for each instance of a particular problem, a solution would have to be re-discovered. 8 / 70
  9. 9. Collaborative Application Development Among New York State Agencies commercial applications are modified and extended, the risk is that they grow tentacles that lock them in to the enterprise architecture in deleterious ways. As COTS applications are modified to align as closely as possible with business needs, and when those business needs are far from the central functionality provided by the COTS application, we end up with a situation in which we no longer have the opportunity to substitute one COTS solution for another, as when monopolistic costs becomes an overarching problem. Data Standards, Data Directories, and Ontologies. The notion of Interoperability is focused on the ability of disparate applications to share data. To assure inter-operable applications can occur, New York’s data models must use a common enterprise data model. Turning again to Open Standards, organizations which set international data exchange standards (now largely an exercise in defining the schemas that comprise XML “documents”) have methods for arriving at such frameworks. Care is needed in expressing what is meant by data standardization, since there is a history in the industry of misusing standardization. As a State enterprise, we need to be wary of rigid data definitions, specifically in terms of standard sizes for data fields across agencies. Past attempts to reconcile one standard across every agency, and their external partners, stalled development and ultimately failed. A more practical alternative is a publish-and-subscribe paradigm (which is like the service registry idea today and what DTF did in OTC), where agencies can publish their data exchanges or services and participating agencies subscribe to them. This means the subscriber knows what they are getting (individual name as first, middle initial, last or as a string) and the subscriber can react - “let the buyer beware.” They don’t have to use the service, but if they want to then they can react. Otherwise the service publisher may have to change the service to meet some standard thus limiting productivity. Each consumer can do what he wants with the information returned. Collaboration. Additionally, New York State governmental entities should work toward operating an integrated enterprise team. To that end, procurements should be guided by a single, integrated governance structure which is aligned with the enterprise strategy and informed by individual and collective agency requirements. 9 / 70
  10. 10. Collaborative Application Development Among New York State Agencies C. State Enterprise Architecture Principles In 2004, the OCIO approved and published “Enterprise Architecture.” The principles in this document were subsequently tied to the Annual Technology Plan, an on-line database of IT projects maintained by the OCIO which individual agencies update with projects which they individually authorize, and for which they seek funding through the State budget process. In recording their projects, State agencies are obliged to explicitly show adherence to these principles. Appendix B contains the entire text of the New York State Government Enterprise Architecture principles. These principles are consistent with both the new OCIO principles above, as well as the specific objectives listed later in this document for Service Oriented Architecture and Interoperability, and agile development method. Though these principles arise from separate discussions and are aimed at addressing somewhat different problem spaces, they are stunning in their consistency. Here we cite those principles which deal especially with application development. 1. Maximize cost-effectiveness through shared use. 2. Reduce complexity. 3. Maximize flexibility. 4. Facilitate information sharing. 5. Facilitate data warehousing and data centric end-user solutions 6. Promote enterprise-wide data standardization 7. Facilitate common solutions for business processes shared by agencies. 8. Facilitate code re-use and smooth integration of business processes. 9. Business process re-engineering will be considered when defining requirements. 10. Achieve a standard look-and-feel to GUI design. 11. Define standard office automation suites 12. Employ thin client solutions where appropriate. 13. Promote interoperability and integration across all applications 14. Favor applications which are highly partitioned, modular in design, with components that are maximally decoupled and which use standards-based messaging protocols internally and externally. 10 / 70
  11. 11. Collaborative Application Development Among New York State Agencies 15. Achieve a uniform look-and-feel 16. Establish a finite set of reproducible configurations. 17. Assure scalability. 18. NYeNET will be the statewide backbone for enterprise applications. 19. Enterprise systems will adhere to all applicable security, confidentiality, and privacy strictures. 20. A single method of user authentication will be used across all applications. 21. Application-level security and data retention requirements will be maintained. 22. There will be a formal governance process for planning and managing the enterprise. 23. There will be a formal Enterprise Architecture review process. 24. Disaster recovery and business continuity plans will be in place. 25. New technological advances will be continuously reviewed. 26. Relevant business area experts will participate in application development. 27. All acquisition decisions will be made with a view of their impact on the overall enterprise. 28. A Total Cost of Ownership metric will be used in examining the cost of information technology. 29. Applications will be delivered in phases (“chunked up”) so as to prove interim deliverables for users. 30. Industry standard software engineering practices will be used in application development. 31. To the fullest extend possible, acquired applications and infrastructure will be commercially viable, proven solutions. 32. Products and solutions should use open standards to facilitate inter-operability. 11 / 70
  12. 12. Collaborative Application Development Among New York State Agencies D. Approaches Specific to Application Development The sets of principles describes above speak to the management of the entire computing enterprise. To examine these principles in light of the application development efforts that most concern us, we drill down further to explore specific approaches to application design. Software development is a complex undertaking that requires significant expertise and skill. It also requires active participation by all those involved in the process. 13 Portability Versus Interoperability Throughout the 1990’s, a discussion raged especially between Sun Java adherents on the one side and promoters of the Windows-only variant of Java, J#, on the other. Sun argued that the model for software development going forward should be based on Portability: code—written in Java—should be able to run anywhere, regardless of the platform. Microsoft countered that the better paradigm is Interoperability: software can be written for any platform, as long as it can make remote calls to programs that are resident on other platforms. To demonstrate its position, Microsoft introduced the Simple Object Application Protocol (SOAP), an open exchange pattern which can be used to send corresponding messages between applications regardless of the platform or environment in which they reside.14 Both stands have been cooperating winners. While Java has become the leading programming language and run-time environment in the new millennium, SOAP (which now only stands for SOAP) has become the quintessential messaging service for Web Services that reside within a Service Oriented Architecture (SOA). Interoperability is about the federation of resources. Portability is about the consolidation of resources. While consolidation—the reduction of products, services, and methods to a limited number of brands, models, and instances—is sometimes an effective way to achieve cost-savings, it can also limit innovation and increase risk through the cost of consolidation itself. (Re-training a workforce to use only one code base or language or hardware platform, for example, tosses away one’s knowledge investment in the hopes of distant efficiency brought about through homogenization.) Federation, on the other hand, acknowledges that one size cannot fit all, but seeks to strike a balance between central standards and unbridled choice. Interoperability permits that balance to take place. 13 Ambler, Scott W. 2002 Agile modeling; Effective Practices for eXtreme Programming and the Unified Process. John Wiley and sons, New York, p. 126 ff. 14 Seely, Scott and Kent Sharkey 2001 SOAP: Cross Platform Web Services Development Using XML. Pearson Education, New York. 12 / 70
  13. 13. Collaborative Application Development Among New York State Agencies Portability and the Benefits of Open Development There are instances where the same code fragment, compiled program, or program suite (application) can be profitably used on separate platforms. For example, to achieve a certain level of performance on a distributed network, it is sometimes useful to run an instance of the same program at two locations. In other cases, it might be useful to share a consolidated code base with other organizations simply as a cost- saving measure. Why write a program twice when you can re-use what already exists? Despite this fair weather principle, it is often difficult to develop code that exactly meets the need of two organizations. A municipality using a common code base might need to extend that code to meet local requirements, while still wanting to benefit from continuing enhancements made to the core. Organizations working closely together on common code might have to include far more functionality in the shared code to address the separate needs of both organizations. Use of shared code must be elective, because only individual organizations can judge whether code it might share fits its local business needs. However, collaborative methodology exist which provide choice while still providing a high degree of sharing and collaboration. The Open Source development method provides such an approach. Interoperability and the Benefits of SOA In his celebrated book on Service Oriented design, Thomas Erl asserts that it is important to understand why both vendors and users within the IT industry are taking pains to adopt Service Oriented Architecture. He lists the following strategic goals and benefits that result from Service Oriented Architecture:15 • Increased intrinsic interoperability - Efficiencies are gained today through the sharing of data by organizational units and the applications which support them. Applications which are not interoperable must be integrated at significant cost. Service Orientation makes interoperability a native (intrinsic) part of applications so that consideration of integration disappears as a special requirement. 15 Erl, Thomas 2007 SOA; Principles of Service Design. Prentice-Hall, New York, p. 55 ff. 13 / 70
  14. 14. Collaborative Application Development Among New York State Agencies • Increased federation - Federation unified information resources while allowing their individual autonomy and self-governance. SOA fosters creation of standardized and externally orchestratable (composable) services each of which encapsulates a piece of the enterprise.16 • Increased vendor diversification options - Diversification allows selection of best-of-breed products and technology innovations. SOA allows a diverse portfolio to be used together. This can be done cost-effectively if they are interoperable, thus avoiding integration costs (if integration is at all possible). • Increased business and technology domain alignment - Interoperability allows for re-use of services in different orchestrations (applications). This also allows for rapid response to changes in business rules and regulations because by changing a single service, all those which depend upon it automatically benefit from that change. As a result, applications maintain their alignment in a more responsive fashion. • Increased Return On Investment (ROI) - A key metric in assessing how cost effective a software development undertaking might be is ROI. • Reduced IT burden • Increased organizational agility The diagram on the following page shows how Labor interoperates with other governmental and private agencies. We show that while Labor might develop Services for internal consumption (i.e. for assembly into composite applications that it uses within its own agency), these same services are available to other agencies for re-use in the composite applications serving their business ends. A banking service developed for the Trade Act Assistance program in Labor can be consumed by other agencies which have programs that need banking services, without the need for each to write their own private interface to banking institutions. 16 Self-governance means that individual development communities (e.g. IT units in various state agencies) may work in a quasi-independent fashion, but within the overall governance framework of the enterprise. During early discussions of the OASIS standards committee on the SOA Reference Model, a question was raised about whether it was the responsibility of the orchestrator to have enough reach to compose services, or whether services must be written so that they are composable. [See:, Wed 25 May 2005 20:29:20 -0400.] The conclusion reached was that services must be written (according to a standard) to be composable. That assures, for example, that one agency can build an application using services developed in another agency. In effect, services must conform to an overall community standard to be composable. 14 / 70
  15. 15. Collaborative Application Development Among New York State Agencies Figure 1: Labor-centric view of how the Services and Portals it contributes are used to achieve interoperability. 15 / 70
  16. 16. Collaborative Application Development Among New York State Agencies Waterfall Development Style versus an Agile, Evolutionary Style Old views of software engineering persist. Here’s what we thought. Software is architected in pretty much the same way houses are architected. First you go out and collect requirements, finding out what the customer wants to have built—maybe a compact house for heat efficiency, but with a large, comfortable master bedroom and two smaller bedrooms for the kids, with a second bathroom shared by the two. A living room with a view would be nice, and a “rec room” that the family can tear apart without worrying about visitors. With the requirements in hand, the architect (and his engineers) goes about creating a design. In practice, the functional choices that the buyer has to make are quite restricted. There are only a few kinds of rooms that can be built. And rooms are such that a bedroom can generally be turned into a study or solarium without much ado once the kids leave. What’s left after those few choices are aesthetic issues, what software developers see as the look-and-feel of an application. Business processes are not like residential dwellings. Not only are processes complex, with many unique elements, they are a moving target. Homes, too, sometime fail to address the need of growing families. Homes that no longer fit a family’s needs can be enlarged or remodeled to accommodate changing requirements. But there comes a point where it is better to trade up (sell the old house for a new one). In software development terms, trading up is re-writing an application, throwing out the old and replacing it with a new, improved version. That is a costly approach for an IT project because business requirements change vigorously, and the length of time to design a new solution is quickly outpaced by shifting requirements. By the time the new application is built, it needs to be reconstructed. The sequential approach to software development—where analysis must be complete before design begins, and design must be complete before implementation can start—is called the Waterfall Method. The Waterfall Method is an engineering approach, akin to what an architect does is drawing plans for a house, to be handed off later to a construction company. Three generations of application developers have learned this method as the canonical approach to programming. Waterfall does, in fact, work quite well where static products—like homes—are being engineered. Development of the code associated with a space shuttle, or an office automation product like Microsoft Word™, yield well to Waterfall. But for most business processes, agile and evolutionary development is a better path. Agile software development evolved in the mid 1990s as part of a reaction against “heavyweight” methods. Agile methods are a family of development processes, not a single approach to software development. In 2001, 17 prominent figures in the field of agile development (then called “light-weight methodologies”) met in Utah to discuss ways of creating software in a lighter, faster, more people-centric 16 / 70
  17. 17. Collaborative Application Development Among New York State Agencies way. They created the Agile Manifesto, widely regarded as the canonical definition of agile development, and accompanying agile principles. Some of the principles behind the Agile Manifesto are: • Customer satisfaction by rapid, continuous delivery of useful software • Working software is delivered frequently (weeks rather than months) • Working software is the principal measure of progress • Even late changes in requirements are welcomed • Close, daily cooperation between business people and developers • Face-to-face conversation is the best form of communication • Projects are built around motivated individuals, who should be trusted • Continuous attention to technical excellence and good design • Simplicity • Self-organizing teams • Regular adaptation to changing circumstances The publishing of the manifesto spawned a movement in the software industry known as agile software development. 17 Portability and Collaboration Although Sun Microsystems’ Portability stand has been eclipsed in recent times by SOA and Interoperability, Portability remains an important pattern for software development. Looking at the Labor Department’s partners in other state governments, it is unlikely that a service operated by the State of New York would be consumed by an Unemployment Insurance Taxation application in the State of California. Bandwidth issues, at least at this point in time, would easily confound that effort. But 17 Appendix C contains the full Agile Manifesto. 17 / 70
  18. 18. Collaborative Application Development Among New York State Agencies California might well want to replicate software developed by New York, and execute on its own enterprise platform. When software exchange is made within a collaborative environment, there is an added benefit—noted by Sun—of mutual development. The Open Source movement is best known environment for such exchanges, though there is no requirement that such exchanges only occur within a restricted programming environment such as J2EE. On the contrary, as long as the base requirements for shared software are defined, there are no restrictions on collaboration. This is handy because it means that source code is available to anyone for possible rework for another platform. Hence, many Open Source projects have multi-platform instances of its products. To facilitate both the exchange of software and collaborative development, Labor has established the LaborForge. LaborForge is an Open Source repository for the exchange of code and discussion with other state workforce agencies. Located at, the repository is open to other states, other New York State agencies, and the global development community at large. 18 / 70
  19. 19. Collaborative Application Development Among New York State Agencies E. Open New York (OpenNY) Collaboration comes in many forms. One that has untold success is the collaborative methodology associated with the Open Source software movement. Many discussions of Open Source focus on use of products that result from Open Source collaboration, especially the Linux operating system and the Apache web server. But Open Source is a larger concept. Open Source development is a method of free and open work on projects that groups and individuals find of interest. Open New York gives a face to New York State government’s emerging efforts to share in the development of software components. Components can be as small as code fragments (such as an implementation of an algorithm to calculate least squares that might be reused in any number of statistical programs), or as large as whole applications (like a statewide budgeting system). Open Source development style is largely attributed to Linus Torvald, the individual behind the free Linux operating system effort. Torvald’s method calls for early and frequent release of deliverables, delegation of as much as possible to other partners, and “openness to the point of promiscuity.”18 Unilateral Open Source Arrangements The Office for Disability and Temporary Assistance has begun to share the code base for a Food Stamp application with the State of Wisconsin. This is an example of a unilateral, Open Source arrangement. Wisconsin, the primary developer of the application, has thrown access to the source code open to the State of New York for use as New York sees fit. However, there is a condition. New York and Wisconsin agree to collaborate on future development of the application on an as-needed basis. This means that while the states are not bound to a joint schedule, and while there is no dependency of work done by one state on the other, each agrees to make any future development available to the other party. Government Open Source Repositories Open Source arrangements can be wider than one between two parties. Indeed, thousands of collaborative repositories exist to bring developers together to build joint solutions to common business problems. The well-known SourceForge repository (at boasts over 165,000 projects and over 1 ¾ million participants. Although such open repositories have had wide success, there has been only small interest in shared development of applications for government. 18 Raymond, Eric S. 2001 The Cathedral & the Bazaar. O’Reilly and Associates, Inc., Sebastopol, California. p. 21. 19 / 70
  20. 20. Collaborative Application Development Among New York State Agencies The New York State Labor Department has established one such repository at The LaborForge site is the center of a cooperative agreement with workforce agencies in other states. The repository is promoted by the U.S. Labor Department’s Interstate Development Group at the University of Maryland and supported by federal grants. Open Source License Agreements Open Source collaboration methodology is a natural approach for within State government. But wider audiences of interest—in other governments as well as among private interests—have potential for yielding better products at a faster pace. Hence, methods are needed for encouraging wide participation beyond New York State agencies. One barrier to open source collaboration which government needs to address is redistribution authorization or, simply, the open source license. Products and materials acquired with public funds do not enter the public domain. Government retains ownership of the snow plows it purchases for the general good; citizens have no public domain right to borrow a plow for their private driveways. The same is true of the products which government develops. Generally, government produces documents and provides services. The availability of government services are controlled by rules and regulations, while the availability of documents fall under freedom of information and records management laws. But government also manufactures property with marketable value, and computer software is arguably the largest portable asset it creates. The federal government promotes free and open distribution of such products through regulations which seek to stimulate economic development for American business. In contrast, state and local governments are largely silent on what distributions might be available. At best, all property is taken together as assets which are to be protected under public finance and public officers laws. The State’s copyright interest in what it produces is clear. But to effectively participate in an Open Source community beyond the State’s own walls, New York must establish so-called copy-left provisions that grants to any collaborator a license to re-use, modify, and reproduce any government work or its derivatives. Copy-left license provisions provide these usage rights to users in exchange for a requirement that any derivative work which that user might make will also be distributed under similar terms.19 Such licenses are called “viral” in that acceptance of (contact with) the initial license causes the propagation of the same terms in subsequent licenses. 19 Dixon, Rod 2004. Open Source Software Law. Artech House Publishers, Norwood, Massachusetts. p. 22 ff. 20 / 70
  21. 21. Collaborative Application Development Among New York State Agencies F. OpenNY Governance and Quality Control Open source software allows users access to a body of source codes which they can technically modify to fix defects or to add new functionalities to meet business needs. In an effort to gain broad acceptance and maintain long-term sustainability, we must coordinate these changes through a formal governing body which should at least provide • Open access to everybody with the right skills to contribute • Meritocracy • Standards • Transparent review and approval processes • Quality control These are critical success factors for an open source community that leverages the power of collective intelligence of its members and provides a high-level of confidence for the use of its software. For instance, Apache HTTP server is the most commonly used web server in the world, as of December 2007, it had about 50% of the market share in a population of about 155 millions sites, according to the statistics collected by Netcraft. Since its inception in 1996, Apache has been developed and maintained by an open community of developers under the auspices of the Apache Software Foundation which consists of a formal governance structure with well-defined roles and responsibilities. For OpenNY, we recommend a very similar, but simplified, approach adopted from OpenSolaris to establish the governance body. In what follows, we describe its structure, roles and responsibilities. OpenNY Governance Structure As a function of the enterprise, the OpenNY Community is structured as an organization of agency participants in which Members are given the right to vote on Community-wide decisions, the most significant of which is to elect an OpenNY Governing Board (OGB) to be responsible for overall day-to- day operations and representation of the organization to third parties. The OGB, in turn, delegates the organization and decision-making for specific OpenNY activities, such as new features development and “bug” fixes, through the creation of Community Groups. Each Community Group consists of participants and contributors, a subset of whom become long-term Core Contributors and are given the responsibility 21 / 70
  22. 22. Collaborative Application Development Among New York State Agencies for governance within the Community Group. Finally, the set of all individuals that have been named by one or more Community Groups as Core Contributors are the Members who are given the right to vote on Community-wide decisions. A schematic of the structure is depicted below. OpenNY Governance Structure OpenNY is a free and openly available resource, yet imposes controls aimed at assuring seriousness of purpose of those who join the community ranks. Although anyone who has access to NYeNET may view the development discussions of the OpenNY Community and use the products of the OpenNY community in accordance with each product's associated licenses, participation in development or decision-making within the OpenNY Community is restricted to persons registered with the OpenNY Community. Registration shall be available to any Figure 2: Participating groups in an Open Source governance person who has not been removed by structure. previous action of the Community or forbidden by applicable law or regulation. Roles Various terms are used to describe the people who are involved in the OpenNY Community efforts, based on their recognized contributions, length of commitment, and current activity. The OpenNY Community recognizes three levels of involvement by registered persons: 1. Participant. Any registered person who participates in the OpenNY Community, either through general discussion areas or within one or more Community Group efforts, shall be termed an OpenNY Participant. 2. Contributor. A participant who has been acknowledged by one or more Community Groups as having substantively contributed toward accomplishing the tasks of that Community Group, or 22 / 70
  23. 23. Collaborative Application Development Among New York State Agencies by the OGB for at-large contributions, shall be termed an OpenNY Contributor. Such designation is permanent and persists regardless of the person's current level of activity or status within the Community. A Contributor may request that their status not be published or published only in the form of a pseudonym that is unique within the Community. 3. Core Contributor. A Contributor who is an active and sustained contributor to any Community Group and accepts designation as such by said Group shall be termed a Core Contributor for said Group and granted the status of Member for the OpenNY Community as a whole. Disputes Disputes among participants arising from their participation in any Community Group effort shall be resolved by said Group according to its normal decision-making procedures. If such a dispute falls outside the scope of any existing Community Group, or extends to the OpenNY Community at-large, then any of the participants may appeal to the OGB, or to one or more committees designated by the OGB, to serve as an arbitrator in order to resolve the dispute. Suspension and Expulsion of Participants The OGB is responsible for ensuring that participation in the OpenNY Community is constructive in nature and not harmful to other participants. When abuse of the Community is reported to the OGB, the OGB shall review a participant's actions according to the following procedure: 1. The OGB shall notify the participant via electronic mail that his or her participation in the OpenNY Community is under review and that the review will be completed within thirty (30) days. The review shall occur in a closed session and shall not be made public unless an action is made to suspend the participant. 2. The OGB may order a temporary removal of the participant's write access to the OpenNY Community infrastructure until the review is complete. 3. Upon conclusion of the review, the OGB may, upon affirmative vote by two-thirds (2/3) of the OGB in office, suspend the participant's write access to the OpenNY Community infrastructure for a period of up to six (6) months and, if said participant has previously been subjected to such a suspension, recommend expulsion of the participant from the OpenNY Community . 4. A suspended Member may attend and vote at a meeting of the Members, in person or by proxy, subject to the normal rules of conduct for that meeting. 23 / 70
  24. 24. Collaborative Application Development Among New York State Agencies A Participant under suspension may be expelled from the OpenNY Community by an affirmative vote of a two-thirds majority of the Members of record. The effect of such a removal shall be the immediate expiration of any grants of Core Contributor status to that Participant and removal of the Participant's write access to the OpenNY Community infrastructure until such time as a subsequent act of the Members revokes the expulsion. Membership The OpenNY Community is a membership-based organization in which active long-term contributors are given the authority to participate in decisions through voting and consensus. These Members are responsible for ratifying Constitution, electing an OGB to oversee the day-to-day operations of the OpenNY Community, and providing guidance to the OGB. The initial set of Members for the OpenNY Community shall be those natural persons designated by the existing OpenNY projects as Core Contributors to their projects. The initial set is intended to be a representative sample of the Community's existing core contributors and need not include every person that might fit that level of involvement. The initial set shall be admitted upon the affirmative vote of the OGB at the initial meeting of the OGB. Thereafter, persons shall be admitted as Members of the OpenNY Community upon accepting the designation of Core Contributor for one or more OpenNY Groups and receipt by the Secretary of their willingness to accept the status of Member. Each grant of Core Contributor status, including the grants that form the initial set of Members, shall have a duration of two (2) years and is renewable by the granting Community Group. A Member may be recognized as a Core Contributor by multiple Community Groups, thereby qualifying for Member status multiple times, and each grant is independent from any others. Each such person counts as only one (1) Member regardless of the number of grants upon which their Member status is based. A Member's membership shall be terminated upon the expiration of all prior grants of Core Contributor status or upon his or her earlier resignation, removal, or death. A Member who is expelled from the OpenNY Community shall also be removed from the Membership and shall not be eligible for subsequent grants of Member status until such time as a subsequent act of the Members revokes the expulsion. Community Group Voting Procedures Purpose. Most of a Community Group's work is done by individual Contributors within a climate of ongoing discussion, informal group consensus, and a general desire to meet the community's shared 24 / 70
  25. 25. Collaborative Application Development Among New York State Agencies objectives. However, occasional conflict is an inevitable part of working together to solve complex problems. The Community Group voting procedures are designed to support both friendly collaboration and individual innovation, while at the same time providing adequate oversight for the OpenNY Community as a whole by enabling equal participation in formal decisions regardless of a person's affiliation or location. The Community Group voting procedures determine how future conflicts will be resolved so that volunteers on a project need not fear differences of opinion: contrary ideas can be voiced without derailing progress. The objective is to avoid unnecessary conflict over changes, while at the same time encouraging opinions that work towards producing quality products in a timely manner. Location. Decisions by a Community Group are made by vote or consensus on a public meeting place of the Community Group that is most applicable to the action being discussed. For example, Community Group decisions that are specific to a single project are made through discussion and voting/consensus on the asynchronous meeting place (e.g., mailing list) for that project, whereas general Community Group decisions are made on the Community Group's general discussion place. Some discussions, such as investigation of security issues prior to their publication and nominations for Core Contributor status, are expected to take place in private prior to any public vote by the Community Group. Votes. Each Core Contributor of a Community Group has the right to one binding vote on each proposed action of the Community Group. Other Participants may express their opinions through non-binding votes. A Core Contributor may change their own vote on a given action until such time as the action is no longer applicable (e.g., decisions regarding a given product release are final once the release is made public). Votes are expressed in the range [-1,1] with the following meanings: Yes, "Agree," or "the action should be performed." A +1 vote indicates that the voter is in favor of +1 implementation of the proposed action. Abstain, "no opinion," or "I am happy to let the other group members decide this action." A vote in the interval between 0 and 1 (such as +0.5) can describe varying degrees of support for the ±0 proposal but still counts as an abstention. A vote in the interval between 0 and -1 (such as -0.9) can describe varying degrees of disagreement with the proposal but still counts as an abstention. No, "Disagree," or "the action should not be performed." On actions where consensus is required, -1 this vote counts as a veto if it includes an explanation of why the veto is appropriate and at least one other Core Contributor agrees that the explanation is valid ( regardless of their own vote). 25 / 70
  26. 26. Collaborative Application Development Among New York State Agencies Voting Systems. Decisions of a Community Group are determined by several related voting systems, chosen per action type based on the degree of consensus or quorum needed to make the decision: Voting Approved by Action Types System Putbacks (when subjected to vote). Initiate At least three (3) binding +1 votes and new project. Terminate project. Addition Consensus no -1 votes (i.e., unanimous with a of Core Contributor. Removal of Core minimal quorum of three votes). Contributor (subject must abstain). At least three (3) binding +1 votes and Design choice. Release plan. Product more +1 votes than -1 votes (i.e., Majority majority vote with a minimal quorum of release. Facilitator nomination. Documentation (when subjected to vote). three +1 votes). Approval is assumed unless a -1 vote is received, after which the action is Assumed reverted or the issue is decided by Putbacks. Documentation. Majority or Consensus vote, depending on the type of action. Voting Period. Actions requiring a vote shall have a voting period of no less than seventy-two (72) hours. Actions are actionable before the voting period expires if all Core Contributors have voted and the issue would be approved by those votes. Appeal. Once a decision is made, an action can only be revisited (brought to another vote) if at least one voter on the prevailing side of that decision declares a desire to change their vote and proposes a new vote on the action. There is no mechanism for escalation of decisions legitimately made according to these Community Group voting procedures. If it is believed that these procedures were not followed and, after being informed, the Community Group refuses to revisit the decision, then a review of the Community Group's procedures may be requested of the OGB. Upon review, the OGB may act to terminate the Community Group, partition the Community Group into multiple Community Groups, or provide advice to the Community Group; the OGB shall not make decisions for the Community Group. OpenNY Quality Control Reliability and auditability are some of the benefits for open source. As the source code is openly published and widely distributed, defects can be detected and fixed more quickly than a commercial product can. The National Alliance for Medical Image Computing sponsored by National Institute of Health is an excellent example of how an open source community can develop mission critical tools for common good. It facilitates parallel development and testing of software projects in multiple sites in 26 / 70
  27. 27. Collaborative Application Development Among New York State Agencies multiple configurations (hardware, operating systems, compilers, etc.). Results from a build/test sequence are transmitted to a central server using standard internet protocols. The server produces concise dashboards, summarizing the current state of a software system. The dashboards link to detailed reports on inter- and intra-configuration results. Testing results are tracked over time, allowing developers to trace the history of development. The Open Source Visual Toolkit (VTK) quality dashboard is a good example of such a quality dashboard and a snap shot of it is shown in the figure below:20 Figure 3: Visual Toolkit quality dashboard A new build is conducted nightly and tested against a set of benchmark cases. Defects are detected immediately and posted with sufficient information so that community members can take the appropriate action in a timely fashion. It has been reported some defects were fixed within several hours. We recommend OpenNY to follow a similar approach. 20 On on-line example of the VTK can be found at 27 / 70
  28. 28. Collaborative Application Development Among New York State Agencies G. Service Oriented Architecture A second, modern approach to collaboration arises from emerging technologies that permit composite applications to be built from components developed and operated by different organizations. Service Oriented Architecture is the technology that does this and Services are the “bite sized” pieces of software that can be mixed and matched to build new and innovative works at reduced cost. A Service is a small program that receives requests from other programs, and performs some desirable task on their behalf. Consider, for example, a Direct Deposit Banking Service. Such a Service might receive requests from an Unemployment Insurance Payment program, a Medicaid Payment program, a STAR Tax Rebate Payment program, or a Welfare Payment program. Each of these programsServices themselvesmight send a message to the Banking Service containing the name of the recipient, the recipient’s bank and account information, a payment amount, the New York State fund from which to withdraw monies for payment, and perhaps other data. The Banking Service arranges to make the payment to the payee through direct deposit, and returns a success (or failure) message to the Service that invoked it. If Figure 4: Applications maintained by two agencies this operation were done in a non-Service Oriented share a common Service. way, each of the programs (Unemployment Insurance, Medicaid, etc.) would have to write its own unique extension (in COBOL or Java, or C, or whatever programming language it might use). But in a SOA world, each of the programs can merely consume the one Banking Service  and it doesn’t matter to anyone what programming language might have been used to implement that Service. Services make requests and receive responses from other Services by exchanging messages. A message is a standards-defined data package that travels over a network (an internal network, private extranets, or even the Internet), very much like electronic mail messages are distributed. Unlike email, however, 28 / 70
  29. 29. Collaborative Application Development Among New York State Agencies messages between Services are formed using a protocol named SOAP. All SOAP messages must be written in the XML mark-up language. 21 Although messages are carried over the Internet, they are routed and processed within an organization’s own internal network by one or more computers which comprise an Enterprise Service Bus. The diagram below depicts an Enterprise Service Bus and the broad categories of Services whose messages pass through it. Figure 5: An Enterprise Service Bus routes messages among true Services as well as other, non-Service Oriented components. It might seem curious that we would need an Enterprise Service Bus just to send messages between applications. You would not in the scenario depicted earlier, where a large monolithic program is consuming an outside Direct Deposit Service. But that example is really showing a legacy application or a COTS application (which are not Service Oriented) making a request into a Service Oriented environment. Real Service Oriented applications are composites, entirely make up of Services. The structure of such composite applications looks something like that shown in the following diagram. 21In reality, XML is not much of a programming language at all. XML is more of a way to structure data so that any program can read it. 29 / 70
  30. 30. Collaborative Application Development Among New York State Agencies Whole applications can be created simply by having one Service call another service that calls another Service. Some Services interact with databases. Some do calculations. Some process rules. And along the way, Services make contact with real live humans through Graphical User Interfaces (portals). It can get quite difficult to figure out how an application works in a simple Service Oriented Architecture. But sophisticated SOA applications do not rely on individual Services themselves to know how they should behave in a greater application. Instead, applications can be created by orchestrating the sequence of Services through use of a computer called a Business Process Server that lives on the Enterprise Service Bus. That mouthful of technologies becomes clearer after examining the following diagram. Figure 6: Two, true Service Oriented applications have both program-specific Services as well as shared Services. 30 / 70
  31. 31. Collaborative Application Development Among New York State Agencies Here we show how an application is a composite of Services. Some Services are specific to the Unemployment Insurance program and would not be used anywhere else. Others are specific to the Food Stamps Program and have no value outside of that domain. Many services, however, are program agnostic. They can be used by any composite application that needs the service they provide. Services can be written so that each Service knows what other Service it should contact, what information it should send, and what data it should receive back, if any. But there is also another way to build applications from Services, and that is by independently orchestrating them. A Business Process Server, mentioned above, is a computer that executes another kind of XML script which tells the process server which Services to invoke and in what sequence. The diagram above would look the same even with use of a process server. A process server, however, provides an additional layer of abstraction so that agency workflow processes can be composed and re-composed without resorting to change to code in even the small, Services. Legacy applications (whether they are newer, object-oriented works or older, mainframe works) can participate in a Service Oriented Architecture, even though they are not themselves Service Oriented. This is also true of commercial enterprise (COTS) applications. Consider, for example, the OCIO’s Annual Technology Plan (ATP) application and various commercial project management applications. The legacy ATP application is a well-engineered, web-based, client-server application. It is not Service Oriented, but there are object libraries in this java application that could be leveraged to allow the exchange of data in a Service Oriented fashion with other applications.22 The OCIO uses the Mercury project management system. Other agencies use other mainstream products, such as Primavera and Microsoft Project. None of these commercial project management systems are Service Oriented. They can be made to join a Service Oriented Architecture, however, by using SOA wrappers that map native I/ O streams to a messaging interface through an Enterprise Service Bus. Linking the COTS applications to the ATP application in this manner would allow agencies to update their projects once, in their local systems, and have them automatically provide the data needed by the ATP. In this way, the general statewide ATP and the more detailed agency project systems can maintain perfect synchronization, while eliminating double work. Note also that SOA development can be started even before every other partner is ready. Under the New York State Elderly Pharmaceutical Insurance Coverage (EPIC) program, the Department of Health must 22 The OCIO has announced that it would like integration between the ATP application and the separate Intent to Purchase (ITP) database application. If both were Service enabled, they could exchange messages without rewriting the applications into a consolidated, monolithic application. If both were truly Service Oriented, linking the two applications would be only a few weeks work. 31 / 70
  32. 32. Collaborative Application Development Among New York State Agencies verify the income of participants. Tax and Finance provides the Department of Heath through an income look up service. Health cannot directly consume Tax’s service to get this information because it does not have the infrastructure. So presently, Health sends Tax a file of records to look up (5 million). Tax’s internal code reads the EPIC file and then calls its service that does the look-up and adds the data to the file. In the future Health can call the service directly either in batch at night or in-line during the filling out of the Epic application. On Tax’s side of the conversation, it will only require enabling the Service for external invocation by Health. Similarly, Labor has built a Service for capturing debarment data from the Workers’ Compensation Board (WCB). Because the Board has not yet established a Service Oriented framework, the connection between Labor and the Board is a traditional, point-to-point integration strategy. Internally within Labor, however, an application uses this Service to query a joint Labor-WCB database (maintained at Labor) in response to public inquiries on Labor’s public web site. The Service is used again internally at Labor to gather data on debarments made by Labor’s Worker protection programs. In time, the Service will be directly consumed by WCB as part of it’s own debarment application. Moreover, the Service is also ready for consumption by the Office of the Attorney General as part of its Project Sunlight initiative, and by the Comptroller as part of its Vendor Responsibility initiative. (Both applications involve collecting information for public exposure in different formats and for varying purposes.) Such data exchanges can be accomplished in a one-by-one effort to create common XML documents (i.e. data sets expressed in the XML markup language) between individual applications. A more efficient way to do this, however, is to use generally agreed upon data standards. Many such standards already exist. The OASIS standards organization, in particular, has been instrumental in developing XML document standards in a wide number of subject areas. Where there are no such standards available outside New York State government—and this will be the larger situation, agencies will want to establish a central data standards board. Such a board would logically come under the direction of the OCIO. 23 Note, however, that data standards in an open world provide flexibility that past efforts lacked. We need to be wary of rigid data definitions, specifically in terms of standard sizes for data fields across agencies. Past attempts to reconcile one standard across every agency, and their external partners, stalled development and ultimately failed. A more practical alternative is to use the Service Oriented publish and subscribe paradigm where agencies can publish their data exchanges or Services and participating agencies may subscribe to them. This means the subscriber knows what they are getting (e.g., individual name as first, middle initial, last or as a single text string) and the subscriber can respond 23 See the OCIO Enterprise Architecture principles, especially item 3.2.3, in Appendix B for a parallel view. 32 / 70
  33. 33. Collaborative Application Development Among New York State Agencies accordingly. Agencies don’t have to use the service, but if they want to then they are free to consume the Service under the published terms of use. Another strong feature of SOA is that it can be more secure, especially from a disclosure perspective. Today, agencies send files around or they give partner agencies access to only parts of their applications, like Tax and Finance’s provision to Labor. This allows for general searches by name, for example, but there is really no way today to verify whether any particular name search is warranted. Future implementation as a Service allows access to the application based on explicit parameters. Moreover, the Tax application which Labor uses now is a poor fit for users at Labor. But by consuming Tax’s Service, instead of Tax’s application, Labor would put Tax’s data on a screen designed by Labor to meet its business requirements and its users’ needs. What this means is the Labor application (not the user) can determine if there is a business need, thus eliminating random walks through the Tax data by Labor employees. Service Oriented Architecture also provides opportunity to further the effort to apply performance measurement to State programs. With communications between Services traveling across an Enterprise Service Bus, opportunities become available for developing metrics which measure work flow. Because business services are aligned perfectly with business processes, measurement of Service work flow provides business metrics that can be used for making quality improvements to governmental programs. 33 / 70
  34. 34. Collaborative Application Development Among New York State Agencies H. Service Oriented Architecture – the Business Case Business cases are intended to present the benefits, either financial or services provided, derived from an initiative. Both types of benefits are difficult to articulate in a public service based organization, rather than for a profit-driven organization whose benefits are measured by the impact on the bottom line. Nonetheless, the following section provides a number of areas for evaluation to develop sources of Business benefits that can be attributed to accepting an SOA approach. We know that the private sector’s Return on Investment experience with SOA is stunning. Consider that Helvetia Patria, a leading European insurance company based in Switzerland, saw increased profitability and business agility according to an independent study by Thoughtware Worldwide, LLC. The company also reduced e-business IT operational costs by 59 percent while it experienced an internal rate of return of 26 percent.24 25 In a study conducted by IBM Corporation, all 30 companies investigated that adopted SOA showed major returns. A cellular telecommunications company that was studied created an entirely new service for locating cell phones out of its legacy systems, re-orchestrated through SOA. This capability is estimated to open up a billion dollar market by 2009 for this company.26 In another instance, a large agricultural machinery manufacturer needed to boost its ability to finance sales in its showroom. It tapped SOA not only to improve and expedite current lending practices, but to provide a new lending product to keep pace with competitive alternatives. It was able to double loan application volumes and increase the loan decision rate from 15 percent to 55 percent.27 Ultimately SOA’s success is driven by the realization of benefits delivered to the business in the form of cost and/or service-based improvements. A Service Oriented Architecture provides a number of tools to stimulate these improvements. Although for the purposes of representing a Business Case it is simpler to consider SOA as an enabler for the business improvement opportunities. 24 Thoughtware Worldwide customer case study on Helvetia Patria, “Quantifying the Value of the eBusiness Center: HP Cost Savings Solutions for the Insurance Industry,” July 8, 2005. 25 Thoughtware Worldwide customer case study Helvetia Patria, “Helvetia Patria turns to HP to create industry leading e-Insurance Solutions – realizes 201% ROI,” January 2006. 26 Driscoll, Clement. “U.S.Mobile Resource Management Systems Market Shows Strong Growth in Subscribers and Revenues.” Location Intelligence. January 3, 2006. 27 IBM Global Business Services, 2006. “Service Oriented Architecture; a practical guide to measuring return on investment.” Somers, New York. 34 / 70
  35. 35. Collaborative Application Development Among New York State Agencies Business improvements will be recognized in two areas: IT resources and Program resources. Every department will have a different degree of improvement depending on the current state of their IT and Program processes and staff. Accepting that each department may realize differing levels of benefit, all agencies should consider the following sources of improvement to characterize the possible benefits that could be realized. The IT resource improvements will ultimately be realized through two sources: 1) the reuse of services reducing the amount of software that needs to built and maintained; and 2) the improved agility of the IT staff to provide new composite applications or modify existing services. Shorter durations required to create or modify software translates to fewer resources and cost. The program resource improvements will be realized through two sources as well: 1) improved resource efficiencies; and 2) improved resource effectiveness. Both of these are availed through the improved recognition of Business Process enhancements through Business Process Management practices. Ultimately the business will decide whether efficiencies will be translated as a cost reduction or a repurposing of staff to improve the service delivered by an agency. Both of which are not easily translated to a dollar figure across an enterprise of business services provided. Standards Based Integration To summarize points made in Section D, SOA dictates the use of standards-based interfaces between services. Considering that the services will be either providing access to information or business services, the use of a standard to support the interface will ease the use of services across state agencies or across separate programs within an agency. Standards will ease the effort to leverage other agency or program information and or business services. The use of new or improved data should enhance a business process either through sharper clarity within a business interaction or quicker delivery of services. Consider the business scenario where individual’s data in the control of the Department of Motor Vehicles could be easily accessed to determine whether the person interacting with the application is whom they report to be. The use of a business service would either reduce internal staff requirements or simply make a customer’s interaction with the state easier via a single interface point or clearer interaction creating a simpler, shorter customer-to-government interaction. Consider the business scenario where an individual is registering with Governor’s Office of Regulatory Reform’s on-line license and registration system. By Labor (and other corresponding state agencies) providing access to their UI Employer Registration business service 35 / 70
  36. 36. Collaborative Application Development Among New York State Agencies from each agency, one registration form into the Governor’s Office of Regulatory Reform system could simplify the individuals interaction with the state and standardize the core data representing the business across the various state agencies. Business Process Management SOA utilizing Business Process Management tools (for example, IBM’s Business Process Manager) will improve any institution’s grasp on the Business Processes that support their business and ultimately should improve the institution’s Business Intelligence. Through the management and measurement of key business metrics now available via the Business Process Management tools, it is proven that those things measured will improve over time. As stated earlier, measured improvements in a business process are subject to both the current state of the business process and the ability to institute change in the business process. As with any improvements in the business processes as a result of standardized interfaces, a better understanding of a business process resulting in better decisions regarding the business processes are expected to provide benefit either in the efficiency of the program resources required or an improvement in the services provided to the consumers of the business service. Improved Agility As described in Section 2, SOA enables both the “Speed to Market” and “Agile Systems Development” factors. Any savings within the IT resources in delivering new or enhanced application functionality is a direct savings in the cost of enhancement. As with the program improvements, savings with the IT resources in the delivery of change will translate into further improvements in the delivery of automation support. 36 / 70
  37. 37. Collaborative Application Development Among New York State Agencies I. Service Oriented Architecture Formalism SOA is a conceptual business architecture where business functionality, or application logic, is made available to users, or “consumers”, as shared, reusable services on an IT network.28 Organizations will have two broad classifications of services: (1) business services and (2) technical services. Business services reflect business concepts and events, and thus are excellent organizing principles for an SOA. They could also be called business process services because they are associated with executions of business functions of an organization or business process.29 Technical services are those services that are horizontal in nature or are reusable by all business processes, business units, or process domains. Technical services include security services, logging services, audit services, transformation services and similar ‘IT’ services that would be leveraged by and across all lines of business. In some organizations, these technical services would be described as enterprise common services.30 Characteristics of Services Services, in order to meet the needs of the organization, must meet certain criteria to provide the most value to the organization.  Coarse grained  Well-defined services contracts  Loosely coupled  Discoverable  Durable  Composable  Business aligned  Reusable 28 Marks, Eric A. and Bell, Michael, Michael 2006 Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology, Bell, Michael John Wiley & Sons, Inc., Hoboken New Jersey. p 1. 29 Ibid. Page 37. 30 Ibid. Page 38. 15 Ibid. Page 38. 37 / 70
  38. 38. Collaborative Application Development Among New York State Agencies  Interoperable  Stateless  Dynamically Bound31 Coarse Grained Services Services should be coarse-grained entities. Service granularity depends upon how much functionality a service encapsulates and exposes. Coarse-grained means that services should represent business functions, processes, or transactions and encapsulate other fine-grained components or services within them. A service that is too big or too coarse-grained will suffer from performance issues and will not be reusable. A service that is too fine-grained will be too narrow in scope to meet requirements of multiple business processes.32 Well-Defined Service Contracts Services must have well-defined contracts that separate the functionality of the service from its specific technical implementation. The service contract informs consumers what the service does as well as how to consume or use the service. Services contracts present the service functionality to the outside world in a standardized, interoperable fashion while hiding the specific internal technical details of the services. 33 Loosely Coupled In the services design aspect of the term ‘loosely coupled’ means designing services that specific implementations of services can be replaced, modified, and evolved over time without disrupting the current service consumers and the overall activities of an SOA.34 “Coupling refers to the number of dependencies between modules. There are two types of coupling; loose and tight. Loosely coupled modules have a few well-known dependencies. Tightly coupled modules have many unknown dependencies. Every software architecture strives to achieve loose coupling between modules. Service 31Thomas Manes, Anne 2006 Service-Oriented Architecture: Developing the Enterprise Roadmap, Version 2.0. Burton Group, Midvale, Utah p 41. 32 Marks, Eric A. and Bell, Michael 2006 Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology, John Wiley & Sons, Inc., Hoboken New Jersey. p 39-40. 33 Ibid. Page 40. 34 Ibid. Page 40-41. 38 / 70
  39. 39. Collaborative Application Development Among New York State Agencies oriented architecture promotes loose coupling between service consumers and service providers and the idea of few well-known dependencies between consumers and providers. 35 Discoverable Services should be discoverable. This means that their contracts are published and visible to an intended audience – the consumers. Discoverable services implies that the service contracts – WSDL documents in the case of Web Services - are published to a location where they can be discovered, whether that location is a service registry, metadata repository, subdirectory, or some known location. Once the service contracts are published, they must be advertised to potential consumers.36 Stateless One of the major characteristics of a service is that it is Stateless. It should not have to know the process state that invokes it (that for the process or composite application). If the service knows state it get vastly more complicated because it has tests to see what has to be done. This vastly reduces reusability because before you can use the service you have to figure out the state to call it in, and you may even have to create a new state that has to be coded in the service. Dynamically Bound A service consumer that needs a service discovers what service to use based on a set of criteria at runtime. The service consumer asks a registry for a service that fulfills its need. The Registry returns all entries that support this. The entries also contain information about the service, including transaction fees. The consumer selects the service (provider) from the list based on the lowest transaction fee. Using a pointer from the registry entry, the consumer then binds to the provider of the service. The description of the service consists of all the arguments necessary to execute the service. The consumer formats a request message with the data, based on the description provided by the pointer. The consumer then binds the message to a transport type that the service expects and sends the service the request message over the transport. The service provider executes the [service] and returns a message whose format is also specified by the service description. The only dependency between the provider and the consumer is the 35Thomas Manes, Anne 2006 Service-Oriented Architecture: Developing the Enterprise Roadmap, Version 2.0. Burton Group, Midvale, Utah p 49. 36Marks, Eric A. and Bell, Michael 2006 Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology, John Wiley & Sons, Inc., Hoboken New Jersey. p 41. 39 / 70
  40. 40. Collaborative Application Development Among New York State Agencies contract, which the …registry provides. The dependency is a runtime dependency, and not a compile-time dependency. All the information the consumer needs about the service is obtained and used at runtime.37 Durable Services should be durable yet elastic. Durable services are those that map to lasting business or process themes.38 Composable Services should be Composable. Composable services are designed to be incorporated into other services as composite services as necessary. In addition, composable services can be assembled into orchestrated process flows. In this sense, composable services are stateless and anatomic in nature. They stand on their own yet rely on other services or infrastructure for state and context.39 The goal for service design is to identify the smallest unit of software that can be reused in different contexts.40 A service’s composability is related to its modular structure. Modular structure enables services to be assembled into applications the developer had no notion of when designing the services. Using preexisting, tested services greatly enhances a system’s quality and improves its return on investment because of ease of reuse.41 Business Aligned Services should be business aligned. Service identification and analysis should begin with business imperatives and business requirements. Services, or business services, represent business concepts and match business needs as determined through business strategy and planning and the associated business discovery processes that should precede an SOA initiative.42 37Thomas Manes, Anne 2006 Service-Oriented Architecture: Developing the Enterprise Roadmap, Version 2.0. Burton Group, Midvale, Utah p 41. 38 Ibid. Page 42. 39 Ibid. Page 42. 40Thomas Manes, Anne 2006 Service-Oriented Architecture: Developing the Enterprise Roadmap, Version 2.0. Burton Group, Midvale, Utah p 42. 41 Ibid. Page 59. 42Marks, Eric A. and Bell, Michael 2006 Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology, John Wiley & Sons, Inc., Hoboken New Jersey. p 43. 40 / 70
  41. 41. Collaborative Application Development Among New York State Agencies Reusable Services should be reusable. This is a function of proper services identification, analysis, and design. Implement services that have clear and defined reuse across and within business processes and that have multiple consumption patterns in your current and planned business processes.43 Interoperable Services should be interoperable. Interoperability is the ability of systems using different platforms and languages to communicate with each other. Each Service provides an interface that can be invoked through a connector. An interoperable connector consists of a protocol and data format that each of the potential clients of the service understands. Interoperability is achieved by supporting protocol and data formats of the service’s current and potential clients.44 Location Transparency Location Transparency is a key characteristic of service-oriented architecture. Consumers of a service do not know a services location until they locate it in the registry. The lookup and dynamic binding to a service at runtime allows the service implementation to move from location to location without the client’s knowledge. The ability to move services improves service availability and performance.45 Service Development Process The development life-cycle of services includes the following practices: Identification of Services The initial step involved in the development of services is to first identify candidate business services, which can later be analyzed for their utility and feasibility, leading to selection of a set of final business services to be designed and realized for use. Candidate Service Analysis Analysis of potential services is a top-down process of logical operations and transformation mechanisms to study and inspect enterprise core entities, ascertain business ideas and concepts, and lead to the 43 Ibid. Page 43. 44Thomas Manes, Anne 2006 Service-Oriented Architecture: Developing the Enterprise Roadmap, Version 2.0. Burton Group, Midvale, Utah p 48. 45 Ibid. Page 59. 41 / 70
  42. 42. Collaborative Application Development Among New York State Agencies formation of final business services. Services analysis on candidate business services can address consolidation, decomposition, reuse, simplification, and refactoring of legacy assets as dictated by agency imperatives. These activities present opportunities to transition to shared reusable services in a loosely coupled architecture. This service identification and preliminary analysis is a primary responsibility of the UISIM Development Board. Services Design The service design phase enables the implementation of physical services through a series of design activities. The service design process transforms final business services into physical solution services. Inputs into service design are a set of target services that meet organizational needs and have been prioritized and selected from the list of final candidate services. Service design begins with a final business services granularity map that facilitates the visualization of encapsulated business process, their desired business value, and their potential as physical solution services. The relative granularity of services helps determine their viability and appropriateness as part of the SOA strategy. The service design phase is not focused on the underlying technology or implementation details of service components. Source code is irrelevant to physical service structure design because service design activities tackle conceptual, logical, and physical compositions of services, examine their encapsulated business processes, and facilitate establishing reliable and reusable physical services.46 Realisation of Services From service analysis and design, the next step is to generate physical solution services that correlate to the services designs. Solution services are physical implementations of service abstractions. They consist of components, which are comprised of business processes. Solution services may also contain other services. They should provide access and exposure to external consumers, yet they have a distinct internal architecture that comprises business and technological ingredients.47 46 Marks, Eric A. and Bell, Michael 2006 Service-Oriented Architecture: A Planning and Implementation Guide for Business and Technology, John Wiley & Sons, Inc., Hoboken New Jersey. p 123-124. 47 Ibid. Page 137. 42 / 70