The document discusses the evolution of a distributed middleware platform called Metro used by Telenor Fixed Line Services over the past 20 years. It describes the platform's goals of standardizing integration between systems and describes how it has grown from a small initial platform to one handling large volumes of traffic through the use of tools like Maven, Confluence, and Jira. It also discusses how the platform team standardized processes, documented guidelines, provided certification courses, and enforced code quality to improve efficiency and scale the platform to support many projects.

1. Distributed Middleware Factory Ramón Arellano (Accenture) and Paul René Jørgensen (Telenor)

2. About us Ramón Arellano Work experience 6 years of experience as a technical consultant from Accenture E-mail [email_address] Paul René Jørgensen Work experience 5 years of experience from Telenor E-mail [email_address]

3. Distributed Middleware Factory - definition Distributed In the geographical sense of the word, i.e. various locations spread around the world Middleware The communication layer that lets applications interact across hardware and network environments Factory A facility used to manufacture goods in an effective way, using standardized industrial processes and specialized equipment

4. Distributed Middleware Factory – definition continued Multiple teams in different locations, producing software for system integration, on the same platform, using standardized architecture and tools

5. So what are we going to talk about? Middleware platform for Telenor Fixed Line Services Metro - A distributed middleware factory In this talk we’ll share our experiences from its evolution and some secrets from the past 6 years

6. Epochs in Metro history Epoch 0 – Medieval Age Epoch 1 – Gunpowder Age Epoch 2 – Enlightenment Epoch 3 – Industrial Age Epoch 4 – Modern Age

7. Epoch 0 – Medieval Age Year 2000 Challenges Where is the vision for the future? Use of the existing middleware platform is not mandatory Kings and kingdoms and their point to point integrations Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

8. The Metro Vision All integration between systems for Telenor Fixed Line services to be implemented through Metro platform Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

9. Metro platform goals To offer test and production environments to all development project To enforce architecture and development standards To supply a common set of core components Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

10. Distributed development model The platform is managed and controlled by a small, centralized team The Telenor employer selects software supplier The software supplier performs the implementation on the platform Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

11. Distributed vs Centralized Employer chooses software supplier Development is done where the supplier is Easier to get new developers up and running Requires a stronger regime and documented standards Platform team chooses software supplier Development is done in-house Easier communication between all teams involved Does not scale so easily Platform Platform Employer Employer Employer Employer Employer Supplier Employer Employer Employer Employer Employer Supplier Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

12. Epoch 1 – Gunpowder Age Year 2001 Challenges The volume and complexity of Telenor Common architecture How do we organize SI-testing? Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

13. Statistics In an average hour the platform is processing 61 200 web requests from the internet 115 200 EJB invocations 2 880 000 database queries 65 MB log messages (1,5GB in a day) Volume Over 50 systems integrated Average 200 deployments a year Over 250 different developers from all over the world have written code Metro platform team guides up to 30 teams at the same time Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

14. Name a technology, and we’ve got it EBXML Tibco B2B SOAP, Proprietary HTTP/XML Web Services Proprietary XML (JItegra Java COM bridge) IIS / ASP J2EE, RMI, Corba Java Terminal, Screen Scraping, MQ, CICS IBM Mainframe (MVS) Siebel XML Siebel JDBC, ODBC, DB2Connect Oracle, Sybase, MySQL, DB2 Protocol Technology Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

15. Technology at the base J2EE is the chosen technology IBM WebSphere is the chosen application server Started off with IBM WebSphere 3.5 and J2EE 1.1 WSAD as IDE (based on Eclipse) IBM MQ Server for messaging Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

16. Layered architecture Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

17. Informal roles Experience has shown us the importance of having roles in a team The roles are more or less self imposed Examples The hardware geek The Maven guru The support girl The documentation dude The tool guy Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

18. Test Environments Implementation, building and unit testing on developer’s laptop System integration testing on 60 blade servers running the full J2EE runtime environment One for each development project! Acceptance Test environment administered by the platform team Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

19. Epoch 2 - Enlightenment Year 2002-2003 Challenges We want to be more innovative How do we get implementation projects to help develop the platform? How do we organize our work and control the deliverables? Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

20. The 80/20 rule Innovative and flexible leaders introduced the 80/20 rule Developers were allowed to use 20% of their time to learn and play with new technologies Highly motivating, and has undoubtedly been fruitful for Metro Technologies such as multicasting (Log), FishEye (CVS), were introduced by experimenting Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

21. Boiling soup on stone The platform team is responsible for setting the standards for many projects and developers Dave Thomas's book, “The Pragmatic Programmer”, talks about “boiling soup on stone” Introducing new technologies and standards by curiosity and envy The whole package more valuable than each part Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

22. Metro Roadmap Our common workflow and methodology for all projects A set of phases with corresponding deliverables Waterfall model (may jump backwards and iterate) Each project gets their own supervisor responsible for QA Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

23. Roadmap control in Jira We’ve got a tool to enforce Metro Roadmap Atlassian’s inexpensive, but excellent tool Jira provides workflow functionality Controls the flow and stores all document deliverables It’s the platform team’s communication channel with development projects Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

24. Epoch 3 – Industrial Age Year 2004-2005 Challenges Lack of documented standards New developers must get up and running fast We feel we could be more efficient in getting new applications out Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

25. Document everything Roadmap workflow Architecture principles Development environment setup Naming standards Step-by-step guidelines Testing and debugging Building and deployment FAQ Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

26. Documentation in Confluence Atlassian’s professional wiki, Confluence, for all documentation Wiki: easily updated, everybody gets the current document version Ported existing documents from MS Word Also used for blogging news and alerts through mailing lists and RSS feeds Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

27. We use Open Source Result of the 80/20 rule We’ve found free tools to improve our platform We love open source, and we give back Maven plugins Confluence plugins XRadar Jira plugins Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

28. Leading Star Example Leading Star Driven development Improves application consistency TelenorShop – a fictitious application It shows live examples of all our common components and frameworks in use Source code for all Metro applications is openly available internally “Take a look at TelenorShop, and see how we’ve done it there” Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

29. Automation through 3rd party tools Automatic builds with unit testing using Maven Generating web-site with reports and JavaDoc using Maven Continuous integration using CruiseControl Mapping between data models using Dozer Support request through Jira Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

30. Automation through self-made tools Applying for deployment slots to acceptance test environment Generating overview of dependencies between all J2EE applications on server Searching logs in all environments Updating application configuration Backing up WebSphere Application server configuration Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

31. Metro Certification A good factory scales easily New resources must be brought up to speed – fast No development without certification J2EE is a prerequisite Certification courses for Developers (2 day course) Project Leads (1 day course) 100 certified developers and project leads Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

32. Other certification benefits Efficient development Consistent solutions Stable solutions Reusable services Better developer attitude “ Ah, so that’s why we have to...” Externals pay a course fee ($) Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

33. Metro Certification content Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

34. Epoch 4 – Modern Age Right now, right here Challenges Middleware is only noticed when something goes wrong How do we govern dependencies? Deployments are held back due to poor code quality Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

35. Commercialization We must learn how to better sell ourselves internally We want more applications on the platform and everyone to be aware of what Metro has to offer How Create a logo Offer introduction presentations Show off statistics of usage Present at JavaZone Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

36. Service Level Agreements SLAs between API owners (the kings) Wanted since the medieval age! No access to remote API without a signed SLA It will give us valuable information about inter-application dependencies We must work on enforcing this in the modern age! Requires a new tool (Any suggestions?) Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

37. Code Quality Step-up the code QA work Make it easier for the reviewers to focus on code that is being altered Reviewers are notified immediately on CVS commit We’ll introduce Crucible into the review process Defines a workflow, creates an arena for dialog Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

38. Architects must write code “ Real insight comes from active coding” Practices of an Agile Programmer

39. Summary Key tools Key lessons

40. Key tools Confluence (wiki) for documentation No delay from writing to publication Jira No more forgotten issues in inboxes all over the place Issue management and process (roadmap) workflow Maven Stable and proven build system Eclipse based IDE Allow only one IDE Be an expert to support the developers

41. Key Lessons Easy to understand workflow for everybody – Roadmap Documented standards and guidelines – Wiki And a way to teach them (Certification) Decision from the top : All integration between systems to be implemented through the platform

42. Any questions? paul-rene.jorgensen@telenor.com / ramon.arellano@accenture.com