![Secrets forthe success
Scrum
[Dev] Agility
Retrospectives
Frequent releases/user demos
Notion of Progress
User Interface
Motivated Team Focus
Usable and simple
Aesthetic
Innovative
(not what you would expect in a bank)
Team principles - quality
Continuous improvement
Individual initiative&strengths
Redundancy in skills
Very low turnover
Work-life balance
Challenges
Latest&Greatest technology
Physical workspace
Great relationships
OVERVIEW](https://image.slidesharecdn.com/20150522w05-architectingalargesoftwareproject-lessonslearned-joopedromartins-distributionversion-150522121333-lva1-app6891/75/ITARC15-Workshop-Architecting-a-Large-Software-Project-Lessons-Learned-10-2048.jpg)
Uploaded byJoão Pedro Martins
ITARC15 Workshop - Architecting a Large Software Project - Lessons Learned
The document presents insights into architecting a large software project, detailing the development of an internal banking application that supports over 3000 users. It shares lessons learned from the project, encompassing best practices in team management, technology choices, and code quality, while addressing challenges experienced during the development process. The speaker emphasizes the importance of effective communication, user collaboration, and continuous improvement to successfully deliver a quality product.
More Related Content
![[第35回 Machine Learning 15minutes!] Microsoft AI Updates](https://cdn.slidesharecdn.com/ss_thumbnails/20190427ml15minmicrosoftai-190427060706-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to ITARC15 Workshop - Architecting a Large Software Project - Lessons Learned
ITARC15 Workshop - Architecting a Large Software Project - Lessons Learned
- 1. Architecting a LargeSoftware Project – Lessons Learned João Pedro Martins ITARC 2015 IASA WORLD SUMMIT
- 2. Ground Rules andExpectations ∃ Break Interaction expected – share your experiences Software Architecture level (yes, technology is included)
- 3. João Pedro “jota”Martins CTO @ |create|it| Software Architect TechEd 2006 – “Iron Architect” Winner BizTalk Server MVP 2006-2011 Co-founder - GASP + APPU Windows Azure Insider
- 4. What’s this sessionabout? Architecting a Large Software Project – Lessons Learned
- 5. Overview Internal Banking applicationfor corporate credit allocation. 3000+ users. Dev started Jun-2011, R1 installed Jul-2014. Currently on progressive rollout with updates. Web-based, running inside a Windows Forms shell, developed on Microsoft technologies. Scrum - 56 sprints, ~15 dev-years Core team: 14 (5 devs) + ~10 in other teams Integrates with 11 systems THEPROJECT
- 6. Current statistics Applicationcode- LOCs Presentation:63,664 Backend:116,145 Business Rules(xml):36,473 Database:~4,000 Automatedtests:157,586 And also… GeneratedCodeLOCs 81VisualStudioProjects 24Services with213Operations 86Presentation views 1,481classes,96databasetables 1,755AutomatedTests,~80%coverage of backendcode. 100%StyleCopcompliance >15libraries andpackages~800,000 THEPROJECT Storiesand Issues ~550userstories 1,064issues:98%closed,78%bugs.0.04% showstoppers. Inlast5sprints… 56%timenewdevelopments,44% fixes/enhancements Production updatesevery4days
- 7. ConceptualArchitectureandTechnologies THEPROJECT browser dataservicesweb Controller View Model business logic ruleengine integration tables + views data access stored procedures browser web services data NET 4.0 CODESMITH + NETTIERS TEMPLATES T-SQL SQL SERVER + VS DATA TOOLS SSMS NET 4.0 WCF NXBRE LOG4NET UNITY (ENTLIB) VS UNIT TESTING RHINO.MOCKS SPREADSHEET.NET QUICKGRAPH.NET TOOLKIT CREATE .NET 4.0 ASP.NET MVC3 LOG4NET JSON.NET UNITY (ENTLIB) TOOLKIT CREATE BALSAMIQ MOCKUPS HTML + CSS JQUERY KNOCKOUTJS UNDERSCOREJS KENDOUI MOMENTJS TOASTR
- 8. Architecture design: fromTheoretical toPragmatictoImplementable Theoretical Implementable ESB/Integration Bus BizTalk Server SOAP Web Services with bank’s custom ServiceHost Enterprise Rules Engine BizTalk Server BRE / Excel Services NxBRE (OSS) – XML Rules Engine Distributed Cache VelocityCTP3 / Windows AppFabric Cookies + Web Server Afinity Load Tests VS2010 Test with distributed agents VS2010 single server load Application Lifecycle Management (ALM) TFS SVN + Issue Tracker + Excel Product/Sprint Backlogs Increasing levels of contraints limit the choices: - Cost, time, available IT, risks, governance, security, skillsets, …
- 9. Enough context Now, whatwe learned…
- 10. Secrets forthe success Scrum [Dev]Agility Retrospectives Frequent releases/user demos Notion of Progress User Interface Motivated Team Focus Usable and simple Aesthetic Innovative (not what you would expect in a bank) Team principles - quality Continuous improvement Individual initiative&strengths Redundancy in skills Very low turnover Work-life balance Challenges Latest&Greatest technology Physical workspace Great relationships OVERVIEW
- 11. Classeswithtoomanyresponsibilities LEARNING#1 Service Implementation Business Manager Data AccessLayer Operation Contracts Data Contracts Business Manager Thesegottoolarge WCF Physical Hosting (.svc’s)
- 12. How toworkaround this? SOLIDprinciples - Single Responsibility Domain Driven Design – vertical slices of behavior Separation of Concerns – partial classes + smaller classes When in doubt: • Create another class • Use interfaces LEARNING#1
- 13. Logging Do you havedetailed logging enabled in your production environments? LEARNING#2 Is this useful?
- 14. Logging Instead of this(the “Laconic Logging” Anti-Pattern)… … do something like this… LEARNING#2
- 15. Logging-hints Log operation detailsvs Security/Privacy Use end-to-end correlation. Define your logging policy. Beware of impact on performance. Beware of storage space required. DO cleanup/archive. LEARNING#2
- 16. Interfaces vs Inheritance “Whyextends is evil - Improve your code by replacing concrete base classes with interfaces” (Allen Holub) http://www.javaworld.com/article/2073649/core-java/why-extends-is-evil.html LEARNING#3 Dependency injection interfaces. Very limited use of OO inheritance - Base Data Contracts with minimum property set (e.g., id + name) - About ~10 uses in total
- 17. Dependency Injection Indirection wheninstantiating objects: - Container builds and reuses objects - Flexibility and reduced coupling - Supports mocking for automated tests Interception: - Cache - Logging - Profiling - Exception Shielding Fully configurable. LEARNING#4 Caller Target Interceptor – Call handler
- 18. Dependency Injection "All problemsin computer science can be solved by another level of indirection“ (David Wheeler) LEARNING#4 Config If asked for IServiceA create ServiceA instance creates&uses A
- 19. Dependency Injection pitfalls Initialsetup can be demanding (skills+time) Programming configurations (complex debugging) Impact on runtime performance Productivity (F12 goes to interface and not implementation) LEARNING#4
- 20. Cache Cache transparentlyviainterception + configuration. Cachebefore accessing the network. Initially designed for 3freshness configs. Idempotence as a side-benefit. LEARNING#5 Presentation Layer Whenever this assembly is used … and a method with this name is called … apply this interceptor with this configuration
- 21. Cache: ooops! Business informationpresented must be accurate – and data is not stable in external systems. Very little external reference data. User authorization. Transparent, configuration-based cache, is convenient - however, you can’t selectively expire contents LEARNING#5
- 22. Transparent caching pitfals Peroperation call, not per business entity E.g.,“method_1_2_3”ascache_key,insteadof“client_1” You don’t control cache keys on the runtime (general purpose cache-key: hash generator of input parameters) Hard Impossible to track dependencies in complex business models. => Cache invalidation is all-or-nothing LEARNING#5
- 23. Automated tests Requirement: 80%coverage by automated tests in service layer. Team principle: the AGILE team is not afraid to change any piece of code for fear of breaking something. Approach: service-level, end-to-end tests - Visual Studio Tests framework - Not unit tests integration, end-to-endtests - Depend on external data - Sprint Backlog: One service operation one test set LEARNING#6
- 24. Automated tests –mmmm… Test suite takes too long to run (~2-3h) : - SQL scripts SQL Database Snapshots - Service layer tests Business layer tests External data not stable – mocks But: how to test complex business cases dependent on external data of which we can’t be sure? LEARNING#6
- 25. Automated tests –moremmmm… Have a Test King in the team to nurture and run tests VS2012test runner worse/slower than VS2010 (!) Smart asserts can help improve code-coverage: Tool recommendation: SSMS to generate T-SQL from data LEARNING#6
- 26. Code Conventions Agree oncoding conventions and stylecop compliance at start of project. Architect/Dev Lead name all the main artifacts: service contracts, database artifacts, etc. Strive for consistency. LEARNING#7
- 27. Code Conventions notes Focuson code legibility: - Comment your code (take special care with algorithms) - Don’t use var for non-anonymous types - Don’t overdo Linq statements Mistakes will happen, and rename refactors will be needed (mixing PTwith EN is frequent). Standardize verbs in services/methods, db naming (ex: List vs GetAll) Do NOT argue tab size. When in doubt, use defaults. LEARNING#7
- 28. Negotiation…withyourteam,andwiththecustomer Always voice youropinion, focusing on what you think is the best for the project architecture-wise. Create a trust relationship. When your recommendation is not followed, and you are sure you are right, present objective arguments – don’t be emotional. Argue for as long as you must, but no longer. And don’t say “I told you so”. Accept defeat, make compromises. LEARNING#8
- 29. Negotiation…somemorenotes Be attentive ofthe other’s possible hidden motivations, but be careful in exposing them. Consensus is not always possible. Your options will be questioned, and sometimes you will be wrong. Remember the 3 views of architecture: theoretical, pragmatic, implementable. Ask open questions. LEARNING#8
- 30. Functional team/domain experts Theyare your peers, and part of the team. You depend on well written and clear user stories. Domain experts that understand Scrum, priorities and constraints make the difference. Rely heavily on them and their tools. When you don’t understand, ask questions until you do. LEARNING#9
- 31. Functional Team do’sand don’ts Sometimes the way a story is written crystalizes a way of implementation. Some stories will be hard to understand and decompose into tasks. Ask for clarifications and don’t implement blindly. Business context is sometimes missing. Tendency to “follow the old ways”. Tendency to abide to single-user/hierarchical requests.
- 32. Use Your Brain:designelegantly You are not paid to write code, you are paid to think and communicate. Think things through before committing to a solution. Try to isolate and design autonomous and change- tolerant components. Step back, look at the larger picture. As an Architect, you DO NOT have to be a technical expert in everything: focus on capabilities and structure. LEARNING#10
- 33. Impediments to usingyour brain Interruptions, background noise, phones, no whiteboard, lack of natural light, music on headphones, time or budget pressure, too much coffee, personal problems, lack of sleep, … What’s your style: collaborate then design, or design and then collaborate? Isolate yourself to design. Make drawings, and then document your proposed solution. *Thinking is hardwork.
- 34. Wireframes Create and discuss mockupspre- implementation. Ps: if you have an UI expert in your team, don’t let business experts create them. Tool recommendation: BalsamiqMockups LEARNING#11
- 35. UsabilityTests Usability tests aresimple! Just looking at users during training uncovered both problems and ideas for improvement. LEARNING#12
- 36. Revisiting 4 technicalchoices KnockoutJS or MVC3? Took time to decide and spike, there was an initial setback with KO and adoption was reversed. 2nd attempt and investment proved correct. NxBRE Rules Engine QuickGraph.net3.6 DistributedCache XML-based rules engine DLL. XML file can be replaced without recompilation. Works fine and is fast, but hard to code and read. Jury is still out. Formal Architectural feedback was tacitally dismissed as non- pragmatic, and package was used. VelocityCTP3 was refused as non- supported. AppFabric not available in Windows 2003. Oracle Coherence never provided.
- 37.
- 38. Your team isan extension of your body.
- 39. Use diagrams tocommunicate and structure your ideas. NOT MY SKETCH
- 40. Use an issuetracker, designate someone(s) to do the triage, and configure mail alerts, your pages/modules, team, and sprints. Teach the client how to use it for bugs & enhancements.
- 41. Know thy user’spc: @start, 1GB RAM, IE8, WinXp, 1024px Javascript + IE8 + 1GB RAM recipe for disaster.
- 42. 4 layers &no distributed cache mean no real-time features
- 43. Be lazy. Don’twaste time coding your own, special, data- access layer/library/.... Scavenge codeplex, github, nugget, etc. for assets&tools to reuse or buy.
- 44. Use extension methods– don’t pollute your classes with auxiliary methods (ex: finders in collections) * and kill those «helper» classes, too
- 45. Just 3 more… Humans make mistakes. Scripts don’t. Use scripts and obsessively automate repetitive tasks or installations. Know your branches, merges, shelves, labels, versioning (just use best practices, don’t invent). Innovating and surprising your customer, and the cherry on the cake, makes a world of difference.
- 46. Closing message It’s anarchitect job to address the clients’ needs and deliver quality products. Three stone cutters were asked abouttheir jobs. The first one replied, “I’m paidto cutstones.” The second replied, “I usespecialtechniquesto shapestones inan exceptional way,hereletme showyou.” He proceeded to demonstrate. The third just smiled and said, “I buildcathedrals.” -Ricardo Semler Hope I helped!
- 47. João Pedro “jota”Martins jota@create.pt (+351) 96 782 5537 blogit.create.pt/joaomartins/ twitter.com/lokijota pt.linkedin.com/in/joaopedromartins/ ITARC 2015 IASA WORLD SUMMIT Thanks! Questions?Thoughts?
- 48. |create|it| Started 2001 @Lisboa, Portugal Systems Integrator Team of 26 Microsoft Gold Certified Partner Azure BizTalk Office365 SharePoint Umbraco NopCommerce
Editor's Notes
- #2 Logo font: Monserrat Regular Logo color: #9a8e5e (154,142,94) Morning workshops 09:00–12:30 Hello, good morning. My name is João Martins, but people usually just call me “jota”, which is probably also easier for you to say . We’re going to spend the next 3,5 hours discussing real-life experiences in architecting and implementing a software project I did in Portugal, both the good and the bad. But before we go on, I’d like to clarify some ‘rules’ and expectations regarding this session.
- #3 First, there will be a break. [PRECISO DE SABER SE HÁ E QUANDO É EXPECTÁVEL]. Second, this session relies on there being interaction and questions, or else it’ll be just me passing slides for the duration. Ask questions, clarifications, comments, share your own experiences. Third, a lot of what we’ve heard yesterday was somewhat on a conceptual level. This session is much more operational, covering specific programming concepts, team dynamics, user experience, etc. and the learnings from the concrete experience of setting up a specific project for what turned out to be much longer than we expected. This said, I’d like to have an idea how this maps to your own expectations: is this what you are looking for? What is your role in your organizations?
- #4 GASP was the Portuguese software architecture group, a former IASA chapter, where I was the president, and APPU is the Portuguese usability professionals organization, of which I was the chairman. GUARDAR O COMENTÁRIO SOBRE AS SESSÕES DO GASP PARA O PRÓXIMO SLIDE.
- #5 Architecting >> design & communication, what I consider to tb the two key skills of na Architect: DESIGNING AND THEN MATERIALIZING A SOFTWARE APPLICATION. Large >> what is large? A few months back I met a couple of people from Deloitte, who were working in I think Norway, with a fully portuguese team of 200 people, creating the software system for an insurance company. That is not a project, but a program. This session is about a single project. It’s also probably not large for countries like the US and maybe Sweeden, but we consider it large in relative terms compared to what we usually do in Portugal, large because of the 3 years it took to go live, and large in terms of complexity: We used to joke and say that – after this – is rocket Science. Software project >> technology and engineering in one specific project. Goal getting code running. Lessons Learned >> mistakes and successes. Two notes: a) a few years back when I led the IASA Chapter in Lisbon, the most value we took were the experience-sharing sessions, usually under NDA, about projects we did. This is part of the goal for this session. b) I’ll be Talking about several different types of learnings, some technical, some architectural, some human-related, some functional.
- #6 Quickly describe what is the goal of the application, the proactive allocation of credit limits to companies. This allows for: Proactive sales with immediate on-the-spot approval Improved response time Automation of credit concession operations on Clients’ web portals/apps + sales people tablet apps Detalhar o porquê dos 3 anos in simple terms, this was not due to delays, but due to the fact that the definition of “Minimum Viable Product” was a moving target for 3 years. For us, this could have gone live after 1 year. MIG, PAT, SNL, RB + 4 keyusers GDC/GPC: JAAlmeida, NLetras, ANDré, OCTávio, Teresa, Outra funcional DW: LA, DC: 2
- #7 Number of bugs increased after rollout, 64->78%. Most of them in the UI, which is not covered by tests. Interesting! **** MANUAL CODE TOTALS ***** Web C# LOCS: 32480 Services C# LOCS: 101438 Integration C# LOCS: 8708 Mock/Tests C# LOCS: 120437 Mock/Tests SQL LOCS: 37149 Common/Proxy/Tools C# LOCS: 5999 Custom SP/Function LOCS: 2590 HTML LOCS: 11341 Javascript LOCS: 17957 CSS LOCS: 1886 Services Rules XML LOCS: 36473 GRAND TOTAL: 408938 **** ARTIFACT TOTALS ***** Total Projects: 81 Total SQL Tables: 96 Total SQL Stored Procedures: 1186 Total SQL Functions: 47 Total SQL Views: 70 Total HTML Views/Partial Views: 86 Total Javascript functions: 1868 Total WCF Services: 24 Total WCF Service Operations: 213 Total WCF Data Contracts: 336 Total C# Public/Partial/Static Classes: 1481 Total Automated Tests: 1755 **** GENERATED CODE TOTALS ***** Total DAL C#: 741293
- #9 Como defino: Conceptual: o que queremos, em teoria Pragmático: o que temos, tendo em conta a realidade (ex: não há service bus, regras de governance internas) Implementável: o que temos de fazer, tipicamente por constraints temporais ou de custo SKILLSETS: The customer – a scrum house - wanted us to assume the role of Product Owner, for example.
- #12 Notion of progress: In sprint Measurement of speed every sprint Estimation of remaining work via velocity/story points estimation
- #13 Managers: Transaction Script pattern http://martinfowler.com/eaaCatalog/transactionScript.html Organizes business logic by procedures where each procedure handles a single request from the presentation. http://martinfowler.com/eaaCatalog/serviceLayer.html Service Layer - Defines an application's boundary with a layer of services that establishes a set of available operations and coordinates the application's response in each operation.
- #14 http://en.wikipedia.org/wiki/SOLID_%28object-oriented_design%29 Single responsibility principle - a class should have only a single responsibility (i.e. only one potential change in the software's specification should be able to affect the specification of the class). Martin defines a responsibility as a reason to change, and concludes that a class or module should have one, and only one, reason to change. As an example, consider a module that compiles and prints a report. Imagine such a module can be changed for two reasons. First, the content of the report could change. Second, the format of the report could change. These two things change for very different causes; one substantive, and one cosmetic. The single responsibility principle says that these two aspects of the problem are really two separate responsibilities, and should therefore be in separate classes or modules. It would be a bad design to couple two things that change for different reasons at different times. Separation of concerns (SoC) is a design principle for separating a computer program into distinct sections, such that each section addresses a separate concern. A concern is a set of information that affects the code of a computer program. --- Open/closed principle - “software entities … should be open for extension, but closed for modification.” Liskov substitution principle - “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program.” See also design by contract. Interface segregation principle - “many client-specific interfaces are better than one general-purpose interface.” Dependency inversion principle - one should “Depend upon Abstractions. Do not depend upon concretions.” Dependency injection is one method of following this principle.
- #15 If you don’t have detailed logging, how do you diagnose problems? Wait for the repro? Look at the data? How useful are stack traces, really? Where’s the application context, the security context, etc.?
- #16 Also, obviously make sure your client instance is not null, or your logging will possibly cause an error.
- #17 No caso do impacto na performance, falar do caso do NxBre, que gera tanto logging que se torna inutilizável. A política de logging é um documento com até 2 páginas a dizer como usar os níveis, que tipo de logging é útil e não, dando exemplos, e como usar as classes para o realizar.
- #18 Pode haver discussão aqui. Reler bem e preparer o Holub.
- #20 Impacto no desempenho: do as little as possible and async if possible (ex: logging)
- #22 Remark that this also applies when your code that depends on ClassA intervenes.
- #23 Impacto no desempenho: do as little as possible and async if possible (ex: logging)
- #24 Juval Lowy – one order of greatness more time, if you have to access the network.
- #27 Dar o exemplo do AS. Que corre os testes todos os dias. Princípio broken window of not letting them rot.
- #32 Tive inúmeras situações destas: Pequenos compromissos que se tornam grandes compromissos (a história da sra bimbo) Pessoas que não aceitaram opções alternativas para não dar o braço a torcer, e acabaram por ser emocionais Exemplo: - Alteração de dtsx para ficheiro, após envolvimento de arquitecto do banco
- #33 Examples: Boss da BE não querer perder a face (ou o RB corar durante a reunião acesa) - Exemplo do Luis Alves, de lhe pedir ajuda em vez de fazer a sugestão que tinha em mente, para algo que ele não iria gostar (acesso directo a SP em vez de web services)
- #34 Clarificar que neste caso a responsabilidade funcional não ficou do nosso lado. Ex: Haircut? Pagaré? Letra? We had a project recently for which 80-90% of the features are critical to be in the Minimum Viable Product. And we have one at the moment where they say they use scrum, but with 3 to 4 month sprints. Because “they can’t stop business users from asking things”.
- #35 Business context is sometimes missing. > letras, pagarés, haircuts, descobertos, MLPs, LOs, … > lots of banking-specific terms. Follow old ways – if people are used to winforms apps, they’ll design screens that look and interact like winforms apps. Single user requests – triângulo do filtro “pequeno demais” – which was the same size as the one in Excel!
- #36 E.g., Maestro, Chef, Juiz.
- #37 They say the brain uses up 20% of your energy intake (RMR, Resting Metabolic Intake, while representing only 2% of you bodies’ mass.
- #39 Explain what we did: observe training, sitting next to users passively, with a tablet and an excel file. These tests were imensely useful. We uncovered problems, we identified missing or interesting new functionality. Examples: auto-complete a proposal (if current credit = x, new credit limit = x) The 1st comment in the excel file from na older person The observation with the Young woman who used the scroll wheel to zoom out automatically The camera videos take a huge amount of time to process. And if the client trusts you, they are not really needed.
- #40 https://quickgraph.codeplex.com/
- #42 Self explanatory. I personally prefer to create a mental model and then discuss, but theycan give you ideas, and discussion makes the architecture more solid. EXPAND THIS COMMENT They are your peers, trust tem. “Take care of her and she’ll take care of you”.
- #43 I’m a visual thinker, I need whiteboards, pencils, sheets of paper. A whiteboard was one of our requests to the client when the project started, and it was granted.
- #44 Sistematizar número mínimo de campos: header com estado/id proposta/cliente/ambiente.
- #46 UI responding to changes detected in database or service-layer. Ex: a report generated asynchronously is ready.
- #47 Posso dar exemplos aqui: LogExpert. SSMS. ServiceStack’s library. NewtonSoft JSON, tool to export certificates, powershell, …
- #50 Posso falar dos bonecos, posso falar dos gráficos, do shortcut que releva os detalhes das operações para diagnosticar problemas nos dados <- provavelmente o mais importante.
- #51 I find this last sentence truly inspiring, and not for the religious theme
- #52 Logo font: Monserrat Regular Logo color: #9a8e5e (154,142,94)