Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monolith to Microservice, Waterfall to agile – Success with Camunda

977 views

Published on

Friedbert Samland, Project Manager IT Application Deutsche Telekom IT and Willm Tüting, Managing Director conology GmbH, describe the transformation of a monolith application to a Microservice-based Camunda inherited architecture. Coming from a technical point of view, with BPEL we had to abstract to a business-oriented process flow using BPMN. Just changing a few letters? Far from it!

Setting the scene of the challenges of Telekom IT as a large IT company changing from an organizational implemented waterfall development process, into a software development company facilitating agile principles. Striving the raising complexity due to an agile transformation as well as the utilization of the Cloud and DevOps.

We will highlight the Pros and Cons and how to tackle the challenges of a big transformation initiative within the current teams. We will take a journey through several viewpoints where we show you challenges and advantages we met on our way so far.

Published in: Technology
  • Be the first to comment

Monolith to Microservice, Waterfall to agile – Success with Camunda

  1. 1. CAMUNDACON 2019 DEUTSCH TELEKOM IT GMBH LIVE IS FOR SHARING © Deutsche Telekom IT GmbH– 2019 July
  2. 2. In this presentation, we will beyour presenter to show youhow we split up our monolith into small pieces of microservices FRIEDBERT SAMLAND (PROJECTMANAGER TELEKOM IT) WILLM TÜTING (MANAGING DIRECTOR CONOLOGY GMBH) HELLO & WELCOME!
  3. 3. PRESENTATION AGENDA Companyprofile: TelekomIT GmbH ShortHistory How CamundaFits intoMicroservices From MonotlithtoMicroservice Whatwebuild 01 02 04 05 03
  4. 4. CAMUNDACON 2019 4DEUTSCHE TELEKOM IT GMBH WHO WE ARE AND WHAT WE DO We areresponsibleforthe design,development andoperationof allits own andtransferredIT systemsto supportbusinessprocessesat DeutscheTelekomAG (in particularTelekom Deutschland,T-SystemsandGHS) Thepeopleat TelekomIT dealwith hundredsof technologies,vendorsand serviceproviders WHAT WE DO TelekomIT GmbH isthe internalIT serviceprovider of DeutscheTelekom AG. DeutscheTelekomIT hasaround9,700employees where6,500 are locatedin Germanyand approximately3,200 employeesareworking at T- Mobileand T-Systemsin EuropeandBrazil. Our totalbudgetisaround€ 1.9 billion. WHO WE ARE
  5. 5. CAMUNDACON 2019 5DEUTSCHE TELEKOM IT GMBH WHAT WE OFFER AT TELEKOM IT TelekomIToffersa largevariety of solutionsforourcustomersinsidethe DeutscheTelekomAG E2E responsibilityforRetailandWholesalecustomersofTelekomDeutschland.Costumer Solution’smainpartsare CRM, CustomerServiceandTechnicalServiceforFixed-Net, Mobileand IPproducts. CUSTOMER SOLUTIONS E2E responsibilityforITsystemsin theTechnicalServiceof TelekomDeutschland.This incorporatesITsolutionsforplanning,buildingandoperatingtelecommunication networks. TECHNICAL SOLUTION ImplementsandoperatesITsolutionsforthe businessconsumersdepartmentof Telekom Deutschland.Productslike PBX in thecloud aretheir business BUSINESS SOLUTIONS Responsibilityforplanningthe completeIT roadmapandoperationprocessesinTelekom IT. DIGITAL TRANSFORMATION AND OPERATION
  6. 6. CAMUNDACON 2019 6DEUTSCHE TELEKOM IT GMBH WHAT WE DO AT CUSTOMER SOLUTIONS? CustomerSolutionsisresponsibleforall productsandserviceswhich interactwith thedifferenttypesof customers TheMagentaServiceApp isone of ourfavoriteproducts. APP PROGRAMMING Developingsoftwareforbuildingandbilling individualcarrier fixed line connections. WHOLESALE INDIVIDUAL MARKET Developingandmaintainingsystemsto helpourcustomerswith their problemsandwishes.Self-servicein caseof troublewith their fixed lineor assistancetodiagnoseerrorswith thetelephoneline. SERVICE AND ASSURANCE Developingandoperatingthe WITAplatform for telecommunicationcarriers in Germany. WHOLESALE MASSMARKET Managementof approx.20 millionof direct customerswith fixed lineor mobileproducts. CRM-SYSTEM FOR RETAIL CUSTOMERS
  7. 7. CAMUNDACON 2019 7DEUTSCHE TELEKOM IT GMBH A WHOLESALE USECASE FROM FULFILLMENT ProviderchangeTelekomto 1&1 inshort STEP 01 End-customerwantsto changeits fixed line fromTelekom to 1&1 STEP 02 Newcarrier1&1gets theorder from end-customer STEP 04 WITA terminatescontractwith Telekom andorders theline for 1&1 STEP 03 WITA getstheorder forprovider changefrom 1&1 STEP 06 End-customer hasnewfixed line connection at1&1 STEP 05 1&1gets thefulfillment messagefrom Telekom andinformsits new end- customer
  8. 8. “DOIT FASTMEANS, DO IT AUTOMATICALLY” OUR PROCESSING Thedifferentwholesaleusecasesare processedin a standardizedway. Get wholesaleordersin a digital wayto processthem immediately. ON TIME Takecare of the orderin everysituationspeciallyin exceptionalsituations RESPONSIBILITY Runordersfullyautomatedat nearly95% (actual85%) EFFICIENCY
  9. 9. CAMUNDACON 2019 9DEUTSCHE TELEKOM IT GMBH A SHORT TIMELINE OF WHOLESALE BUSINESS Wholesalebusinessfrom2007until today 20082007 20182017 Start processingTAL orders with Vodafone and1&1 PRODUCTION Developmentof a processorientedand automatedway for processingwholesale orders DEVELOPMENT Inflexibilityin developmentand deploymentdecreased by firstagileattempts CHANGE OUR WAY Telekomstartedthe GIGABIT initiativeby buildinga new businesssolutionforordering FTTXlines MSA STARTED
  10. 10. CAMUNDACON 2019 10DEUTSCHE TELEKOM IT GMBH THE AS-IS – UP TO THE MARKET? FAR FROM IT! Themajorobstacleswefacedin2017 Timeto Market 12+month Large Releases 1000+PT Media gaps Alot ofmanualtasks Outdated technology 10+yearsold VendorLock High costs Hardware dependnt Limited &slow Waterfall 3+monthperphase Regression runtime 2+days Inflexible 5+dayssetuptime Compliance Decreased quality
  11. 11. CAMUNDACON 2019 11DEUTSCHE TELEKOM IT GMBH THE FIRST STEP – DID IT HELP? SADLY NOT! Waterfalltoagile... Requirement 3 month allworkstartsatonce 3 month 3 month Development allworkend atthesametime
  12. 12. CAMUNDACON 2019 12DEUTSCHE TELEKOM IT GMBH THE FIRST STEP – DID IT HELP? SADLY NOT! ...is notenoughif youdon‘tadaptit end-2-end! HeterogenousITheterogenousIT &processes Largerequirementsstill Largerequirementsstill 3 month3 month 3 month Development Lesson 01 Helpsto overcome longcycle-timesforsmall requirements&bugfixes Partially agile is nonsense Requirement
  13. 13. CAMUNDACON 2019 13DEUTSCHE TELEKOM IT GMBH THE TO-BE – JUST BUZZWORDS? FAR FROM IT! Fourmajordriverswe identified... but we were lackingthe powerto changethem 01 02 03 04 CLOUD Thepower of theclouddramatically reducesruntimesandenablesastaged fine-gaineddeliveryapproach. DEVOPS Automation& Self-Serviceaspartsof ourDevOpsphilosophyarethe keysto quality& speed. SAFE Introducinga new organizationwith shortercycle-timesend-to-endenables flexibilityandspeed. MICROSERVICES An approachto partitionthe monolithand enablecross-functionalwork.
  14. 14. CAMUNDACON 2019 14DEUTSCHE TELEKOM IT GMBH THE TRANSITION - LUCKY US Light at theendof thetunnel:A newproductaroseonthe horizon... OUR SUCCESS FORMULA "TheForce" waswith us! THE RIGHT REQUIREMENT Thenew product'FTTH' shouldchangethe wholecompany THE RIGHT DIMENSIONS All dimensionscouldbechanged: Specification,implementation,testing,etc. THE RIGHT STAKEHOLDERS Stakeholderswere onoursideand approvedour new approachbasedona new architecture
  15. 15. CAMUNDACON 2019 15DEUTSCHE TELEKOM IT GMBH SAFE Agile @ scale Lesson 02 Agile isn’tbottom-upnortop-down,successneeds both! From 0 to agile E2Eis nonsense
  16. 16. CAMUNDACON 2019 16DEUTSCHE TELEKOM IT GMBH CLOUD - NO READY MADE SOLUTIONS Cloudisn't easy,it's a newparadigm, it addscomplexity, butit ... FACILITATES STAGING New stagesarejusta matter of need& costs, but notset-uptime SCALES WITH THE NEEDS If youneedmore “hardware”youjust provisionit! REDUCES DEPENDENCIES Youdon’tneedto care thatmuch about maintenancein developmentanymore. ALLOWS CONSTANT UPDATES Youcan constantlystayup-to-datewith your frameworks BRINGS ALONG A LOT OF BENEFITS Reducedf.e. thestandardmaintenancetaskat a significant level. ENABLES SELF-SERVICE Developerscanperformall theirtasks independentof a centralizedfunction Lesson 03 Benefits take time and the way bears a lot of pitfalls Cloud has a high learning curve
  17. 17. CAMUNDACON 2019 17DEUTSCHE TELEKOM IT GMBH DEVOPS - EMBEDDED ENABLER? Startingwith DevOps is a hugemindchange forDev & Ops people FRAMEWORK MANAGEMENT INFRASTRUCTURE AS CODE CONTINUOUS EVOLUTION COMPLEX TOOL STACK HIGH DEGREE OF AUTOMATION HANDS OFF! AVOID ‘LATEST’ CHANGE AS DEFAULT AVOID TO DO ALL- AT-ONCE THINK BIG, START SMALL Lesson 04 Don’t lose track. Keep focused on benefits instead of beauty. DevOps is a huge topic
  18. 18. CAMUNDACON 2019 18DEUTSCHE TELEKOM IT GMBH MICROSERVICES – PROMISES AND CHALLENGES Thehopeswere high, theriskstoo? PROMISE 03 typicalinterfacechangesof partnersystemsshould onlyaffectfew microservicesPROMISE 02 with increasingloadit shouldbe possibleto only scaleup PROMISE 01 typicalbusiness requirementsshouldonly affectfew microservices CHALLENGE 05 complexcentral mechanismsshouldbe developedonce and resused CHALLENGE 04 the numberof microservicesshouldbe neithertoo high nor too low BASIC FACTS Next to promisesand challengeswe hadsomebasicfacts thatmade uscautious. 1.5 MILLION Lines of Code 400+ WEBSERVICES Components 80+ OPERATIONS USED Interfaces
  19. 19. “FIRSTPRODUCTIVE, VERSION AFTER8 MONTHS” OUR NEW DEVELOPMENT We identifiedthreemajorchanges. Changedevelopmentfrom BPEL to Java andmodusoperandi from waterfallto agile SPEED UP Changefrom skill-basedtailoredteamsto international cross-functionalteams CROSS-FUNCTIONAL TEAMS Increasedevelopmentefficiencyby using Camunda, SPRINGand otherstate-oftheart technologies. EFFICIENCY
  20. 20. CAMUNDACON 2019 20DEUTSCHE TELEKOM IT GMBH MICROSERVICES: ARCHITECTURE OF THE MONOLITH Frommonolithtomicroservicearchitecture As-Is-Architecture BPM-Engine (Oracle SOA Suite / BPEL) Fulfillment („KP9“) SOAServices+Interfaces Assurance („KP10“) Preponed PreOrder („PPO“) Order State Manageme nt („ASV“) tight coupling Microservice PreOrder Microservice PreOrder Microservice Order Microservice Order Microservice Assurance Microservice Assurance Microservice Business Control Microservice Business Control Target Architecture REST API MessageBroker … REST API REST API REST API REST API REST API REST API REST API … … … loose coupling Where is Camunda? Stay tuned!
  21. 21. CAMUNDACON 2019 21DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Our 5-Stepapproach DATA 01
  22. 22. CAMUNDACON 2019 22DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Step1: Categorizedata 22 Order D1:Datareceived fromfrontends Comments Notifications ContactPersons Dates D3:Dataexchangedwithbackends ... Addresses Contracts Assets ... Reposonsesof Fullfillment Telephonenumber ... ReservationResults ... Customerdata Notifications Backend System A Backend System B Backend System c ... Backend System D Backend System D ... ... Backend System F,G Backend System F,H ... Backend System E ... ... --- ... D0:Businessconfiguration Errormatrix ... Calendar Systemdatatables D2:Datagenerated/calculatedbythesystem Timestamp Ordercount state/Statehistory ... Errors connectedOrders ... Products System messages Manualsteps Orderpositions
  23. 23. CAMUNDACON 2019 23DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Our 5-Stepapproach DATA 01 02 FUNCTIONALITY
  24. 24. CAMUNDACON 2019 24DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Step2: Categorizefunctionality 24 F2) End-to-endbusinessprocesses  Fulfilmentprocesses,e.g. provision,servicechange,providerchangeetc.  runtimevariesstrongly between„very short/online“(afewsecondse.g. inquiries)forsynchronousprocessesand„verylong“(upto2years,e.g. orders)forasynchronousprocesses F4) Domainspecificprocesses  Reusableprocesssegmentsthatarecalledbytheend-to-endbusiness processes  createrequeststobackendsystems  evaluateresponsesfrombackendsystems  runtimeis normallyshortto medium(afewhours),butcanalsobehigher forprocesssegmentsthatwaitforresponsesfrombackendsystems F3) Provideand/ orcalculatedata  Calculation-andcountingfunctions(e.g. calculationofdatesand countersforplanningagreements)  Informationfunctions(e.g. „readorder“and„searchorder“)  Configurebusinessparameters(e.g. errorassessmentmatrix,notification codetable,calendar)  runtimeis normally“veryshort / online“(afewseconds) F1) Complexcentralmechanisms  Selectionofthecorrectend-to-endprocessforanincomingorder („processframeworkfunctionality“)  Coordinationofmultipleorders(e.g. “orderbracket”,cancellation, rescheduling,waitingfor openorders)  Coordinationofdifferentbusinessclusters  ExceptionHandling+Portal-Stop  Timeout-Monitoring
  25. 25. CAMUNDACON 2019 25DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Our 5-Stepapproach DATA 01 02 03 CATEGORIES FUNCTIONALITY
  26. 26. CAMUNDACON 2019 26DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Step3: Defineandassignmicroservicecategories 26 Category A: Business Process Microservices Microservicesinthiscategory  containthelogictocontrolcomplex central mechanisms(functionallevel F1)  realiseshortorlongrunningend-to-endprocesses (functional levelF2)  donotmanage(persistent)data,  choose thecorrectend-to-endprocessforan incomingorder  delegateworktoprocesssegments( Domain microservices)  informabouttheprocessstatusandthestatushistory Category B: DataMicroservices Microservicesinthiscategory  managethebusinessconfiguration (datacategoryD0)  managedatareceivedfromfrontends (datacategoryD1)  managedata,thathasbeencreatedand/orcalculated byothers(datacategoryD2)  expose informationandcalculationfunctionsfor thesedatacategories(functionallevelF3)  implementa datadeletionstrategy(deletedataas soonasitisn’tneededanymore) Category C:Domain Microservices Microservicesinthiscategory  managedataexchangedwithbackends (datacategoryD3)  expose informationandcalculationfunctionsfor thesedatacategories(functionallevelF3)  implementa datadeletionstrategy(deletedataas soonasitisn’tneededanymore)  realiseshortrunningprocesssegments(functional levelF4)  thatcreaterequeststobackendsystems  thatevaluate(intermediaryand final)responses frombackendsystems
  27. 27. CAMUNDACON 2019 27DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Our 5-Stepapproach DATA 01 02 03 04 CATEGORIES COMMUNICATIONFUNCTIONALITY
  28. 28. CAMUNDACON 2019 28DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Step4: Designcommunicationof microservicecategories Backend(s) Frontend(s) Category B „Data Microservices“ API API synchronous asynchronous call businessprocess callbackend/ publish data receive response calldomainspecific MS signalsuccessful completionorerror startbusiness process reador writedata readdata reador writedata readdata readdata Category C „Domain Microservices“ reador writedata publish data determineprocess status(+history) Category A „BusinessProcess Microservices“ API API API
  29. 29. CAMUNDACON 2019 29DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Our 5-Stepapproach DATA 01 02 03 04 CATEGORIES MICROSERVICES COMMUNICATIONFUNCTIONALITY 05
  30. 30. CAMUNDACON 2019 30DEUTSCHE TELEKOM IT GMBH MICROSERVICES: ARCHITECTURE OF THE MONOLITH Orchestrationvs. Choreographyinspiredby TheMicroservicesWorkflowAutomationCheatSheet from BerndRücker Blog Choreography MessageBroker Microservice 1 Microservice 2 Microservice 3 Microservice n … Orchestration BPM Engine Micro- Servic e1 Micro- Servic e2 Micro- Servic e3 Micro- Servic en … Our solution: „choreographed orchestration“ MessageBroker Micro- Service 1 Micro- Service 2 Micro- Service 3 BPM Engine BPM Engine BPM Engine Micro- Service n … BPM engine is a central manager  monolithic! microservices are loosely coupled, but where are the processes? orchestratio n choreograph y not every microservice needs a BPM engine central process monitoring and management
  31. 31. CAMUNDACON 2019 31DEUTSCHE TELEKOM IT GMBH MICROSERVICES: PARTITIONING OF THE MONOLITH Step4: Designcommunicationof microservicecategories Category A „BusinessProcess Microservices“ Backend(s) Frontend(s) Category B „Data Microservices“ DataAPI Process API DataAPI synchronous asynchronous callbusiness process callbackend/ publish data receive response startdomain specificprocess Process API signalsuccessful completionor error reador writedata readdata reador writedata readdata readdata Category C „Domain Microservices“ Camunda Camunda reador writedata publish data startbusiness process determineprocess status(+history)
  32. 32. CAMUNDACON 2019 32DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA Camundaof coursewas not theonlytool, but... DEV OPS Test Monitor Integrate DeployPlan Code Build Operate RabbitMQ Swagger Spring Cloud Config
  33. 33. CAMUNDACON 2019 33DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA One of ourMajorBusinessProcessMS Domain MS
  34. 34. CAMUNDACON 2019 34DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA Communicationpatternbetweenmicroservices:Call a domainmicroservice Domain MS
  35. 35. CAMUNDACON 2019 35DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA Howa domainmicroservicecanwork Domain MS
  36. 36. CAMUNDACON 2019 36DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA Communicatonpatternbetweenmicroservices :Receiving response from thedomainmicroservice Domain MS
  37. 37. CAMUNDACON 2019 37DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHAT WE BUILD WITH CAMUNDA WhereCamundasupportsusstrongly Domain MS Noninterrupting scope for concurrent flows Interrupting scopes for specical situations Scoping
  38. 38. CAMUNDACON 2019 38DEUTSCHE TELEKOM IT GMBH MICROSERVICES: WHY CAMUNDA IS A GAMECHANGER ... Camunda wasa majordriver andenablerbecauseof: Business - IT Alignment Development Operations WHY CamundaMAKES THE DIFFERENCE 01 02 03 BUSINESS IT ALIGNMENT • HumanTasks • Designprocessesastheyrunlateron • Visualizecomplexlogic DEVELOPMENT • Samelanguagewith the business • Lightweight engine • Native JAVA support OPERATIONS • TheCamunda Cockpit • TheToken-Concept • Supportsourcustom ExceptionHandling
  39. 39. CAMUNDACON 2019 39DEUTSCHE TELEKOM IT GMBH MICROSERVICES: TRANSITION OF THE MONOLITH AuditView Due to decoupling we decided to use separate DBs for every microservice using Camunda. The Cockpit isforoperations,butfor 3rdlevel onwardsit requireda differentsolution. SITUATION: ANALYSIS OF A BUSINESS PROCESS Caller Sequence: Creates an easy to follow overview of the flow of a businessprocess. Logging Insights: Got enriched with all underlying logging data relevant for 2nd, 3rd, Last-level-Support as well as the development teams. Performance Indications: Gives indications about runtime, bottlenecks,timeoutsandotherperformancerelatesissues.
  40. 40. CAMUNDACON 2019 40DEUTSCHE TELEKOM IT GMBH MICROSERVICES: TRANSITION OF THE MONOLITH Compliance-by-default Due to the global distributed teams, multiple vendors and sensitive data, a highly automated solution was required. The result is an architecture that enables compliance-by- default. SITUATION: GLOBALLY DISTRIBUTED TEAMS Clean Definition: While defining the Specifications the business analystsenrich theAPIswith complianceattributes. Automated pseudonymization: When required an automated tool createspseudonymizedsetsof therequireddata. Close Collaboration: Due to the short cycle times of providing consistentdatacollaborationisenhancedsignificantly.
  41. 41. CAMUNDACON 2019 41DEUTSCHE TELEKOM IT GMBH YOUR SPEAKERS Let‘sconnect! Willm Tüting conology GmbH +49 175 520 96 02 willm.tueting@conology.net Friedbert Samland Deutsche Telekom IT GmbH +49 6151 5819011 friedbert.samland@telekom.de
  42. 42. CAMUNDACON 2019 42DEUTSCHE TELEKOM IT GMBH WHAT'S-NEXT– ARE WE DONE? FAR FROM IT! Backup
  43. 43. CAMUNDACON 2019 43DEUTSCHE TELEKOM IT GMBH Classic Architecture Processframe Businessprocess Step A Step D Call Call AMQ P REST Microservice-Architecture (MSA) Activity Management Order- Date Bestand lesen (BV-W) ProcessData Management Proces s-Data Bestand lesen (BV-W) AssetManagement Asset- DataStep B Step C REST REST MICROSERVICES: TRANSITION OF THE MONOLITH TheCloud-Bridge Forntend(s) Order Cloud-Bridge Step B Step C Call Call Call Call BackendBBackendAD Call Call
  44. 44. CAMUNDACON 2019 44DEUTSCHE TELEKOM IT GMBH MICROSERVICES: TRANSITION OF THE MONOLITH EH Service ExceptionHandling Scope Task 1 an erroroccurs in Task 3Task 2 „Phase 1“ End EH Scope Start EH Scope ErrorHandler manual user chooses a reaction read reaction from FBM read mode from FBM automatic create listof possible reactions jump to chosen milestoneStart mode? catch error(errorcode) claimerror handling task (polling) Start (normal) End „Exception Handler“ (runs permanently) create error handling task handle “Reject” Camunda REST API complete error handling task Reject chosen? no yes End deleteToken (and thus cancel theEHTask) create Token 1 2 3 4a start EH process 4b 5 6 7a 7b x Other reaction was chosenReject waschosenShared call 8 8 8a 8b 9 10 Task 4 Task 5 „Phase 2“ xx External Task Client Reject 11 12 implicit milestone explicit milestone Implicit milestoneImplicit milestone Restart„Phase1“ was chosenas reaction Fehler- Bewertungs- Matrix

×