• Like
Refactoring toward deeper insight   java forum
Upcoming SlideShare
Loading in...5
×

Refactoring toward deeper insight java forum

  • 731 views
Uploaded on

Refactoring Toward Deeper Insight …

Refactoring Toward Deeper Insight
DDD Findings in Batch Processing, a Case Study When I was introduced to the Domain-Driven Design (DDD) approach close to ten years ago, it provided me with some of the missing pieces I needed to implement Object-Orientation in an effective way. And over the years I've been coming back to Eric Evans' very rich and deep book many times to discover something new to help me design better software - thinking tools and practical design advice, in the small and in the large. Over the years Object-Orientation has become less important to me, but DDD is still my default starting point when I am helping teams to refactor their architectures and take control over their code bases. Many teams have already made attempts to implement DDD, but very often they don't get the effects they were hoping for. It turns out that DDD is hard to get right. In a current project I have been involved in yet another effort to implement DDD on a legacy code base. And I have made some interesting findings. Batch processing scenarios opened up my eyes to some intrinsic problems with the DDD approach. Issues that have been have been nagging me over the years became very clear. And yet again I managed to gain deeper insight in the DDD approach and come up with some quite interesting ways to implement it.
Andreas Brink, factor10

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
731
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Disclaimer!!! .NET not Java Work In Progress…Refactoring Toward Deeper InsightDDD Findings In Batch Processing A Case Study Andreas Brink, factor10
  • 2. Danica Pension• Core financial business system.• ASP.NET / RDBMS / Integrations• Generations of technologies / architectures.• Lots of financial business rules.• Lots of Batch Processing.• Mission: Taking Control of the Code Base – DRY, Understandability, Testability, Automation•  DDD/DM + ORM fits well!
  • 3. My View of the DDD Toolbox Model Design & Impl.Philosophy & Principles • Entity• Ubiquitous Language • Aggregate• Model Driven • Value Object• Declarativity • Respository• Distilling The Model • Service • Specifiction• Breaktrhough • Side-Effect Free Functions• … • … Strategic Design • Bounded Contexts • Responsibilty Layers • …
  • 4. Domain Model Pattern (DM)Basic Building Blocks Supple Design• Entity, • Assertions• Aggregate • Side-effect free• Value Object Functions• Specification • Standalone Classes• Factory • Closure of Operations Design Sweet-Spot • Understanding & Communication • Testability, Executable Specifications But, not like the Office Object Model… • Must scale • Not one single file on disk
  • 5. Implementability SomeLayer… Repository ORM Service• Object Navigation does not scale  Repositories• DM does not scale well with composition & coupling Services• Problem Solved !?!?
  • 6. Implementation Mess Repository Some ORMScenario… Service• Less focus on Domain Model• Services – The Ultimate Loophole – Touches big parts of the system – horizontally, vertically – Side Effects Understandability & Testing Problems• Decentralized Flow Control & Data Access…  Global Optimization & Generic Processing hard or impossible  Performance Problems
  • 7. Why DDD is Hard Repository Some ORMScenario… Service• Model Design is hard to begin with – OO Expertise is still quite rare • Have to be a Design Patterns / ORM / Architecture Expert• Fail to get an Easily Consumable & Testable Model• Same old Procedural Service Soup (+ some entities…)
  • 8. My DDD Challenge• Reclaiming the Domain Model – Easy Reasoning, Consumption & Testing• REAL Separation of Concerns – Not just a complex web of objects and method calls behind superficially simple interfaces• And with Batch Processing in the mix… IS THIS POSSIBLE??
  • 9. Batch in the mix…• ”ORM is not for Batch, use the right tool…”• DDD/ORM vs Stored Procedures• Service Chattiness  Performance Problem• Batch becomes a domain service in itself – Business Workflow as a mini program – Hard to decompose/compose without Service – I want the business rules in the Domain Model…
  • 10. Billing Batch – Pseudo Codeforeach (PlanAgreement planAgreement in GetPlanAgreements()){ Agreement agreement = GetAgreement(planAgreement.Id); foreach (PlanCategory planCategory in GetPlanCategories(planAgreement.Id)) { PremiumTable premiumTable = GetPremiumTable(planCategory.Id); foreach (PlanInsurance planInsurance in GetPlanInsurances(planCategory.Id)) { Insurance insurance = GetInsurance(planInsurance.InsuranceNumber); InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber); AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber); double premium = CalculatePremium(planAgreement, agreement, planCategory, premiumTable, planInsurance, insurance); List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account); ... ... ... } }}
  • 11. Billing Batch – Levels Agreement• Agreement• PlanAgreement Category • PlanCategory • PremiumTable Insurance • PlanInsuranceHistory Misc… • LatestPlanInsurance • InsuranceHistory • … • LatestInsurance • … • InsuranceAccount • AdviceHistory • …
  • 12. Batch Observations• High input/output entity ratio – 11 Entity Types as Input – Often Complex – 2 as Output (1 Create, 1 Update) – Simple – Simple State Semantics – Opportunities for Caching – (Responsibility Layers Analysis…)• Data is centered around a few central business keys.Potential for generalizing / streamlining the batch processing pipeline??
  • 13. Billing Batch – Loops FlattenedPlanAgreement planAgreement = null;Agreement agreement = null;PlanCategory planCategory = null;PremiumTable premiumTable = null;foreach (PlanInsurance planInsurance in GetPlanInsurances()) { if (planInsurance.PlanAgreement != planAgreement) { planAgreement = planInsurance.PlanAgreement; agreement = GetAgreement(planAgreement.Id); } if (planInsurance.PlanCategory != planCategory) { planCategory = planInsurance.PlanCategory; premiumTable = GetPremiumTable(planCategory.Id); } Insurance insurance = GetInsurance(planInsurance.InsuranceNumber); InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber); AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber); double premium = CalculatePremium(planAgreement, agreement, planCategory, premiumTable, planInsurance, insurance); List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account); ...}
  • 14. Billing Batch – Levels Flattened Agreement Category Insurance Misc… Agreement Category Insurance Misc… Agreement Category Insurance Misc…• Agreement • PlanCategory • PlanInsuranceHistory • …• PlanAgreement • PremiumTable • LatestPlanInsurance • … • InsuranceHistory • LatestInsurance • InsuranceAccount • AdviceHistory
  • 15. Billing Batch – Entity Level Keys Agreement Category Insurance Level Level Level Key Key Key Agreement Category Insurance Misc… Agreement Category Insurance Misc… Agreement Category Insurance Misc…• Agreement • PlanCategory • PlanInsuranceHistory • …• PlanAgreement • PremiumTable • LatestPlanInsurance • … • InsuranceHistory • LatestInsurance • InsuranceAccount • AdviceHistory
  • 16. Billing Batch – Generic Cursor Style Master Keys EntitiesAgreement Level Plan Agreement Agreement Insurance HistoryCategory Level Advice History Premium Table Plan CategoryInsurance Level Plan Insurance History … …• Cursor Semantics• A Set of Master Keys Drives the Cursor• Entities Associated with Keys in Master• Each Row Contains Entities for a Unit-Of-Work
  • 17. Entity Level Keys Level Keys Plan Insurance Agreement Level = 4501 Agreement ID = 4501 Category Level = 78  Category ID = 78Insurance Level = ”56076” ID = ”56076”• Map of EntityLevel & Values – Dictionary<EntityLevel, object>• Or derived from Entity Properties
  • 18. The Entity Level Abstractionclass AgreementLevel : EntityLevel {}class CategoryLevel : EntityLevel {}class InsuranceLevel : EntityLevel {}public class PlanAgreement{ [Level(typeof(AgreementLevel), IdentityType.Full)] public int Id;}
  • 19. Entity Cursor: Master + Entitiesvoid Initialize(){ var cursor = EntityCursor.For(SessionFactory, MetaData); // MASTER: IEnumerable<object[]> OR IEnumerable<TEntity> cursor.Master = GetMyMaster(); cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel()); cursor.Add(Query.For<PlanAgreement>()); // ADD MORE ENTITIES TO THE CURSOR... while (cursor.MoveNext()) { var currentPlanAgreement = cursor.Get<PlanAgreement>(); // PROCESS EACH ROW IN THE CURSOR... }}
  • 20. IoC Style + Syntactic Sugarclass MyBatch : BaseBatch{ PlanAgreement planAgreement; EntityLevel[] Levels() { return ... } object[] Master() { return ... } void Initialize() { // Query Defintions that are not simple // Query.For<MyEntity>() Add<PlanAgreement>() .Where(pa => pa.Foo != null); } void ProcessRow() { var foo = this.planAgreement.Foo ... // PROCESS THE ROW... }}
  • 21. Row ProcessingMaster Keys Agreement InsuranceKey 1: Agreement Id ChunkSize: 2 ChunkSize: 2Key 2: Insurance No(1, “InNo-1")(1, “InNo-2")(1, “InNo-3")(2, “InNo-4")(2, “InNo-5")(3, “InNo-6") ...(n, “InNo-n")
  • 22. Row Processing  Chunked Data FetchMaster Keys Agreement InsuranceKey 1: Agreement Id ChunkSize: 2 ChunkSize: 2Key 2: Insurance No(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")(1, “InNo-2") -||- Insurance(1, “InNo-2")(1, “InNo-3") -||-(2, “InNo-4") Agreement(2) Query<Insurance>()(2, “InNo-5") -||- .Where ...(3, “InNo-6") ... Query<Agreement>()(n, “InNo-n") .Where(a => a.Id) .IsIn(1, 2) • Entities are fetched in Chunks • Multiple chunk queries executed in one DB round-trip. • NHibernate MultiCriteria (or Futures).
  • 23. Row Processing  IndexingMaster Keys Agreement InsuranceKey 1: Agreement ChunkSize: 2 ChunkSize: 2Key 2: Insurance(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")(1, “InNo-2") Agreement(2) Insurance(1, “InNo-2")(1, “InNo-3") Insurance(1, “InNo-3")(2, “InNo-4") Insurance(2, “InNo-4")(2, “InNo-5") Insurance(2, “InNo-5")(3, “InNo-6") Agreement(3) Insurance(3, “InNo-6") ... ...(n, “InNo-n")• Each entity is indexed with the identifying level key(s).• Entities in chunks synced with key for current row as the cursor proceeds forward.
  • 24. Entity Groupingclass InsuranceHistory : GroupingEntity<Insurance>{ static readonly Grouping Grouping = Grouping.For<Insurance>() .By(i => i.AgreementId) .By(i => i.InsuranceNumber); public InsuranceHistory(IList<Insurance> values) { ... }}Cursor.Add<PlanInsuranceHistory>();Cursor.Add<PlanInsuranceHistory, PlanInsurance>() .Where(...); // OK to override filter?? • Groups as a 1st class modeling concept • Enriching the Domain Model • “Virtual Aggregate Root” – Model Integrity • Declarative expression (By, Where, Load)
  • 25. Complex Grouping – PremiumTable Row 10-22 23-30 31-45 Interval0-20 100 120 135 Value20-40 110 130 15040-65 130 160 190 Column Interval • Rich Model Abstraction • Complex data structure with lookup semantics • No natural aggregate root • Not cacheable in NHibernate session • Fits well as a GroupingEntity
  • 26. QueryingConceptual API:Cursor.Add(Query entityProvider)Query.For<PlanInsurance>() .Where(insurance => insurance.IsActive) .Load(insurance => insurance.Category)Query.For<AdviceHistory>()Query.For(PremiumTable.ByAgreement) .IndexBy(table => table.TableId)• Filter per Entity – Cursor “Joins” using Shared Level Keys• ORM-semantics: Where, Load• Grouping Entity has query like qualities• Level Queries are statically defined query using Entity Levels Keys to construct underlying ORM query (yes, coupling)
  • 27. Versioningpublic class PlanInsurance{ [Level(typeof(AgreementLevel), IdentityType.Partial)] public int AgreementId; [Level(typeof(InsuranceLevel), IdentityType.Partial)] public string InsuranceNumber; [VersionLevel(typeof(PlanInsurance), IdentityType.Partial)] public int Version;} • Core to many business domains • Has its own set of semantics • Common in Groups – Latest<Insurance> vs InsuranceHistory • Implemented in different ways in the DB • Expressed declaratively • Uniform Query Semantics
  • 28. What About The Services?void ProcessRow(){ ... var premiumService = new PremiumService { PlanAgreement = Cursor.Get<PlanAgreement>(), PlanInsurance = Cursor.Get<PlanInsurance>(), Insurance = Cursor.Get<Insurance>(), Insured = Cursor.Get<Person>(), PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(), PremiumTable = Cursor.Get<PremiumTable>(), RiskTable = Cursor.Get<RiskTable>() }; var premium = premiumService.CalculatePremium(advicePeriod); ...} • Service has pure calculation responsibility • Dependencies are injected by client • Coupling…? Boilerplate Smell…?
  • 29. Conclusions• Data Access Abstraction with Power & Ease of Use• Declarative & Composable Entity Pipeline• Minimizes DB Round-trips; Favors Eager Loading• Repositories Become Redundant• No More Unconstrained Services – “Calculators” / …???• Richer Domain Model – Less Supporting Objects, More Domain Related Objects• DDD/ORM + Legacy DB == True• Composite DB Key Thinking Essential to the Solution• Patching the DB Model with Entity Level Abstraction…• What’s Next? – Lots of Low Hanging Fruit… TOWARDS AN EXECUTABLE ARCHITECTURE…???
  • 30. What’s Next? – Entity Injection Cursor.Add<PremiumCalculator>(); void ProcessRow() { ... var calculator = Get<PremiumCalculator>(); var premium = calculator.Calculate(advicePeriod); ... }• Cursor can inject entity dependencies automatically• Calculators dependencies can be inferred and added to cursor automatically•  ”Calculator” define Cursor Entities Implicitly
  • 31. What’s Next? – Stateful Calculators?class PremiumCalculator class PremiumCalculation{ { ... ... double CalculatePremium(...) {} double Premium; ... ...} } • What if we treated a calculation as a stateful object? • Calculations become data flows through the system • Stateful Objects as the Uniform Expression – Simplifies declarative programming • Captures Multiple / Intermediate Calculation Results • Can be bound to a UI • Additional state in the cursor – UI could add presentation model/wrapper to the cursor
  • 32. What’s Next? – Entity Pipeline class BillingCalculation : EntityPipeline { void Initialize() { Add<PlanAgreement>(); ... } }var monthlyBatch = new BillingCalculation();monthlyBatch.Master = GetMasterForMonthlyBatch();monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);monthlyBatch.Execute();var singleInstance = new BillingCalculation();singleInstance.Master = new object[]{ 24, "InNo-1"};singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);singleInstance.Execute();var nextUIPage = new BillingCalculation();nextUIPage.Add<MyUIModel>();nextUIPage.Master = GetMasterForNextPage();myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())
  • 33. What’s Next? – New Data Providers• File Processing for Data Imports – Prototyped batch framework• Document Based Persistence – Premium Table for example• Hybrid Persistence – Serialized object graphs in SQLServer• SOA Integrations – Loosely Coupled Bounded Contexts• Parallel data fetch – Multiple DBs / Data Services
  • 34. What’s Next? – Business Events• Entity Processing Pipeline seems to be a good environment for triggering and/or handling business events based on persistence events.• Poor man’s Business Events!?!?
  • 35. What’s Next? – Greenfield• Search the Core Domain/Application Semantics – Built-in Versioning from the start e.g. – Semantic Storage…• Streamline – Uniform Expression – Semantics – Patterns• Be Opinionted – Constraints are Liberating•  Executable Architecture
  • 36. Thanks For Listening!!! Questions?