Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Death of the batch job

924 views

Published on

Lang lopende processen bestaan in de meeste systemen. Veelal geëvolueerd naar nachtelijke batch processen, welke langdurig draaien en grote hoeveelheden data aggregeren en wijzigen. Of verborgen in grote hoeveelheden classes die te lang staan te stampen op eigenlijk heel eenvoudige processen. Dat moet toch anders kunnen? We zullen kijken naar een process manager welke zowel de functionele als technische implementatie van lang lopende processen vereenvoudigt. Hoe we beter met de business kunnen communiceren over de te implementeren bedrijfsprocessen, hoe we nieuwe mogelijkheden vinden om deze processen binnen je systeem te implementeren en een eind te maken aan die verschrikkelijk batch processen.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Death of the batch job

  1. 1. Dennis van der Stelt all your batch jobs are belong to us Dennis van der Stelt Software Architect http://dennis.bloggingabout.net/ dennis@bloggingabout.net Particular Software engineer death of the batch job @dvdstelt #nservicebus
  2. 2. Dennis van der Stelt AGENDA
  3. 3. Dennis van der Stelt NServiceBus In Particular
  4. 4. Dennis van der Stelt NServiceBus It’s not like WCF, which does RPC But closer to WCF than to BizTalk
  5. 5. Dennis van der Stelt BUS TOPOLOGY
  6. 6. Dennis van der Stelt Messaging What is it and why do I need it?
  7. 7. Dennis van der Stelt SpatialTemporalPlatform coupling aspects
  8. 8. demo Quick NServiceBus Demo
  9. 9. Dennis van der Stelt NServiceBus Sagas A pattern by relation database community
  10. 10. Dennis van der Stelt
  11. 11. Dennis van der Stelt PROCESS MANAGER process managerInitiating message
  12. 12. demo NServiceBus Sagas
  13. 13. Dennis van der Stelt Sagas Recap “What have you done” – Within Temptation
  14. 14. Dennis van der Stelt HANDLING MESSAGES Behavior like normal message handlers class MySaga : IHandleMessages<MyMessage> { public void Handle(MyMessage message) { … } }
  15. 15. Dennis van der Stelt STARTING SAGAS Extends the default IHandleMessages<T> class MySaga : IAmStartedByMessages<MyMessage> { public void Handle(MyMessage message) { … } }
  16. 16. Dennis van der Stelt STORING STATE Extends the default IHandleMessages<T> class MySaga : Saga<MySagaData>, IAmStartedByMessages<MyMessage> { public void Handle(MyMessage message) { this.Saga.Data.MyStateProperty = Message.MyProperty; } }
  17. 17. Dennis van der Stelt CORRELATING MESSAGES TO SAGA INSTANCE class MySaga : Saga<MySagaData>, IAmStartedByMessages<MyMessage> { protected override void ConfigureHowToFindSaga(SagaPropertyMapper<MySagaData> mapper) { mapper.ConfigureMapping<MyMessage>(m => m.MyProperty).ToSaga(s => s.MyStateProperty); } public void Handle(MyMessage message) { this.Saga.Data.MyStateProperty = Message.MyProperty; } }
  18. 18. Dennis van der Stelt REQUESTING TIMEOUTS Reminders to the Saga itself class MySaga : Saga<MySagaData>, IAmStartedByMessages<MyMessage>, IHandleTimeouts<MyTimeout> { public void Handle(MyMessage message) { this.Saga.Data.MyStateProperty = Message.MyProperty; RequestTimeout<MyTimeout>(TimeSpan.FromSeconds(10)); } public void Timeout(MyTimeout state) { … } }
  19. 19. Dennis van der Stelt SENDING MESSAGES Reminders to the Saga itself class MySaga : Saga<MySagaData>, IAmStartedByMessages<MyMessage> { public void Handle(MyMessage message) { this.Saga.Data.MyStateProperty = Message.MyProperty; this.Bus.Send(new MyCommand()); this.Bus.Publish(new MyEvent()); } }
  20. 20. Dennis van der Stelt SAGA STATE Memento class MySagaData : IContainSagaData { public virtual Guid Id { get; set; } public virtual string Originator { get; set; } public virtual string OriginalMessageId { get; set; } [Unique] public virtual Guid MySagaId { get; set; } } ALTER TABLE [dbo].[MySagaData] ADD UNIQUE NONCLUSTERED ([MySagaId] ASC) ON [PRIMARY]
  21. 21. Dennis van der Stelt SAGA STATE Memento class MySagaData : IContainSagaData { public virtual Guid Id { get; set; } public virtual string Originator { get; set; } public virtual string OriginalMessageId { get; set; } [Unique] public virtual Guid MySagaId { get; set; } public virtual IList<Product> Products { get; set; } } public class Product { public virtual Guid ProductId { get; set; } }
  22. 22. Dennis van der Stelt SAGA STATE Memento class MySagaData : IContainSagaData { public virtual Guid Id { get; set; } public virtual string Originator { get; set; } public virtual string OriginalMessageId { get; set; } [Unique] public virtual Guid MySagaId { get; set; } public virtual IList<Product> Products { get; set; } } public class Product { public virtual Guid ProductId MyUniqueId { get; set; } }
  23. 23. Dennis van der Stelt Death to the batch job Because we don’t want to depend on operations ;-)
  24. 24. Dennis van der Stelt
  25. 25. Dennis van der Stelt scheduled tasks Your CEO had insomnia and was using the system in the middle of the night. The batch job failed somewhere in the middle of updating 74 million records… You need to figure out which row it failed on (how?), why it failed, correct the issue, then start the job again from where it left off, because if you have to start from the beginning, it won't get done before peak hours in the morning. because that sounds better than batch job
  26. 26. Dennis van der Stelt BATCH JOB All customers that ordered $5000 in the last year, get preferred status DateTime cutoff = DateTime.Today.AddDays(-365); foreach(var customer in customers) { var orderTotal = customer.Orders .Where(o => o.OrderDate > cutoff) .Sum(order => order.OrderValue); customer.Prefered = orderTotal > 5000; }
  27. 27. Dennis van der Stelt Tromsø, Norway
  28. 28. Dennis van der Stelt what if we can see things before they happen?
  29. 29. Dennis van der Stelt customer preferred status Dev: Let's say Steve orders something for $100. At what point does that amount no longer count toward Steve's preferred status? BA: That's easy, after 365 days!
  30. 30. Dennis van der Stelt -$300-$100 +$300 DURABLE TIMEOUTS Our way to predict the future 2015 2016 +$100
  31. 31. Dennis van der Stelt DURABLE TIMEOUTS public void Handle(OrderPlaced message) { this.Data.CustomerId = message.CustomerId; this.Data.RunningTotal += message.Amount; this.RequestTimeout<OrderExpired>(TimeSpan.FromDays(365), timeout => timeout.Amount = message.Amount); CheckForPreferredStatusChange(); } public void Handle(OrderExpired message) { this.Data.RunningTotal -= message.Amount; CheckForPreferredStatusChange(); }
  32. 32. Dennis van der Stelt DURABLE TIMEOUTS private void CheckForPreferredStatusChange() { if(this.Data.PreferredStatus == false && this.Data.RunningTotal >= 5000) { this.Bus.Publish<CustomerHasBecomePreferred>( evt => evt.CustomerId = this.Data.CustomerId); } else if(this.Data.PreferredStatus == true && this.Data.RunningTotal < 5000) { this.Bus.Publish<CustomerHasBecomeNonPreferred( evt => evt.CustomerId = this.Data.CustomerId); } }
  33. 33. Dennis van der Stelt Best Practices The silver bullets?
  34. 34. Dennis van der Stelt Rule #1 : Don’t query data Never ever, ever, ever query data - From the saga to another data source - Owned by saga with a 3rd party tool
  35. 35. race conditions do not exist
  36. 36. Dennis van der Stelt STARTING SAGAS What business events can start a Saga? class MySaga : Saga<MySagaData>, IAmStartedByMessages<MyMessage>, IAmStartedByMessages<YourMessage>, IAmStartedByMessages<AnotherMesssage> { public void Handle(MyMessage message) { … if (VerifyState()) MarkAsComplete(); } }
  37. 37. sagas bring agility
  38. 38. Dennis van der Stelt Please… stay in touch! http://dennis.bloggingabout.net dennis@bloggingabout.net

×