The Big Picture - Integrating Buzzwords

Alessandro Giorgetti
Alessandro GiorgettiSoftware Developer at SID s.r.l.
The Big Picture…
Integrating Buzzwords
Alessandro Giorgetti
www.primordialcode.com
When it comes to software development
There’s no definitive «recipe» for the architectures / patterns we use every day.
They are more like a bunch of guidelines that need to be adapted to:
oOur Business / System / Solution.
oOur Environment.
oOur Team.
oOur Skill Level.
We have so many options
Ultimate goals of software development
oUnderstand what the business does.
oBreak down the system in manageable parts.
oPlan / Design / Develop for then evolution of the business.
oReduce the coupling between the parts.
oTake into accout the «Failures», make them part of the design.
oManage the resources in an efficient way.
Ideally, create a Reactive System
https://www.reactivemanifesto.org/
Message Driven
Loose coupling through
async message passing
Resilient
Error containment
Fault tolerance and recovery
Elastic
Scalability
React to workload change
Responsive
Rapid and consistent response times
A Reactive System is made of components that have the following properties:
Be Agile!
Waterfall / Big Design Upfront ™ does not (read always never) work, so «be Agile»!
oDiscovery: figure out how it works.
oPlan / Design / Architect.
oCode like there’s not tomorrow.
oValidate.
oRepeat.
In short: SCRUM + Backlog!! https://en.wikipedia.org/wiki/Scrum_(software_development)
Discovery
Build a logical view of the system. Some practices proven to work very well (for us):
oDDD (Domain Driven Design).
oEvent Stroming.
Domain Driven Design
An approach to software development for «complex systems» that works promoting the
adoption of an evolving domain model, that derives from a collaboration between technical
and domain experts to iteratively refine a conceptual model that addresses particular domain
problems.
Key concepts:
oUbiquitous language.
oBounded contexts.
oContext map.
oAggregates / Commands / Events (Messages).
Event Storming
«Tool / Workshop» that allows to quickly explore complex business domains.
You can figure out how the system works (how much in detail is up to you); you’ll also get an overrall
idea of the «shape» of the system.
Bounded Contexts
oUbiquitous Language: the language defined between users and developers that defines elements of the business.
oAggregates: clusters of objects that can be treated as a unit.
oBounded Context: the environment in which a specific Ubiquitous Language is valid.
o Each Context defines a «business boundary»  technical boundaries are other things follows.
BC 1
Aggregate(s)
Cmd(s) Evt(s)
BC n
Aggregate(s)
Cmd(s) Evt(s)
Msg(s)
Msg(s)
Context Map
Context Maps draw the relationships between the Bounded Contexts, they are:
oConformist relationship: the downstream context might follow and adapt to what the upstream context says.
oPartnership relationships (both context work towards a common goal; the same term might exists in multiple contexts with different behaviors).
oTechnical relationships (how they talk).
A system will almost never be made of a single bounded context, they operate toghether to fulfill the system requirements and implement the business capabilities.
BC 1
BC 3 BC 4
BC 2
External System
From the Bounded Contexts to the code
We have an «abstract» view of the system, we need to convert this in «code» that do things.
Break up the system/application in pieces.
We used to have a magical name for these pieces in the past: «Services».
But what is a Service?
To give a definition for Service (in software development) is not an easy task, because the term is
overloaded… and has different «meanings» at the different «levels» of the architecture.
Service: a definition (sort of)
We can define a service by its fundamentals / principles (the four tenets):
oServices are autonomous.
oServices have well defined boundaries.
oShare contracts and schema.
oServices interoperate based on policies.
All of this to provide the implementation for a specific Business Capability!
What… doesn’t it seem similar to a
Bounded Context ?!
oThey are Autonomous / have Explicit Boundaries: encapsulation, they «contains» everything
that’s needed to implement a business capability.
oThey Share contract and schema: that’s a communication protocol.
BC 1
BC 3 BC 4
BC 2
External SystemBC 1
BC 3 BC 4
BC 2
External System
What… doesn’t it seem similar to a
Bounded Context ?!
oThey are Autonomous / have Explicit Boundaries: encapsulation, they «contains» everything
that’s needed to implement a business capability.
oThey Share contract and schema: that’s a communication protocol.
BC 1
BC 3 BC 4
BC 2
External System
Service 1
Service 3 Service 4
Service 2
External System
A long time ago
in a galaxy far, far away…. SOA
Service Oriented Architecture
The system should have been made of many services interacting between each others.
A couple of problems of SOA:
oOver time it became too much tied to a specific technology (web services).
oIf done wrong you’ll end up having coupling when one service directly calls into another (let’s build a
distributed monolith… with latency and reliability problems).
MicroServices: SOA v2
Microservices is a variant of the service-oriented architecture (SOA) architectural style that
structures an application as a collection of loosely coupled services.
In a microservices architecture, services should be fine-grained and the protocols should be
lightweight.
Small, independent applications, loosely coupled, communicating with each other in an
asynchronous manner, and can fail independently, and if one or some fails, the system as a
whole still works, though the functionality might be downgraded when there are failures.
What metric will you use to define «micro» ?
SOA / Microservices: the «good» side
oEasier to reason about Small Services.
oNo single point of failure.
• Be careful of shared data! (Data duplication is not evil!).
oServices are independently deployable.
• They can be developed and deployed at different speeds.
oEasier to adopt newer technologies (retire vs rewrite).
• Even different technologies (the system might not be omogeneous).
• They are a “good way out” from the legacy monolithic applications.
oEasier to scale out smaller services.
SOA / Microservices: the «bad» side
oMany moving parts: need Context Maps.
oMany technologies: need more skills.
oVersioning.
oMultiple Repositories: ALM (Application Lifecycle Management) can be quite complex.
Coupling ? Use Async Messaging!
We should reduce the coupling between the services in order to:
oAllow for independent evolution of each service (development / deployment).
oMake it easier to scale out part of the System.
oLimit the friction between the services.
oExchange information between services implemented with different technologies.
Design the communication protocol around «messages».
oMessage Driven Architecture.
But wait… The «service» as an … ?
Let’s go back to the definition of «Microservice»:
•Small, independent applications,
•loosely coupled,
•communicating with each other in an asynchronous manner,
•…and if one or some fails, the system as a whole still works, though the functionality might be
downgraded when there are failures.
…Sounds like an Actor!!
Microservice: the traits of an Actor
ACTOR
•Have an internal State.
•Have behavior.
•Send messages / communicate with other
actors.
•Create / spawn child Actors.
•Change its own behavior and process the next
message it receives differently.
SERVICE
•Autonomous / Have Boundaries.
•Implement a business capability.
•Send messages / communicate with other
services.
•Might be a composition of «micro»services.
•That’s the Service internal implementaion:
mind your own business!
An Actor System as Infrastructure ?
We can design our whole system as a collaboration of Actor Systems and Actors,
exchanging messages using the transport infrastructure exposed by the Actor Framework.
Technically speaking:
oAn Actor System can be made of a Cluster of Nodes.
oEach Node hosting one or more «MicroServices».
oEach Service being implemented as an Actor (with many children Actors).
The Goodz:
It «easy» to do if you know the Actor Framework:
oCommunication problems between nodes are already solved.
oLocation transparency (allows to interact with actors regardless the physical machine
they are deployed).
oElasticity: provided by the framework (Routing).
oResilience: benefits from the error containment / error management strategies.
oAvailability / Latency ?! everything is already async since day one… and you should
have faced those problems from the beginning.
The Badz:
oYou are tieing yourself to a single technology/platform (unless the framework spans multiple
platforms).
oMessage processing will be «serialized» for each actor: carefully design for parallelization
(routing strategies) and performances.
oReliability: Beware of the message delivery patters (at most once, at least one, exactly once).
So using an Actor Framework as the only backbone of the architecture might not be a good
idea, you can opt for using a more «traditional» approach with queues and service busses .
A Service in a Distributed Environment:
OWNS and PROTECTS its data… it’s responsible for them and their exposure to the rest of the
system.
Ideally, everything that «belongs» to a service is managed by the service itself:
oFrom the Databases  to the Backend  to the UI components (widgets).
oThe Team responsible for the service should have expertise in all the areas.
This also means: Composition Patterns.
A Service spans multiple «technologies»
and «responsibilities»
Product Catalog Inventory Pricing
Backend
( .net / java / go / … )
Web Portal
( web api / html /
javascript / …)
Mobile Apps
( .net / xamarin / swift /
… )
What about the Deployment Unit(s) ?
Team 1
Team 1
Team 1
Team 2
Team 2
Team 2
Team 3
Team 3
Team 3
What’s inside a «Service» ?
A service is a composition of components and smaller «services»… that collaborate to
implement a specific business capability!!
The term service is «overloaded» and has slightly different properties:
oA service, in this context, is a support entity for other components in order to fulfill very specific
tasks.
oMight not be fully autonomous.
oThere will be coupling!
How do we «code» the Service ?
Inside a Service / Microservice we can follow any architectural style we want!
They are «small» and «autonomous» and «easily replaceable»!
(Given we respect the communication protocol contracts).
Multi-Tier Layered Architecture
CQRS / ES (Aggregate  Domain Object)
Microservices + UI Composition
… and when we finally start coding…
Some things we never consider:
Keep in mind the fallacies of distributed systems:
1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.
And we always underestimate:
1. Error Handling, Failures and Recovery.
2. Performance and Bottlenecks.
3. Blocking operations.
4. Concurrency.
5. Consistency.
6. State Management / Locks / thread sync.
7. Parallelism.
8. Availability.
Implementing an «enterprise level distributed» solution is expensive!
Multithreaded / Distributed Systems
are hard… DAMN HARD!
oComplexity is spread all around place (not all piled up, like a monoloth).
oHard to debug / troubleshoot.
oDesigning for Eventual Consistency is not easy (it changes the UI paradigm and the backend:
forget ACID and distributed transactions, use workflows!).
oMultiple services  multiple Teams  multiple Projects to manage ALM ?!?.
oDeployment and Versioning: every service live on its own!
oAuthentication & Authorization ?
…this is where you usually see us cry…
Why don’t we bring the Reactive System
principles inside the Service ?
https://www.reactivemanifesto.org/
Message Driven
Loose coupling through
async message passing
Resilient
Error containment
Fault tolerance and recovery
Elastic
Scalability
React to workload change
Responsive
Rapid and consistent response times
We might consider using some tools that helps
us reason about such Reactive Systems.
Why not implementing a Service with a
Framework that:
oAllows us to scale up and down (vertically).
oAllows us to scale in and out (horizontally).
oAvoid most manual thread management sorting out resourse
contention.
oIt’s Fault Tolerant and promotes Error Containment by design.
oAligns very well with the DDD concepts.
if (Service isMadeOf(Aggregates))
return Actors;
AN AGGREGATE…
•Has internal State.
•Protects the invariants.
•Has behavior.
•Exchanges information with other aggregates
(optionally with messages).
•Root + Children.
AN ACTOR…
•Has internal State.
•The internal State is private.
•Has behavior.
•Sends messages / communicates with other
actors.
•Creates / spawn childs Actors.
•Changes its own behavior and process the next
message it receives differently.
The Actor Model can be a very viable
solution to implement a Service
A single MicroService can be made of several Actors!
Benefits:
oFocus on the business problems, the domain models and the
workflows.
oTechnical problems will be addressed by the framework itself
(multithreading, timeouts, error containment).
The Actor Model can be a very viable
solution to implement a Service
There will be Pain points:
oEverything will be asynchronous by design.
oHarder to debug and trace what’s going on.
oEmbrace Eventual Consistency and design the user experience for it!
Code snippets
A QUICK AND DIRTY CQRS / ES SERVICE, POWERED BY THE ACTOR
MODEL
GitHub repositories
Akka.Net 101 – Source Code from my Akka.Net Introductory Course: a series of lessons /
tutorials that will guide you through most of the features of Akka.Net (Actor’s communication,
Routing, Remoting, Clustering, etc…).
https://github.com/PrimordialCode/akkadotnet101
BigBrother – early stages of a system supervision dashboard: it contains a sample application
implementing the CQRS / ES pattern with NStore and Akka.Net Actors.
https://github.com/PrimordialCode/BigBrother
The Big Picture - Integrating Buzzwords
The Domain:
Events and Aggregate (using NStore)
https://github.com/ProximoSrl/NStore
The Commands…
The CommandHandler…
…and its base class
The projections
The Aggregate as an Actor
Two Ways of doing it:
1. Use Akka.Persistence or similar from other Actor Frameworks!
• PersistentActors – to implement the aggregates.
• PersistentViews – to implement the projections.
• You are tying yourself to the Framework.
2. Provide your own «wrapper» around the Aggregate:
• Can swap the Actor Framework without reimplementing the actor.
• Can use the Aggregate outside the Actor Framework (might be dangerous, it violates the singleton pattern, you will
loose the benefits of thread safety, lock free code, resource contention, etc…).
• Easier to add an Actor Framework in an ongoing project.
Let’s go for option 2.
The «Actor per Aggregate» pattern
• Wraps a single instance of the CommandHandler.
• We can opt for a ‘singleton’ pattern for each
Actor that maps to an Aggregate:
• No Synchronization problems when
accessing the database!
• The command handler can cache the
aggregate instance.
• Sets a Timeout to shutdown the Actor if it’s not
doing anything.
• Can signal the ProjectionActor that new data is
available.
Projection Engine
•Can be started / stopped.
•Can be signaled.
•Must be ‘reactive’ even when doing heavy work.
Waiting Working
PollImmediately
TimeOut
(start poll operation again)
PollingCompleted
StartPolling
StopPolling
PollImmediately
ProjectionActor
The Projection Actor
In this journey we saw that:
oMany theories / disciplines / practices developed over the years share a common ground!
oIt’s a good thing to reason about the system as a whole: use Event Storming (or something
similar) find out the Bounded Contexts and build Context Maps!
oEncapsulation and Composition work: break out the system in manageable parts and let them
communicate.
In this journey we saw that:
oIn an highly concurrent and distributed scenario an Actor Framework can be the ideal solution, at least in some parts of
the system.
oA Microservices-like approach seems a nice fit applications engineered with DDD concepts.
oA single Service can be implemented with Aggregates communcatin with Events and Messages  CQRS / ES style 
«Actor per Aggregate» pattern.
o«Integrate» several «frameworks/tools» through the lifecycle of the project: DDD / Microservices / Messages /
ActorModel.
oExpertise comes at a price!
Let’s keep in touch
Dott.ing. Alessandro Giorgetti
Facebook: https://www.facebook.com/giorgetti.alessandro
Twitter: @a_giorgetti
LinkedIn: https://it.linkedin.com/in/giorgettialessandro
E-mail: alessandro.giorgetti@live.com
Blog: www.primordialcode.com
Alessandro Giorgetti
www.primordialcode.com
1 of 57

Recommended

netsuite-integration-whitepaper by
netsuite-integration-whitepapernetsuite-integration-whitepaper
netsuite-integration-whitepaperOlivier Gagnon
314 views6 slides
DDD by
DDDDDD
DDDAntonio Radesca
761 views38 slides
Let's talk about... Microservices by
Let's talk about... MicroservicesLet's talk about... Microservices
Let's talk about... MicroservicesAlessandro Giorgetti
573 views73 slides
Microservices Architecture by
Microservices ArchitectureMicroservices Architecture
Microservices ArchitectureAlessandro Giorgetti
269 views68 slides
Architecting for Change: An Agile Approach by
Architecting for Change: An Agile ApproachArchitecting for Change: An Agile Approach
Architecting for Change: An Agile ApproachBen Stopford
1.7K views81 slides
Domain Driven Design by
Domain Driven DesignDomain Driven Design
Domain Driven DesignMuhammad Ali
99 views27 slides

More Related Content

Similar to The Big Picture - Integrating Buzzwords

Designingapplswithnet by
DesigningapplswithnetDesigningapplswithnet
DesigningapplswithnetDSK Chakravarthy
615 views46 slides
Software Architecture for Robotics by
Software Architecture for RoboticsSoftware Architecture for Robotics
Software Architecture for RoboticsLorran Pegoretti
2.3K views7 slides
Microservices architecture enterprise architecture by
Microservices architecture enterprise architectureMicroservices architecture enterprise architecture
Microservices architecture enterprise architectureAdhiguna Mahendra
84 views57 slides
ASAS 2014 - Simon Brown by
ASAS 2014 - Simon BrownASAS 2014 - Simon Brown
ASAS 2014 - Simon BrownAvisi B.V.
1.2K views77 slides
The Clean Architecture by
The Clean ArchitectureThe Clean Architecture
The Clean ArchitectureDmytro Turskyi
60 views18 slides
Application Of A Macbook Pro And Os X by
Application Of A Macbook Pro And Os XApplication Of A Macbook Pro And Os X
Application Of A Macbook Pro And Os XKris Cundiff
2 views46 slides

Similar to The Big Picture - Integrating Buzzwords(20)

Software Architecture for Robotics by Lorran Pegoretti
Software Architecture for RoboticsSoftware Architecture for Robotics
Software Architecture for Robotics
Lorran Pegoretti2.3K views
Microservices architecture enterprise architecture by Adhiguna Mahendra
Microservices architecture enterprise architectureMicroservices architecture enterprise architecture
Microservices architecture enterprise architecture
ASAS 2014 - Simon Brown by Avisi B.V.
ASAS 2014 - Simon BrownASAS 2014 - Simon Brown
ASAS 2014 - Simon Brown
Avisi B.V.1.2K views
Application Of A Macbook Pro And Os X by Kris Cundiff
Application Of A Macbook Pro And Os XApplication Of A Macbook Pro And Os X
Application Of A Macbook Pro And Os X
Kris Cundiff2 views
Timeless design in a cloud-native world by Uwe Friedrichsen
Timeless design in a cloud-native worldTimeless design in a cloud-native world
Timeless design in a cloud-native world
Uwe Friedrichsen8.7K views
Java TechTalk "Spring boot made life easier with Kubernetes and Microservices" by GlobalLogic Ukraine
Java TechTalk "Spring boot made life easier with Kubernetes and Microservices"Java TechTalk "Spring boot made life easier with Kubernetes and Microservices"
Java TechTalk "Spring boot made life easier with Kubernetes and Microservices"
2009 10-08 soa-og_itil_does service in it service rhyme with service as in so... by Peter Rosenberg
2009 10-08 soa-og_itil_does service in it service rhyme with service as in so...2009 10-08 soa-og_itil_does service in it service rhyme with service as in so...
2009 10-08 soa-og_itil_does service in it service rhyme with service as in so...
Peter Rosenberg298 views
Freedomotic v1.5 whitepaper by freedomotic
Freedomotic v1.5 whitepaperFreedomotic v1.5 whitepaper
Freedomotic v1.5 whitepaper
freedomotic348 views
Moving to Microservices with the Help of Distributed Traces by KP Kaiser
Moving to Microservices with the Help of Distributed TracesMoving to Microservices with the Help of Distributed Traces
Moving to Microservices with the Help of Distributed Traces
KP Kaiser241 views
Operator-less DataCenters -- A Reality by Kishore Arya
Operator-less DataCenters -- A RealityOperator-less DataCenters -- A Reality
Operator-less DataCenters -- A Reality
Kishore Arya76 views
Operator-Less DataCenters A Near Future Reality by Kishore Arya
Operator-Less DataCenters A Near Future RealityOperator-Less DataCenters A Near Future Reality
Operator-Less DataCenters A Near Future Reality
Kishore Arya118 views
Agile architecture upload by The Real Dyl
Agile architecture uploadAgile architecture upload
Agile architecture upload
The Real Dyl550 views
Microsoft BizTalk server seen by the programmer’s eyes by Sandro Pereira
Microsoft BizTalk server seen by the programmer’s eyesMicrosoft BizTalk server seen by the programmer’s eyes
Microsoft BizTalk server seen by the programmer’s eyes
Sandro Pereira1.4K views
service orentation documentation by pavan nani
service orentation documentationservice orentation documentation
service orentation documentation
pavan nani107 views

More from Alessandro Giorgetti

Angular Unit Testing by
Angular Unit TestingAngular Unit Testing
Angular Unit TestingAlessandro Giorgetti
2.7K views71 slides
AngularConf2016 - A leap of faith !? by
AngularConf2016 - A leap of faith !?AngularConf2016 - A leap of faith !?
AngularConf2016 - A leap of faith !?Alessandro Giorgetti
261 views49 slides
AngularConf2015 by
AngularConf2015AngularConf2015
AngularConf2015Alessandro Giorgetti
581 views42 slides
TypeScript . the JavaScript developer best friend! by
TypeScript . the JavaScript developer best friend!TypeScript . the JavaScript developer best friend!
TypeScript . the JavaScript developer best friend!Alessandro Giorgetti
981 views32 slides
«Real Time» Web Applications with SignalR in ASP.NET by
«Real Time» Web Applications with SignalR in ASP.NET«Real Time» Web Applications with SignalR in ASP.NET
«Real Time» Web Applications with SignalR in ASP.NETAlessandro Giorgetti
5.4K views33 slides
DNM19 Sessione1 Orchard Primo Impatto (ita) by
DNM19 Sessione1 Orchard Primo Impatto (ita)DNM19 Sessione1 Orchard Primo Impatto (ita)
DNM19 Sessione1 Orchard Primo Impatto (ita)Alessandro Giorgetti
702 views46 slides

More from Alessandro Giorgetti(7)

Recently uploaded

AI and Ml presentation .pptx by
AI and Ml presentation .pptxAI and Ml presentation .pptx
AI and Ml presentation .pptxFayazAli87
13 views15 slides
JioEngage_Presentation.pptx by
JioEngage_Presentation.pptxJioEngage_Presentation.pptx
JioEngage_Presentation.pptxadmin125455
6 views4 slides
Bootstrapping vs Venture Capital.pptx by
Bootstrapping vs Venture Capital.pptxBootstrapping vs Venture Capital.pptx
Bootstrapping vs Venture Capital.pptxZeljko Svedic
14 views17 slides
Sprint 226 by
Sprint 226Sprint 226
Sprint 226ManageIQ
10 views18 slides
Quality Engineer: A Day in the Life by
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the LifeJohn Valentino
7 views18 slides
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Lisi Hocke
35 views124 slides

Recently uploaded(20)

AI and Ml presentation .pptx by FayazAli87
AI and Ml presentation .pptxAI and Ml presentation .pptx
AI and Ml presentation .pptx
FayazAli8713 views
JioEngage_Presentation.pptx by admin125455
JioEngage_Presentation.pptxJioEngage_Presentation.pptx
JioEngage_Presentation.pptx
admin1254556 views
Bootstrapping vs Venture Capital.pptx by Zeljko Svedic
Bootstrapping vs Venture Capital.pptxBootstrapping vs Venture Capital.pptx
Bootstrapping vs Venture Capital.pptx
Zeljko Svedic14 views
Sprint 226 by ManageIQ
Sprint 226Sprint 226
Sprint 226
ManageIQ10 views
Quality Engineer: A Day in the Life by John Valentino
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the Life
John Valentino7 views
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by Lisi Hocke
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Lisi Hocke35 views
FOSSLight Community Day 2023-11-30 by Shane Coughlan
FOSSLight Community Day 2023-11-30FOSSLight Community Day 2023-11-30
FOSSLight Community Day 2023-11-30
Shane Coughlan6 views
Ports-and-Adapters Architecture for Embedded HMI by Burkhard Stubert
Ports-and-Adapters Architecture for Embedded HMIPorts-and-Adapters Architecture for Embedded HMI
Ports-and-Adapters Architecture for Embedded HMI
Burkhard Stubert26 views
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with... by sparkfabrik
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
sparkfabrik8 views
FIMA 2023 Neo4j & FS - Entity Resolution.pptx by Neo4j
FIMA 2023 Neo4j & FS - Entity Resolution.pptxFIMA 2023 Neo4j & FS - Entity Resolution.pptx
FIMA 2023 Neo4j & FS - Entity Resolution.pptx
Neo4j17 views
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm15 views
Fleet Management Software in India by Fleetable
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India
Fleetable12 views
Introduction to Git Source Control by John Valentino
Introduction to Git Source ControlIntroduction to Git Source Control
Introduction to Git Source Control
John Valentino6 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi215 views
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin96 views
Myths and Facts About Hospice Care: Busting Common Misconceptions by Care Coordinations
Myths and Facts About Hospice Care: Busting Common MisconceptionsMyths and Facts About Hospice Care: Busting Common Misconceptions
Myths and Facts About Hospice Care: Busting Common Misconceptions

The Big Picture - Integrating Buzzwords

  • 1. The Big Picture… Integrating Buzzwords Alessandro Giorgetti www.primordialcode.com
  • 2. When it comes to software development There’s no definitive «recipe» for the architectures / patterns we use every day. They are more like a bunch of guidelines that need to be adapted to: oOur Business / System / Solution. oOur Environment. oOur Team. oOur Skill Level.
  • 3. We have so many options
  • 4. Ultimate goals of software development oUnderstand what the business does. oBreak down the system in manageable parts. oPlan / Design / Develop for then evolution of the business. oReduce the coupling between the parts. oTake into accout the «Failures», make them part of the design. oManage the resources in an efficient way.
  • 5. Ideally, create a Reactive System https://www.reactivemanifesto.org/ Message Driven Loose coupling through async message passing Resilient Error containment Fault tolerance and recovery Elastic Scalability React to workload change Responsive Rapid and consistent response times A Reactive System is made of components that have the following properties:
  • 6. Be Agile! Waterfall / Big Design Upfront ™ does not (read always never) work, so «be Agile»! oDiscovery: figure out how it works. oPlan / Design / Architect. oCode like there’s not tomorrow. oValidate. oRepeat. In short: SCRUM + Backlog!! https://en.wikipedia.org/wiki/Scrum_(software_development)
  • 7. Discovery Build a logical view of the system. Some practices proven to work very well (for us): oDDD (Domain Driven Design). oEvent Stroming.
  • 8. Domain Driven Design An approach to software development for «complex systems» that works promoting the adoption of an evolving domain model, that derives from a collaboration between technical and domain experts to iteratively refine a conceptual model that addresses particular domain problems. Key concepts: oUbiquitous language. oBounded contexts. oContext map. oAggregates / Commands / Events (Messages).
  • 9. Event Storming «Tool / Workshop» that allows to quickly explore complex business domains. You can figure out how the system works (how much in detail is up to you); you’ll also get an overrall idea of the «shape» of the system.
  • 10. Bounded Contexts oUbiquitous Language: the language defined between users and developers that defines elements of the business. oAggregates: clusters of objects that can be treated as a unit. oBounded Context: the environment in which a specific Ubiquitous Language is valid. o Each Context defines a «business boundary»  technical boundaries are other things follows. BC 1 Aggregate(s) Cmd(s) Evt(s) BC n Aggregate(s) Cmd(s) Evt(s) Msg(s) Msg(s)
  • 11. Context Map Context Maps draw the relationships between the Bounded Contexts, they are: oConformist relationship: the downstream context might follow and adapt to what the upstream context says. oPartnership relationships (both context work towards a common goal; the same term might exists in multiple contexts with different behaviors). oTechnical relationships (how they talk). A system will almost never be made of a single bounded context, they operate toghether to fulfill the system requirements and implement the business capabilities. BC 1 BC 3 BC 4 BC 2 External System
  • 12. From the Bounded Contexts to the code We have an «abstract» view of the system, we need to convert this in «code» that do things. Break up the system/application in pieces. We used to have a magical name for these pieces in the past: «Services». But what is a Service? To give a definition for Service (in software development) is not an easy task, because the term is overloaded… and has different «meanings» at the different «levels» of the architecture.
  • 13. Service: a definition (sort of) We can define a service by its fundamentals / principles (the four tenets): oServices are autonomous. oServices have well defined boundaries. oShare contracts and schema. oServices interoperate based on policies. All of this to provide the implementation for a specific Business Capability!
  • 14. What… doesn’t it seem similar to a Bounded Context ?! oThey are Autonomous / have Explicit Boundaries: encapsulation, they «contains» everything that’s needed to implement a business capability. oThey Share contract and schema: that’s a communication protocol. BC 1 BC 3 BC 4 BC 2 External SystemBC 1 BC 3 BC 4 BC 2 External System
  • 15. What… doesn’t it seem similar to a Bounded Context ?! oThey are Autonomous / have Explicit Boundaries: encapsulation, they «contains» everything that’s needed to implement a business capability. oThey Share contract and schema: that’s a communication protocol. BC 1 BC 3 BC 4 BC 2 External System Service 1 Service 3 Service 4 Service 2 External System
  • 16. A long time ago in a galaxy far, far away…. SOA Service Oriented Architecture The system should have been made of many services interacting between each others. A couple of problems of SOA: oOver time it became too much tied to a specific technology (web services). oIf done wrong you’ll end up having coupling when one service directly calls into another (let’s build a distributed monolith… with latency and reliability problems).
  • 17. MicroServices: SOA v2 Microservices is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services should be fine-grained and the protocols should be lightweight. Small, independent applications, loosely coupled, communicating with each other in an asynchronous manner, and can fail independently, and if one or some fails, the system as a whole still works, though the functionality might be downgraded when there are failures. What metric will you use to define «micro» ?
  • 18. SOA / Microservices: the «good» side oEasier to reason about Small Services. oNo single point of failure. • Be careful of shared data! (Data duplication is not evil!). oServices are independently deployable. • They can be developed and deployed at different speeds. oEasier to adopt newer technologies (retire vs rewrite). • Even different technologies (the system might not be omogeneous). • They are a “good way out” from the legacy monolithic applications. oEasier to scale out smaller services.
  • 19. SOA / Microservices: the «bad» side oMany moving parts: need Context Maps. oMany technologies: need more skills. oVersioning. oMultiple Repositories: ALM (Application Lifecycle Management) can be quite complex.
  • 20. Coupling ? Use Async Messaging! We should reduce the coupling between the services in order to: oAllow for independent evolution of each service (development / deployment). oMake it easier to scale out part of the System. oLimit the friction between the services. oExchange information between services implemented with different technologies. Design the communication protocol around «messages». oMessage Driven Architecture.
  • 21. But wait… The «service» as an … ? Let’s go back to the definition of «Microservice»: •Small, independent applications, •loosely coupled, •communicating with each other in an asynchronous manner, •…and if one or some fails, the system as a whole still works, though the functionality might be downgraded when there are failures. …Sounds like an Actor!!
  • 22. Microservice: the traits of an Actor ACTOR •Have an internal State. •Have behavior. •Send messages / communicate with other actors. •Create / spawn child Actors. •Change its own behavior and process the next message it receives differently. SERVICE •Autonomous / Have Boundaries. •Implement a business capability. •Send messages / communicate with other services. •Might be a composition of «micro»services. •That’s the Service internal implementaion: mind your own business!
  • 23. An Actor System as Infrastructure ? We can design our whole system as a collaboration of Actor Systems and Actors, exchanging messages using the transport infrastructure exposed by the Actor Framework. Technically speaking: oAn Actor System can be made of a Cluster of Nodes. oEach Node hosting one or more «MicroServices». oEach Service being implemented as an Actor (with many children Actors).
  • 24. The Goodz: It «easy» to do if you know the Actor Framework: oCommunication problems between nodes are already solved. oLocation transparency (allows to interact with actors regardless the physical machine they are deployed). oElasticity: provided by the framework (Routing). oResilience: benefits from the error containment / error management strategies. oAvailability / Latency ?! everything is already async since day one… and you should have faced those problems from the beginning.
  • 25. The Badz: oYou are tieing yourself to a single technology/platform (unless the framework spans multiple platforms). oMessage processing will be «serialized» for each actor: carefully design for parallelization (routing strategies) and performances. oReliability: Beware of the message delivery patters (at most once, at least one, exactly once). So using an Actor Framework as the only backbone of the architecture might not be a good idea, you can opt for using a more «traditional» approach with queues and service busses .
  • 26. A Service in a Distributed Environment: OWNS and PROTECTS its data… it’s responsible for them and their exposure to the rest of the system. Ideally, everything that «belongs» to a service is managed by the service itself: oFrom the Databases  to the Backend  to the UI components (widgets). oThe Team responsible for the service should have expertise in all the areas. This also means: Composition Patterns.
  • 27. A Service spans multiple «technologies» and «responsibilities» Product Catalog Inventory Pricing Backend ( .net / java / go / … ) Web Portal ( web api / html / javascript / …) Mobile Apps ( .net / xamarin / swift / … ) What about the Deployment Unit(s) ? Team 1 Team 1 Team 1 Team 2 Team 2 Team 2 Team 3 Team 3 Team 3
  • 28. What’s inside a «Service» ? A service is a composition of components and smaller «services»… that collaborate to implement a specific business capability!! The term service is «overloaded» and has slightly different properties: oA service, in this context, is a support entity for other components in order to fulfill very specific tasks. oMight not be fully autonomous. oThere will be coupling!
  • 29. How do we «code» the Service ? Inside a Service / Microservice we can follow any architectural style we want! They are «small» and «autonomous» and «easily replaceable»! (Given we respect the communication protocol contracts).
  • 31. CQRS / ES (Aggregate  Domain Object)
  • 32. Microservices + UI Composition
  • 33. … and when we finally start coding…
  • 34. Some things we never consider: Keep in mind the fallacies of distributed systems: 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. And we always underestimate: 1. Error Handling, Failures and Recovery. 2. Performance and Bottlenecks. 3. Blocking operations. 4. Concurrency. 5. Consistency. 6. State Management / Locks / thread sync. 7. Parallelism. 8. Availability. Implementing an «enterprise level distributed» solution is expensive!
  • 35. Multithreaded / Distributed Systems are hard… DAMN HARD! oComplexity is spread all around place (not all piled up, like a monoloth). oHard to debug / troubleshoot. oDesigning for Eventual Consistency is not easy (it changes the UI paradigm and the backend: forget ACID and distributed transactions, use workflows!). oMultiple services  multiple Teams  multiple Projects to manage ALM ?!?. oDeployment and Versioning: every service live on its own! oAuthentication & Authorization ?
  • 36. …this is where you usually see us cry…
  • 37. Why don’t we bring the Reactive System principles inside the Service ? https://www.reactivemanifesto.org/ Message Driven Loose coupling through async message passing Resilient Error containment Fault tolerance and recovery Elastic Scalability React to workload change Responsive Rapid and consistent response times
  • 38. We might consider using some tools that helps us reason about such Reactive Systems.
  • 39. Why not implementing a Service with a Framework that: oAllows us to scale up and down (vertically). oAllows us to scale in and out (horizontally). oAvoid most manual thread management sorting out resourse contention. oIt’s Fault Tolerant and promotes Error Containment by design. oAligns very well with the DDD concepts.
  • 40. if (Service isMadeOf(Aggregates)) return Actors; AN AGGREGATE… •Has internal State. •Protects the invariants. •Has behavior. •Exchanges information with other aggregates (optionally with messages). •Root + Children. AN ACTOR… •Has internal State. •The internal State is private. •Has behavior. •Sends messages / communicates with other actors. •Creates / spawn childs Actors. •Changes its own behavior and process the next message it receives differently.
  • 41. The Actor Model can be a very viable solution to implement a Service A single MicroService can be made of several Actors! Benefits: oFocus on the business problems, the domain models and the workflows. oTechnical problems will be addressed by the framework itself (multithreading, timeouts, error containment).
  • 42. The Actor Model can be a very viable solution to implement a Service There will be Pain points: oEverything will be asynchronous by design. oHarder to debug and trace what’s going on. oEmbrace Eventual Consistency and design the user experience for it!
  • 43. Code snippets A QUICK AND DIRTY CQRS / ES SERVICE, POWERED BY THE ACTOR MODEL
  • 44. GitHub repositories Akka.Net 101 – Source Code from my Akka.Net Introductory Course: a series of lessons / tutorials that will guide you through most of the features of Akka.Net (Actor’s communication, Routing, Remoting, Clustering, etc…). https://github.com/PrimordialCode/akkadotnet101 BigBrother – early stages of a system supervision dashboard: it contains a sample application implementing the CQRS / ES pattern with NStore and Akka.Net Actors. https://github.com/PrimordialCode/BigBrother
  • 46. The Domain: Events and Aggregate (using NStore) https://github.com/ProximoSrl/NStore
  • 51. The Aggregate as an Actor Two Ways of doing it: 1. Use Akka.Persistence or similar from other Actor Frameworks! • PersistentActors – to implement the aggregates. • PersistentViews – to implement the projections. • You are tying yourself to the Framework. 2. Provide your own «wrapper» around the Aggregate: • Can swap the Actor Framework without reimplementing the actor. • Can use the Aggregate outside the Actor Framework (might be dangerous, it violates the singleton pattern, you will loose the benefits of thread safety, lock free code, resource contention, etc…). • Easier to add an Actor Framework in an ongoing project. Let’s go for option 2.
  • 52. The «Actor per Aggregate» pattern • Wraps a single instance of the CommandHandler. • We can opt for a ‘singleton’ pattern for each Actor that maps to an Aggregate: • No Synchronization problems when accessing the database! • The command handler can cache the aggregate instance. • Sets a Timeout to shutdown the Actor if it’s not doing anything. • Can signal the ProjectionActor that new data is available.
  • 53. Projection Engine •Can be started / stopped. •Can be signaled. •Must be ‘reactive’ even when doing heavy work. Waiting Working PollImmediately TimeOut (start poll operation again) PollingCompleted StartPolling StopPolling PollImmediately ProjectionActor
  • 55. In this journey we saw that: oMany theories / disciplines / practices developed over the years share a common ground! oIt’s a good thing to reason about the system as a whole: use Event Storming (or something similar) find out the Bounded Contexts and build Context Maps! oEncapsulation and Composition work: break out the system in manageable parts and let them communicate.
  • 56. In this journey we saw that: oIn an highly concurrent and distributed scenario an Actor Framework can be the ideal solution, at least in some parts of the system. oA Microservices-like approach seems a nice fit applications engineered with DDD concepts. oA single Service can be implemented with Aggregates communcatin with Events and Messages  CQRS / ES style  «Actor per Aggregate» pattern. o«Integrate» several «frameworks/tools» through the lifecycle of the project: DDD / Microservices / Messages / ActorModel. oExpertise comes at a price!
  • 57. Let’s keep in touch Dott.ing. Alessandro Giorgetti Facebook: https://www.facebook.com/giorgetti.alessandro Twitter: @a_giorgetti LinkedIn: https://it.linkedin.com/in/giorgettialessandro E-mail: alessandro.giorgetti@live.com Blog: www.primordialcode.com Alessandro Giorgetti www.primordialcode.com