Practical Data Mesh: Building Decentralized Data Architectures with Event Stream

All contents © MuleSoft, LLC
Sydney MuleSoft Meetup
Thank you for joining us today! We will get started at 6 pm.
If you haven’t signed up already, please sign up for today’s event through QR Code.
Welcome

Meet your Sydney Meetup Leaders
2

Meet your Sydney Salesforce Trailblazer Guides
3
Selina Le Khac

Meet your Sydney Meetup Moderators
4

Agenda
5
● Community updates - Harshana Martin
● Practical Data Mesh: Building Decentralized Data
Architectures with Event Stream - James Gollan
● Uniﬁcation of Data Products with Anypoint DataGraph -
Harshana Martin
● Training and Certiﬁcation Update - Raquel Paez Ricciardo
● Trivia
● Networking time!

What’s in Meetups?
6
● Community-led events
● Topics are up to YOU!
● Use cases
● Demos
● How-tos
● Workshops
● Personal stories
Learn - Network - Share
with other MuleSoft users
Join the group at https://meetups.mulesoft.com/sydney/
to be notiﬁed of all future events

Announcements
7

MuleSoft Product Releases

Universal API Management
API Experience Hub
9

Build, Discover, Govern and Manage any API on a uniﬁed platform
Universal API management on Anypoint Platform
Anypoint Platform
Anypoint API Manager | Anypoint API Designer | Anypoint Exchange | Anypoint Community Manager
Anypoint API
Catalog CLI
Discover and
catalog any API
Anypoint API
Governance
Ensure consistent API
quality and security
Anypoint Flex
Gateway
Control and secure
access to any API
Anypoint API
Experience Hub
Engage and create
API Ecosystems
Anypoint Code
Builder
Rapidly Build and
extend APIs
Govern Manage Engage
Discover
Build
NEW
BETA

Anypoint Exchange
Access and work with any API, built anywhere
Consolidate APIs built anywhere
Automatically discover APIs and relevant
documentation from any environment using
CI/CD pipelines
Package your APIs into products
Describe your APIs with asset information, interactive
documentation and simulated behavior
Streamline API consumer onboarding
Empower developers to discover, explore and get
started with your own APIs, assets and best practices
Jumpstart development and governance
Use hundreds of templates, APIs and governance
rules from the MuleSoft ecosystem
Engage

Start fast with simple API portal builder
with pre-built templates designed to
optimize building API experiences
Scale and grow with advanced
Salesforce Builder capabilities
to help API product teams to build, share and
engage with API programs
Built-in API marketplace and
consumption experience
to drive API adoption across internal
communities and partner communities
Anypoint API Experience Hub
Build API portals in minutes for better API engagement
NEW

Anypoint API Experience Hub
Empower developers, product managers & consumers to engage with APIs
Fast and Easy guided UI to build
API Portals
Create branded API portals using
built-in templates powered by
Salesforce Experience Cloud
Manage portal user and
registration requests
Self-service consumption
experience: search APIs, check API
docs, test APIs, request access
Built-in API marketplace to drive
API adoption and value Engage
API
developer
API
Consumer
API Product
Manager

Universal API Management
Anypoint Flex Gateway, API Manager and API Governance
14

Available Now H1 ‘23 H2 ‘23
Flex Gateway 1.5.1 - Release
Note
New Deployment Options
RHEL, OpenShift, CentOS, AWS Linux
Additional Policies
Conditional Routing to multiple Upstream
Endpoints
OAuth2 Introspection, Distributed rate limiting
Policy ordering in local and connected mode
Other Updates
Multiple Upstream Route support mTLS Local
and Connected
Support for additional containers: Podman
Forwarding proxy support
Sharing a port among multiple APIs
High Availability
● Liveness and Readiness Probes in K8s
High Availability
● Increase API limit per gateway to
400
Policies
● JSON threat Protection,
● Schema validation(OAS3)
Other Updates
● Flex CLI
● Policy Development Kit
(PDK) GA
● UX enhancements
● Increase API limit per
gateway to 600 and
then to 1000
Implement modern architecture with ultrafast, distributed API gateway
Anypoint Flex Gateway

Security
TLS support for Runtime Fabric
Flex Gateway
Multiple API Instances per Port
TLS & mTLS
Configure TLS on APIs deployed
to Flex G/W
Connected Apps
Connected apps for client app
management
Multiple Upstreams
Support multiple upstreams per
API with traffic management by
weight and condition
Health Check: Enable validating HTTP
response code for flex API on cadence
CloudHub 2.0: Easily deploy APIs to
CH2.0 from APIM
Governance support: API Governance
integration to evaluate API instances and
Policy configurations
CLI enhancements: Deploy APIs to Flex
via CLI
● Multiple Client iDP providers support
in the same Environment
● Improved experience to identify &
remediate violations detected by
Governance
● Full CLI support in API Manager,
including TLS configurations
● Improved experience to manage
large number of API Instances
● Easily migrate APIs from Flex local
mode to connected mode
Serve data from all your APIs to developers instantly and
manage using centralized control plane
Anypoint API Manager

Conformance
● API Conformance details in
Exchange
Async
● Govern Async APIs, with
OOTB best practice ruleset
Ruleset versioning
● Version rulesets, and
optionally evaluate APIs
automatically
API Lifecycle state
● Evaluate APIs based on their
Life Cycle State
Govern Instances & Policies: Govern
API instances in API Manager;
Platform APIs for API Governance
OOTB Ruleset: New best practice
rulesets for API Management, Flex
Gateway, & Cataloging
Custom ruleset development: New
CLI & tutorials for custom ruleset
development
Validation report: Comprehensive
report to understand all violations in
an API; Enable automations via APIs
● Discover and Govern APIs from 3rd
party Gateways
● Actionable insights: OOTB metrics
and reports for state of union of all
your APIs
● CI/CD: CLI plugins to validate &
enforce governance rules in your
pipeline
● Govern SLA Tiers, Contracts and
Alerts
● OOTB Rulesets for Experience Hub
best practice
● Private proﬁles to generate audit
reports or test new rules without
notifying developers
Gain consistent security and quality without sacriﬁcing development speed
Anypoint API Governance

Platform & Runtime
18

The Platform for the Future
Get started fast Grow with
eﬃciency
Streamline your
operations
Implement tighter
security controls
Fully Managed PaaS
MuleSoft CloudHub 2.0 NEW
CloudHub 2.0

H2 ‘23
H2 ‘23
Available Now
H1 ‘23
CloudHub 2.0
Non-HTTP Protocol Support
Support non-HTTP (TCP, HL7 MLLP, UDP, etc.)
inbound traﬃc through Ingress Controller
Enhanced Egress Control
● Remove default route to internet gateway and
make private space truly private
● Domain-based egress control
VPC to Private Space upgrade
Migrate existing VPCs deployed on CloudHub to
Private Spaces & save time on conﬁguration
Auto-scaling and Elastic
Provisioning
Take advantage of native container auto-scaling
strategy and achieve elastic deployment

CloudHub 2.0
Non HTTP Protocol Support for HL7/MLLP

Anypoint Exchange
Engage with your assets & enable reuse
Creator Permissions
● New permissions group for more granular access
management control.
● Enable users to modify only the assets they created
within a business group.
Asset Usage and Engagement Scoring
● Drive growth, engagement and reuse of application
networks by providing valuable usage insights.
● OOTB dashboards to track your asset catalog & get a
composite view of your assets’ “engagement score”
Search and Discovery
● Upgrades to Exchange search, including UX, advanced
ﬁlters, & improved relevancy of search results
Available Now
Q2 ‘23
Ongoing

Open Kubernetes Ecosystem Approach
Rancher (RKE)
H2 ‘23
Managed K8s deployments on
leading public clouds
Expanded K8s platform and
cloud native support
Deployment across any K8s
platform providers
Strategic partnerships with
major clouds / vendors
Future-proof with
open-source ecosystem
Extend mules to any CNCF
conformant platform
● Native support for running on fully
managed K8s platform with
industry-leading vendors
● Strategic partnership with select
vendors to oﬀer diﬀerentiated
developer and business experiences
● Fast-growing community and freedom of choice
for any LB, ingress, storage, monitoring and
security solution to build your future-proof stack
● Increased developer and devops productivity
with automation by tapping into a massive
ecosystem of tools
● Support of RTF on CNCF conformant K8s
providers including AliCloud ACK,
VMWare Tanzu etc.
● Robust controls to maintain security and
compliance OOTB uniformly across
varying instances of runtime plane
clusters
EKS
GKE
AKS
EKS Anywhere
AliCloud ACK
Available Now
H2 ‘23
Available Now Available Now
Runtime Fabric
VMWare Tanzu
H2 ‘23

Mule 4.5 – What it means for you
Feature & Process Changes
New Features New Release Cadence Support & Upgrade Policy
[Beta] OOTB support for
OpenTelemetry traces for Mule
Applications
Establishing new release train
aligned with other MuleSoft &
Salesforce products
No auto-upgrade for Runtime
Fabric deployments during
support window
Create & access Protobuf
structures natively in
DataWeave
Three new releases per year
(Feb, Jun, Oct) – speed up
innovation with faster,
predictable & consistent release
cycle
Introducing LTS versions to opt
into long term stable version for
improved stability and reduced
operation overhead
Introducing Edge versions to
opt into quarterly feature
enhancement
Tentative Release Timeline: Q3FY24

Anypoint Code Builder
Design, develop, deploy from a
single environment
Use modern,
cloud-native
tooling powered
by VS Code
Go faster with
recommendations
at design-time
Embrace code
reuse with API and
integration
building blocks
Public Beta

Author
RAML &
OAS3 API
Specs
Auto-generate
interactive
documentation
in your API Console
Seamless
integration with
Design Center &
Exchange
Design
API
Specifications
Design-first
approach for API-led
integration
Test API
specification
using the built in
mocking service

Automation
27

Available now in MuleSoft RPA
Credential pool
with user control
Enhance governance level
for the credential pool
Assign the right user to your credentials to
allow fine granular permissions. Each
credential stored in the credential pool can
have specific users and/or user groups
which are allowed to manage the login
information - have more control over your
confidential information.
Govern your credentials

Available now in MuleSoft RPA
Enhanced Web
Recording
Use speciﬁc web element
information for more
stable recordings
New attribute
selection panel
Within the new attribute selection panel the
user can choose more accurate web element
information to create more stable recordings.

Enhanced IDP
capabilities
New Action Step with
Queries for AWS
The new Analyze Document Queries
Action Steps can take natural language
written questions such as “What is the
amount of the invoice?” and perform
powerful AI and ML analysis on the
document to ﬁgure out the desired
information and extract it from the
document without any post processing.
Coming soon in MuleSoft RPA
Specify and extract information
from documents

Coming soon in MuleSoft RPA
Key-Features
● Business Rules and
decisions are integrated in
the Business process
model
● Support of separate
responsibilities for deﬁning
the decision rules and the
business process
● Graphical modelling of
decision models and rules
● Automatic creation of
needed variables / Easy to
use
● Business processes can still
run if only rules needs to
be changed
Move Business rules from
IT to Business
The Integration of a Decision Management
simpliﬁes the process itself and split the
responsibilities for automation design and
business rules. Complex, multi-level business
decisions or rules can be created very easily
using decision trees.
Decision
Management
Collaboratively automate
business decisions

Upcoming Events

Upcoming Community Meetups
Weekly events
• Check out all upcoming events every week here
Over 150 groups world wide
• Join from anywhere!
→ Join the Global Group: meetups.mulesoft.com/online-group-english
→ RSVP for upcoming events: meetups.mulesoft.com/events

All contents © MuleSoft, LLC 34
Become a MuleSoft Mentor
Who are MuleSoft Mentors?
● Individuals eager to become active
contributors across platforms
● Hold 1+ MuleSoft certiﬁcations (not
required!)
Why should you become a Mentor?
● Build your professional and personal
brand
● Help others skill up
● Earn recognition as a Mule expert by
MuleSoft and the broader ecosystem Apply now
mulesoft.com/mentors
A top community program

To conclude this intro...
35
● Fill out the survey!
● Contribute! meetups@mulesoft.com, `Contact the
Organiser` on the event page or post on the Online Group.
● Let’s take a group picture! 📸

Practical Data Mesh
Building Decentralised Data Architectures
with Event Streams
James Gollan
Manager, Enterprise Solutions Engineering ANZ
Conﬂuent

Data Mesh & Data Fabric
01
Event Streaming
02
Demo - Data Mesh in Conﬂuent Cloud
03
37
What we will cover

“Big data was supposed to transform the
enterprise. Existing data architectures, however,
have shortcomings that prevent them from
delivering the promised business value.”
39
Thoughtworks Whitepaper: The data mesh shift

A little
history
● In the 1970s, Online Transaction Processing
Systems (OLTP) began capturing
transactional data.
● In the 1980s, data warehouses emerged as
separate storage and analysis systems.
● Increased adoption of OLTP systems resulted in
a surge of data moving into data warehouses.
● Centralized data teams attempted to make
sense of the data with varying levels of success.

Data Mesh
● Concept ﬁrst spoken about by Zhamak
Dehghani of ThoughtWorks
● Aims to address the limitations of centralised
data architectures - aka the data monolith
● Technology agnostic - a set of four principles!
● The goal is unlocking more value from data
Note: Datamesh evolved in the Data Analytics space, but I
see it as relevant to the operational space as well
Some of these terms may seem familiar - that’s right, it it
similar to microservices

Domain-driven
ownership of data
Data as a product Federated
computational
governance
Self-serve data
platform
1 2 3 4
The Principles of a Data Mesh

Copyright 2021, Conﬂuent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Conﬂuent, Inc.
Domain Domain
Data as a product - domain ownership
43
Data product
Data product
Organisation

Data Fabric
● Integration of data from multiple systems.
● Uniﬁed data access across different sources and
formats.
● Real-time data processing.
● Scalability and ﬂexibility for handling growing
and diverse data.

The Foundational Assumption of Every Database:
Data at Rest
Data at rest
Slow, daily
batch processing
Simple, static
real-time queries
Databases

Real-time &
Historical
Data
A Sale
A shipment
A Trade
A Customer
Experience
A 30 second overview of event streaming
Continuously Process Streams of Data in Real-time
Real-Time Stream Processing
Rich Front-End Customer
Experiences
Real-Time Backend
Operations
Time
The log!

Why is Event Streaming a good ﬁt for meshing?
49
Streams are real-time, low latency ⇒ Propagate data immediately.
Streams are highly scalable ⇒ Handle today’s massive
data volumes.
Streams are stored, replayable ⇒ Capture real-time & historical
data.
Streams are immutable ⇒ Auditable source of record.
Streams are addressable, discoverable ⇒ Meet key criteria for mesh data.
Streams are popular for Microservices ⇒ Adapting to Data Mesh is often
easy.

Overview of the demo scenario
51
Loyalty
Customer
Credit
POS FAST MOVING
MySQL
SLOW
CHANGING
CSV
BATCH
P

Demo
scenario
● Product manager in Customer Service team
wants to build an application to improve
real-time knowledge of the customer’s loyalty
points
● Additionally, needs access to customer
address, credit information, and latest loyalty
transaction

Data quality rules
Define a schema rule to TRANSFORM a field
{
"schema": "...",
"ruleSet": {
"domainRules": [
{
"name": "populateFullName",
"kind": "TRANSFORM",
"type": "CEL_FIELD",
"expr": "name == 'fullName' && value == null ?
message.firstName + ' ' + message.lastName :
value"
}
]
}
}
Define a schema rule to VALIDATE a field
{
"schema": "...",
"ruleSet": {
"domainRules": [
{
"name": "checkSsnLength",
"kind": "CONSTRAINT",
"type": "CEL",
"expr": "size(message.ssn) == 9",
"onFailure": "DLQ"
}
]
}
}
Schema
Registry
Producer
Kafka
serializer
Kafka
deserializer
Consumer
Confluent Cloud
(execute rules) (execute rules)

Domain Domain
Stream sharing - data streaming across
organisational boundaries
54
Data
product
Data
product
Organisation
Domain Domain
Data
product
Data
product
Organisation

Summary
● Event streaming can form the foundation for
Data Mesh
● Focussing on Data as a Product is key
● Data products serve both operational and
analytics use-cases
● Conﬂuent Cloud integrates easily with
MongoDB Atlas

What next?
56
Download the Practical Data
Mesh eBook
Sign up for Conﬂuent Cloud and
receive $400 free credits

Uniﬁcation of Data Products with
Anypoint DataGraph

What we will cover
1. Data Mesh Principles
2. GraphQL and Anypoint DataGraph Overview
3. Technical Details
4. Demo
5. Next Steps

Data Mesh Principles

Data Mesh
4 Pillars in Data Mesh Architecture
Domain/BPG/BU
oriented
decentralised
ownership
and
Architecture
Self-Service
Platform,
Pipelines
and
Infrastructure
Federated
Governance
and
Centralised
Catalog
Data
As
A
Product
Data Mesh Principles
● Domain/BPG/BU
oriented decentralised
ownership and
Architecture
● Self-Service Platform,
Pipelines and
Infrastructure
● Federated Governance
and Centralised Catalog
● Data As A Product

Process
layer
Experience
layer
System
layer
Order API
Store
API
Marketing
Item API
Loyalty
API
Product API
Domain Driven Composable Architecture
Shipment API
Customer proﬁle API
Inventory
API
Receipts
API
Customer
API
Delivery
API
Web Mobile
Partner
In-store
checkout
Web API Mobile API
Partner API In-store API
Domains:
Product
Customer
Payment
Fulﬁlment

GraphQL and Anypoint DataGraph Overview

Process
layer
Experience
layer
System
layer
Order API
Store
API
Marketing
Item API
Loyalty
API
Product API
The evolution of consumption from
a composable architecture
Shipment API
Customer proﬁle API
Inventory
API
Receipts
API
Customer
API
Delivery
API
Web
Partner
In-store
checkout
Mobile

Unify APIs into powerful data
services without new code
Consume many APIs with a single
GraphQL request
Quick to start, easy to maintain: A fully
managed service that’s ready out-of-the-box
NEW
Anypoint DataGraph
Serve data from all your APIs
to developers instantly

How fast can you unify these APIs?
customerEmail
deliveryAddress
deliveryInfo
orderId
origination
shipmentId
taxAmount
…
Sales API
carrier
deliveryDate
id
status
trackingNumber
…
Shipment API
customerEmail
customerId
orderDate
orderId
productInfo
status
totalAmount
…
Order API
MERGE
LINK
Anypoint DataGraph

How fast can you unify these APIs?
customerEmail
deliveryAddress
deliveryInfo
orderId
origination
shipmentId
taxAmount
…
Sales API
carrier
deliveryDate
id
status
trackingNumber
…
Shipment API
customerEmail
customerId
orderDate
orderId
productInfo
status
totalAmount
…
Order API
MERGE
LINK
Anypoint DataGraph
EXAMPLE

Technical Detail

Anypoint DataGraph
Available Now
CLI Support
Automate processes by integrating into existing CI/CD
pipelines
DataGraph Best Practice RuleSets
Identify if REST API(s) adhere to best practices for
DataGraph federation directly through API Designer
Mutations
Create and modify data from linked APIs
CDN Caching & Automatic Persisted Queries
Improve performance and decrease bottlenecks in
network traﬃc by caching GraphQL query results through
automatic persisted queries
What’s New & What’s Coming
NEW

Anypoint DataGraph 2.0
DataGraph Views
Get complete control on who gets access
to what ﬁelds / types of your DataGraph
at runtime for enhanced security
H1 2023 H1 ‘24
GraphQL Re-imagined
GraphQL Federation
Create super-graphs from multiple sub-graphs in
minutes. Sub-graphs can reside in and out of
Anypoint Platform
Flexible Deployment Options
Deploy DataGraph to your data center
or private cloud through Kubernetes
based Anypoint Runtime Fabric or
directly to CloudHub 2.0
GraphQL APIs in Exchange
Catalog any GraphQL API in Exchange
GraphQL console in Exchange
Execute GraphQL operations right in Exchange
for discoverability and testing Custom Resolvers for Federation
Federate REST and GraphQL APIs with
transformation through JavaScript at the data source
level (bottom level APIs)
NEW
NEW
NEW

DataGraph - How It Works & Support
● Every time you add a new API, Anypoint Platform stores it as a graph of metadata
● DataGraph connects those graphs into one unified schema that runs as a single
SaaS GraphQL endpoint
● Can query across the underlying APIs without needing to understand all of the
relationships or specific capabilities
Anypoint DataGraph supports:
● REST APIs with RAML and OAS specifications
● Up to 250 APIs per unified schema
● Up to 16,000 fields per unified schema
● A query depth of 100 nested levels
● A 30 second timeout per query

DataGraph - Network Architecture
Fully managed service within the default region of your business group

Query Types & Mutations
Supported:
● Query type - representing
GET method
● Mutations - for add, update
and delete operations (ie.
POST, PATCH, PUT & DELETE)
○ Added in March 2022
● Level-1 and Nested Types
● Enum Types
● Union Types

DataGraph - Security
DataGraph to underlying APIs
Supported Authentication Methods:
● No Auth (Use if your API is public)
● Basic Auth
● Pass-through authentication
● Client ID enforcement via
○ headers
○ query parameters
● OAuth 2.0 Client Credentials
● Custom
You can also set up mutual authentication (mTLS) between your APIs and DataGraph

DataGraph - Security
API consumer requesting access to DataGraph
Supported DataGraph Policies:
● Client ID enforcement
● SLA based rate limiting
Additional security policies on
the roadmap

Scaling & Performance
By default, DataGraph supports a
maximum of 1 million annual API
calls and 150 concurrent or
ongoing calls per uniﬁed schema
per environment.
This can be customised but you
must allocate the correct number of
vCores for your business group or
organization before scaling.
You can get a courtesy license to
test performance.

Demo

What you will see in the demo...
REST and GraphQL to
GraphQL
Reuse REST APIs and GraphQL APIs as GraphQL
interfaces auto-magically
{ REST : API }
Merge
Datatypes
Stitch together data
from diﬀerent system
APIs as a single type
Link
Datatypes
Create a link between
two types using a
primary key
Customers
Name
Email
OrderID
Orders
OrderID
Status
Address
Type 1 Type 2
ProductID
SKU
DisplayName
img_url
Supplier
InStock?
Type 1
Type 2
Merged Type

Next Steps

Try out Anypoint DataGraph for free!
82
DataGraph is a part of the
Anypoint Platform free trial.
Use our tutorial to try it for
yourself!
Use our tutorial on your
trial account!
Anypoint DataGraph is a
part of the Anypoint
Platform free trial. Sign up
for a new trial account!
Sign up for a free trial!
Note: DataGraph will not be available in existing free
trial accounts. You must create a new trial account.

Training & Certiﬁcation
Update
Raquel Paez Ricciardo - Training Advisor ANZ

Build your skills and resume with support from MuleSoft experts & self-paced resources
MuleSoft Expert-Led & Self Paced Training
Expert led sessions
Access ongoing virtual support from
Salesforce instructors
Visit trailheadacademy.salesforce.com/classes
Visit training.mulesoft.com
Access to our most in-demand, self-paced courses
Learn anytime, anywhere with courses that are
continuously refreshed and updated with the latest
oﬀerings

MuleSo Certifications and Credentials
7
Certifications
Validate proficiency
in a specific role
NEW
Specialist Architect Developer Entry-Level

Explore Salesforce Credentials on the Trailhead Academy Credentials page
Salesforce Credentials
trailhead.salesforce.com

Practical Data Mesh: Building Decentralized Data Architectures with Event Stream

Recommended

Recommended

More Related Content

Similar to Practical Data Mesh: Building Decentralized Data Architectures with Event Stream

Similar to Practical Data Mesh: Building Decentralized Data Architectures with Event Stream (20)

Recently uploaded

Recently uploaded (20)

Practical Data Mesh: Building Decentralized Data Architectures with Event Stream