Power of Transformation with DataWeave 2.X Engine

21st September,2019: The Power of Transformation with Dataweave
Engine 2.0 & MuleSoft Catalyst
Mumbai - India

All contents © MuleSoft Inc.
Agenda
2
2:00 PM Introduction
2:30 PM Technical Session – DataWeave 2.0
3:30 PM Technical Session – DataWeave 2.0
4:30 PM Networking & Break: Tea, Coffee and Snack
5:00 PM Technical Sessions – MuleSoft Catalyst
5:30 PM Wrap-up and Plan For Next Meetup

Introductions
3
• About the organizer:
– Sudeshna Mitra
– Manish Yadav
– Akshata Sawant
– Ranveer Meel
• About the sponsor:
– ACCENTURE & APISERO
A SHOW OF HANDS:
Who is new to this MeetUp?

Speakers
4

Surprises!
5
• Thanks for your positive
enthusiasm …You earned
it ..
• Certification Vouchers
for all!
• Be geared up for the
quiz sessions at the end
of each modules! 

Agenda
6
 DataWeave 2.0
– Introduction
– Quick Walkthrough
– Language
– Map, Filter and Reduce
 MuleSoft Catalyst

Introduction
8
• DataWeave 2.0 for Mule 4 apps
• Mulesoft Expression Language for
– Accessing Data
– Transforming Data
• Entirely replaces everything that MEL does
• No cryptic transformation file (.grf from data mapper)
• Powerful template engine
• Can transform the data to and from any kind of data
– JSON
– XML
– POJO
– CSV
MEL

Quick Walkthrough
DataWeave 2.0

Transform Component
10

Metadata & Sample Data
11

Targets
12

XML Snippet
13

DWL file vs Inline DWL
14
• DWL File referenced through:
– Transform Message Component, Example: see previous slide
– Mule app expression
<when expression="${file::someFile.dwl}" >
• Inline DWL:
– Transform Message Component, Example: see previous slide
– Mule app expression
<set-payload
value="#[output application/xml --- { myroot: payload } ]" />
<salesforce:parameters ><![CDATA[#[output application/java
---
{"territory" : "MULE"}
]]]></salesforce:parameters>

DWL Types
16
• Array (dw::Core)
– An Array can hold elements of any supported type
• Object (dw::Core)
– Represents an object as a collection of Key-Value pairs
• Boolean (dw::Core)
– true/false
• String (dw::Core)
– Defined by use of single or double quotes
• Number (dw::Core)
– 1 type that supports both: floating numbers and integers
• Date And Time (dw::Core)
– Native data types: Date, DateTime, LocalDateTime, LocalTime, Period, Time, TimeZone

DWL Types - Examples
17

DWL Types – Continued
18
• CData (dw::Core)
• Enum (dw::Core)
• Iterator (dw::Core)
• Regex (dw::Core)
• TryResult (dw::Runtime)
• URI (dw::Core::URL)

DWL Structure
19
• Header
– Defines directive that apply to the body
– Optional
– If present, header and body are separated by three dashes ---
• Body
– Template expression to generate the output
– Mandatory

DWL Header – dw and output
20
• %dw
– DataWeave version
– Optional with default 2.0
– Example: %dw 2.0
• output
– Mime type that uses script output
– Optional with default determined by an algorithm (based on inputs used in the body script)
– Example: output application/json
– Most common values: application/xml, application/java, application/csv, text/plain,
application/flatfile etc..

DWL Header - fun
21
• fun
– Defines function
– Alternate to
variable with lambda expression

DWL Header - var
22
• var
– Defines global variables
– Possible forms:
(a) Constant
(b) Lambda expression (function)
– Usage of using
– Local vs global variables
– Usage of single line comment
– Usage of multi line comment

DWL Header - type
23
• type
– Specify a custom type that can be used in the expression

DWL Header - ns
24
• ns
– Import a namespace
– To deal with application/xml

DWL Header - import
25
• import
– Importing a function module
– Automatic import of dw::Core
– 3 ways to import
– Accessing Java Method
– Custom Modules and mapping

DWL Body
26
• Contains the expression which generates the output structure
• Provides a canonical way to generate output

DataWeave Errors
27
• Formatting vs Coding error
1. Check the errors

DataWeave Errors - Continued
28
2. Change the output format to application/dw
If this generates proper output, then earlier error was – Formatting error
If this gives error, then earlier error was – Coding error

DataWeave Errors - Continued
29
3. Evaluate specific Formatting error and fix this
XML Root Element missing error

DataWeave Selectors
30
Selector Type Syntax Return Type
Single Value .key Value that matches the key
Multi Value .*key Array of values that matches the key
Descendants ..key Array of values that matches any
descendant key
Index [<index>] Value at selected array index
Range [<index> to <index>] Array with values from selected indices
Filter [? (Boolean expression)] Array with values for which expression
returns true

DataWeave Selectors - Continued
31
Selector Type Syntax Return Type
XML Attribute .@attribute? String value of the selected attribute
Is Key Present • .key?
• .@attribute?
Boolean
• True if key present in object
• True if attribute present on xml element
Namespace .key.# Namespace of the selected key

DataWeave Selectors - Example
32

DataWeave Selectors – Example Output
33

DataWeave Implicit Variables
34
Variable Description Fields and Examples
server Operating System on
which the Mule instance is
running
• env
• fileSeparator
• userHome
• ip
mule Mule instance on which
the application is currently
running
• clusterId
• nodeId
• version
app Mule application in context • name
• registry
• workDir

DataWeave Implicit Variables - Continued
35
error Error associated with a
Mule Message Object
• description
• detailedDescription
• cause
• errorType
• errorMessage
authentication Access to auth
information
• principal
• credentials
• Properties
correlationId Correlation Id of the
message being processed
• None
• Example: 0-f77404d0-e699-11e7-a217-38c9864c2f8f

DataWeave Implicit Variables - Continued
36
message Mule Message (Payload
and Attributes) being
processed
• payload
• attributes
payload Body of the current Mule
Message Object
• Depends on the current payload
• #[payload.key]
attributes Metadata of the Mule
Message Object
• #[attributes.uriParams.foo]
• #[attributes.queryParams.bar]
vars Variables of the current
Mule event being
processed
• #[vars.myVar]

DataWeave - Writer/Reader properties
37
Format/
Type
Parameter Type Default Description
JSON/
XML Writer
bufferSize Number 8192 Size of buffer writer
deferred Boolean false Deferred Output
duplicateKe
yAsArray
Boolean false JSON duplicate keys processing
encoding String UTF-8 Output character encoding
indent Boolean true Indent the output
skipNullOn String No Skip JSON: arrays, objects
XML: elements, attributes
Both: everywhere

DataWeave - Writer/Reader properties – Continued
38
Format
/Type
Parameter Type Default Description
JSON/
Reader
streaming Boolean false Used for streaming input
XML/
Reader
nullValueOn String blank If tag with empty or blank text
should be read as Null
Options: empty, none or blank
CSV/
Writer
lineSeparator String Line separator for writing CSV
Example: rn
header Boolean true Output header as first line?
separator String , Character that separates records
from another

DataWeave - Flow Control
39
• do
• if
• if else
• if else-if else

DataWeave – Pattern Matching
40
• Literal

DataWeave – Pattern Matching - Continued
41
• Expression

42
• Data Type

43
• Reg Ex – Capturing Group, Positive/ Negative Look Ahead/ Behind

DataWeave – Mule Runtime Functions
44
• p
• lookup
• Utilize Mule namespace for versions > 4.1.4

DataWeave – Custom Module and Mapping
45
• Custom Module
• Custom Mapping
• Local Alias

DataWeave – Customization .. Continued
46

DataWeave – Read, Write and ReadURL function
47
• ReadUrl – Not Cached

DataWeave – Operators
48

Map, Filter and Reduce
DataWeave 2.0

DataWeave – Map, Filter and Reduce
50
• Function that transform arrays
• As input: an array, and a function
– Higher order functions (a function taking another function as an argument, OR a function that returns
another function)
– The input function for map/filter/reduce define the implementation details of the transformation
• Output: array (map/filter), any (reduce)
• Declarative - implementation of iteration is hidden from the client
• Transformations are non-destructive (they do not modify any input)

Why do a talk on them?
51
• They form the core functionality that you’ll need for data
transformations (removing records, modifying records, general
purpose transformation tool)
• A mastery of map/filter/reduce is essential to writing clear and
efficient transformations
• A lot of other functions that work on transforming arrays are just
flavors of map/filter/reduce
• Declarative - implementation of iteration is hidden from the client
• Gives you a great base to build off of because you need to
understand two key concepts: immutability, and higher-order
functions

Filter – Remove elements
52
• Input: An array and a predicate function (function that returns true
or false)
– Predicate function example: (n) -> (n == 2)
• Output: a new array
• Creates new output array, applying the predicate function to every
element in the input array
• If the predicate returns true, add the value to the output, if the
predicate returns false, ignore it
• The only thing filter can do is create new arrays that contain the
same or less elements than the previous array. It cannot change the
elements.

Filter – Continued
53

Filter – Examples
54
// Filter by content
[1,2,3,4,5] filter (mod($, 2) == 0)
// -> [2, 4]
// Filter by index
[2,4,6,8,10] filter ($$ == 4)
// -> [10]
// Filter by nested content (popular use case)
var arr = [{“name”:”Josh”}, {“name”:”Patryk”}, {“name”:”Max The Mule”}]
---
arr filter $.name == “Patryk”
arr filter ((person, index) -> person.name == “Patryk”)
// -> [{”name”:”Patryk”}]

Map – Transform Elements
55
• Input: an array and a mapping function
• Output: a new array
• The mapping function describes how each element in the array will be
modified before it is added to the output array
• map can create a new array that is the result of a modification of
each element in the input array. map cannot modify the number of
elements returned in the resulting array

Map – Continued
56

Map – Examples
57
// Map using each element
[1,2,3] map ($ + 1)
// -> [2,3,4]
// Map using index
[1,2,3] map ({loc: $$, n: $})
// -> [{loc: 0, n: 1},{loc: 1, n: 2}, {loc: 2, n: 3}]
// lambda expression
[1,2,3] map ((value,index) -> {loc: index, n: value})
// -> [{loc: 0, n: 1},{loc: 1, n: 2}, {loc: 2, n: 3}]
// Map nested values down to single-item array (great for list of IDs)
[{id: 123}, {id: 456}, {id: 789}] map $.id
// -> [123, 456, 789]

Reduce
58
• Input: An array and an accumulator function
• Output: a new Any (object, string, Boolean, null, array, etc.) we will
call this the result
• The accumulator function defines how each element in the input
array will be folded into the result
– The value returned from the accumulator function will be passed to the next iteration
• Reduce is the most general-purpose of the three core functions
• Can implement both map and filter in terms of reduce, so map and
filter are just specialized reduce functions
– Keep this in mind as you write your transformations. Don’t use reduce when a map or filter would
work instead

Reduce – Continued
59

Reduce – Examples
60
// Classic example: Adding numbers
[1,2,3] reduce ((n, acc) -> acc + n)
// -> 6
// Classic example: Adding numbers, $$ = Accumulator, $ = value
[1,2,3] reduce ($$ + $)
// -> 6
// A simple joinBy functionality
[“1”,”2”,”3”] reduce ((str, out) -> out ++ “, ” ++ str)
// “1, 2, 3”
// The same thing wrapped in a helpful function
fun myJoinBy(arr, delim) = arr reduce ((str, out) -> out ++ delim ++ str)

Reduce – Examples Continued ..
61
// Flattening an array into a object (common use case)
var arr = [
{dev: "c6824476-b7e2-4e8a-9281-bdc40663bb93"},
{prod: "7f44b450-18ff-45fa-85f3-8ef5c82b4989"}
]
---
arr reduce ((env, obj) -> obj ++ env)
// -> { ”dev”: “c6824476-b7e2-4e8a-9281-bdc40663bb93”,
// “prod”: “7f44b450-18ff-45fa-85f3-8ef5c82b4989” }
// Same result
arr reduce ($$ ++ $)
// Same result
{(arr)}

62
// Make map/filter code more efficient. This loops twice
[1,2,3,4,5] filter (mod($, 2) == 0) map ($ + 2)
// -> [4,6]
// This loops once and accomplishes the same job
[1,2,3,4,5] reduce ((n, acc = []) ->
if (mod(n, 2) == 0) // Filter replacement
acc + (n + 2) // Map replacement
else
acc)
// -> [4,6]

63
// Implement map:
fun myMap(arr, transformFn) =
arr reduce ((e, acc=[]) -> acc + transformFn(e))
// Implement filter:
fun myFilter(arr, predicateFn) =
arr reduce ((e, acc=[]) ->
if (predicateFn(e))
acc + e
else
acc)

Wrapping map/filter/reduce
64
• Passing functions to functions is nice because of the flexibility, but it’s
not always an ideal API because of that flexibility
– Reader needs to understand map/filter/reduce, and then understand the workings of passed function
• To make functions less flexible, but far more readable, you can do the
following
– Name the functionality that the passed-in function is providing
– Wrap the map/filter/reduce code in a function
– Name that function according to the name you came up w/ in step 1
– Make sure that parameters passes to your new function do not take in a lambda

Example
65
// Let’s take a reduce example from earlier. We start with this:
var arr = [{dev: ”123"}, {prod: ”456"}]
---
arr reduce ((env, obj={}) -> obj ++ env)
// -> {dev: “123”, prod: “456”}
// we will name the function as below from highlighted functionlity
fun flattenToObj(arr) =
arr reduce (kvPair, obj={}) -> obj ++ kvPair)
// this can be done with most functions
fun removeOdds(arr) = arr filter ($ mod 2 == 0)
• This is especially helpful if the functions are going to be reused (put them in a
module!)
• Might not be as helpful if not reused, but you will get a nice readability boost
that comes from effective naming of your functions

Moving Forward
66
• map/filter/reduce are a trio of higher-order functions that form the
core of array transformations
– Filter: Remove elements from the array
– Map: Modify elements from the array
– Reduce: Both of above, as well as transform array to any other type. Use it as a last option
• Map and filter tend to be pretty easy to learn, reduce typically
requires more practice to master
• Master these functions first followed by
– mapObject
– pluck
– groupBy, orderBy, distinct
– remove, flatten, joinBy, splitBy, contains, sizeOf, is

mapObject
67
• Input: An Object and a mapping function
• Output: a new Object
• Process both keys and values as a tuple
• The mapping function describes how each key-value pair in the object
will be modified before its added to the output object
• mapObject can create a new Object that is the result of the
modification of each tuple of the input object
– mapObject can modify the property key / value
– mapObject can remove certain property based on conditions
– mapObject can add certain property based on conditions

mapObject – Examples
68
// Modifying below person object for processing
person mapObject {
(($$) : "*****") if ('$$' startsWith "personal"),
(($$) : $) if ('$$' startsWith "professional"),
("processed" : now()) if ($$$ == 0)
}
// With lambda it yields the same result
person mapObject ((value,key,index) -> {
((key) : "*****") if (key startsWith "personal"),
((key) : value) if (key startsWith "professional"),
("processed" : now()) if (index == 0)
})
// -> {
"personal_name": "*****",
"processed": "2019-09-20T16:15:15.343+05:30",
"personal_lastName": "*****",
"professional_experience": 9.9
}

pluck
69
• Input: An Object and a mapping function
• Output: a new Array
• Process both keys and values as a tuple
• The mapping function describes how each key-value pair in the object
will be modified before its added to the output object
• mapObject can create a new Array that is the result of the
modification of each tuple of the input object
– mapObject can modify the property key / value
– mapObject can remove certain property based on conditions
– mapObject can add certain property based on conditions
• Same as mapObject except that this returns an Array instead of
Object

pluck – Examples
70
// Modifying below person object for processing
person pluck $
// With lambda it yields the same result
person pluck ((value,key,index) -> value)
// -> [
"name",
"last_name",
9.9
]

groupBy, orderBy, distinctBy Usage
71

Introduction
73
As businesses conducting digital transformation, success requires
more than implementing new technology; the right operating model,
organizational structure, and approach to execution are also critical.
To address this, Mulesoft developed MuleSoft Catalyst, a unique
delivery methodology and set of packaged offerings, including
customer success programs, training, best practices, assets, and
accelerators, designed to help you achieve your business objectives.

What does MuleSoft Catalyst deliver?
74
Rooted in Mulesoft’s work with over 1,600
global teams and refined over the past
decade, MuleSoft Catalyst is based on
three core pillars:
• Business outcomes: Define KPIs with
stakeholder alignment.
• Organizational enablement: Ensure
organizational readiness with Anypoint
Platform.
• Technology delivery: Enable team
readiness to build APIs and integrations.

Business Outcomes playbook
75

Technology Delivery
76
.Technology delivery is broken down into two parts:
• Anypoint Platform Playbook
• Projects Playbook

Anypoint Platform Playbook
77

Projects Playbook
78

Organization Enablement
79
Organization Enablement is broken down into three parts:
• Center for Enablement Playbook
• Internal Support Playbook
• Training Playbook

C4E Playbook
80

Internal Support Playbook
81

Training Playbook
82
• Train the initial teams
• Develop the broader training plan
• Launch Experiential Learning Opportunities
• Train Additional Teams
• Conduct Assessment

Take a stand !
83
• Nominate yourself for
the next meetup speaker
and suggest a topic as
well.

What’s next
84
• Share:
– Tweet your pictures with the hashtag #MuleMeetup #MuleSoftMeetup
– Invite your network to join: https://meetups.mulesoft.com/mumbai/
• Feedback:
– Contact your organizers Manish Yadav, Sudeshna Mitra, Akshata Sawant and Ranveer Meel to
suggest topics
– Tweet your organizers at @SudeshnaMitra14, @Manish_Kyadav, @sawantakshata02 and
@meel_Ranveer
– Contact MuleSoft at meetup@mulesoft.com for ways to improve the program
– Follow us on Instagram (mumbai_mulesoftofficialpage)
– Your Feedback is Food for us
• Our next meetup:
– Date: TBD
– Location: Mumbai
– Topic: TBD

Networking time
Introduce yourself to your neighbors!

See you next time
Please send topic suggestions to the organizer

Power of Transformation with DataWeave 2.X Engine

Power of Transformation with DataWeave 2.X Engine

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Power of Transformation with DataWeave 2.X Engine

Similar to Power of Transformation with DataWeave 2.X Engine (20)

More from Manish Kumar Yadav

More from Manish Kumar Yadav (8)

Recently uploaded

Recently uploaded (20)

Power of Transformation with DataWeave 2.X Engine