API Community Manager - Andrew Dent

All contents © MuleSoft Inc.
Agenda
2
• Community updates
• Introducing API Community Manager - Andrew Dent
• Troubleshooting Mule apps - Vitalii Mykytenko
• Networking
• Close

What’s next
4
• Share:
– Tweet your pictures with the hashtag #MuleMeetup
– Invite your network to join: meetups.mulesoft.com/sydney
• Feedback:
– Contact MuleSoft at meetup@mulesoft.com for ways to improve the program
and suggest topics

Speakers for Meetup
5
Would you like to present at the next meetup?
Complete this form to submit your idea:
https://forms.gle/Cwks461kz8exKrzi8
Prizes are available for speakers!

All contents © MuleSoft Inc. 6
32

September 3 - 4, 2019 | Hyatt Regency
Vision keynote:
Brent Grimes, SVP Global CS MuleSoft
Product keynote:
Uri Sarid, CTO, MuleSoft
Thought leader:
Frank Liberio, former CIO of McDonald’s
David Berlind, Editor-in-Chief, ProgrammableWeb
Breakout tracks mapped to your
journey with MuleSoft
Get started
Expand your knowledge
Take a deep dive
Achieve business outcomes
Pre-conference training, certification exams, Expo Hall, networking events

September 3 - 4, 2019 | Hyatt Regency
Register for CONNECT using this promo code:
dev-syd-cn19
to receive a free Duffle Bag on the day!

9
Register now at meetups.mulesoft.com/sydney

https://connect.mulesoft.com/events/connect/sydney/training-certification
11

On the day
12

Introducing API
Community Manager
Andrew Dent, MuleSoft

Troubleshooting Mule apps
Vitalii Mykytenko, MuleSoft

Speakers for Meetup
15
Would you like to present at the next meetup?
Complete this form to submit your idea:
https://forms.gle/Cwks461kz8exKrzi8
Prizes are available for speakers!

Tuesday 3 September at CONNECT
Register now at meetups.mulesoft.com/sydney
Next Sydney Meetup:

Key characteristics of a mature API strategy
Only 15% of IT
leaders report their
organization has a
mature API strategy
mandated across the
entire company.
Used to
build
integrations
Built into dev
process for
new projects
Managed like
products
Reusable
Monetized
Discoverable
Exposed to
3rd parties

The API economy: It’s game time
$1 trillion in total economic profit globally
could be up for grabs through the
redistribution of revenues across sectors
within API ecosystems.
McKinsey & Company:What is really takes to capture the value of APIs
https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/what-it-really-takes-to-capture-the-value-of-apis

Developers are the new customer
Forums
Support
Docs
APIs
Case
Studies
Tutorials
Events
User
Groups

API portals the new battleground
Uber: Personalized journeys
Twilio: Self-service support
Stripe: Tools for developer
success
NordeaBank: Gathering
developer input into roadmap

...but it takes a village
Capabilities Coordinated teams
• Full lifecycle API management
• Content management system
• Personalization
• Forums
• Chat
• Support case management
• Engagement analytics
• Central IT
• API product managers
• Content/brand marketing
• Developer relations
• Technical writers
• Support professionals
• Business development

Introducing API Community Manager
Engage an ecosystem of
developers and partners
● Start from preconfigured
templates to quickly drive
API product adoption
● Personalize experiences
with clicks not code
● Engage consumers through
forums, chat and case
management
● Track API program KPIs in
real-time

Critical capabilities
Out-of-the-box themes Drag-and-drop builder Personalized portals
Interactive docs Forums & Cases Engagement Analytics

Extending the capabilities of Anypoint Exchange
11
C4E Architect
API
Developer
Technical
Writer
API Consumer
Marketing
manager
API Product
Manager
Brand/Graphic
designer
Exchange services
API Community Manager
- Branded, personalized
experiences
- Forums, blogs and events
- Support case management
Exchange API portals
- Auto-generated
- Searchable catalog
- Lightweight branding

Leapfrogging the competition
Full Lifecycle API Management Digital Experience Platforms

MuleSoft’s approach combines technology and methodology

Catalyst Mobilize
COLLABORATIVE VALUE ASSESSMENT
API PROGRAM WORKSHOPS
API STRATEGY BLUEPRINT

Vitalii Mykytenko
August 2019
Troubleshooting Application
Performance

• 19+ years working in IT
• 5+ years working with heavily loaded near real-time large scale
environments processing billions of records daily
• Joined MuleSoft as a Customer Architect in November 2017
• Specialising in Performance Troubleshooting
Introduction
2

All contents © MuleSoft Inc.All contents © MuleSoft Inc.
Agenda
3
1. Memory Distribution
– Virtual vs Physical
– Fragmentation
– Swap purpose
– JVM memory key consumers
2. Deploying Applications. Common Problems
– On-Premise
– CloudHub
3. Useful tools and JVM arguments
4. Takeaways
5. Q&A
6. Additional Materials

Memory Distribution. Virtual vs Physical
● Processes never works with physical
memory directly
● Own VIRTual address space for every
process
● Process can allocate more than an actual
physical memory is available
● top command:
○ VIRT: Addressed virtual space
○ RES: Actual size in physical memory
swap
0x0000
0x4a00
...
0xXXXX
Process X
Virtual Memory X Page Table
Virtual
Address
Physical
Address
0x0000 0x0000
0x4a00 0x5000
...
0xXXXX 0xZZZZ
0x0000
0x2000
...
Physical Memory
0x5000
...
0x7000
...
0xYYYY
0xZZZZ
0x0000
0x4a00
...
0xXXXX
Process Y Virtual
Memory Y Page Table
Virtual
Address
Physical
Address
0x0000 0x1000
0x4a00 0x6000
...
0xXXXX 0xYYYY
Kernel

Memory Distribution. Fragmentation
Memory fragmentation
● External fragmentation
● Internal fragmentation
payload1(3)
payload2(2)
free(payload2)
payload4(6) ?
payload3(4)
Payload
Memory Block
Internal
fragmentation

Memory Distribution. Swap
– If I increased the SWAP partition does it mean a
system has more memory to process data?
– Cache never stored in SWAP
– Active memory pages never stored in SWAP
– Can be controlled using vm.swappiness parameter
Key Difference: Virtual memory is an abstraction of
the main memory. It extends the available memory
of the computer by storing the inactive parts of the
RAM content on a disk. Whenever the content is
required, it fetches it back to the RAM. Swap
memory or swap space is a part of a disk drive that
is used for virtual memory. Thus, both are also used
interchangeably
page E page K
page A
page B
Physical Memory
page D
E
page F
page G
page H
page I
page E page K
page C
swap in
swap out
swap/page
file

Memory Distribution. JVM
JVMProcess
Heap: 1GB
New Generation: 512MB
Old Generation: 512MB
Native Memory: Variable
Metaspace: 256MB
Thread Stack: 1MB/thr,
160-180 threads
Code Cache: up to 240MB
by default. Usually
considerably less
Internal: used by NIO
buffers, streaming, etc. By
default has no explicit limit
Others: GC, Compiler,
Symbol, Arena Chunk...
Fig. Typical memory distribution for an
application deployed to 1GB Heap runtime
Recommendations
● Always reserve 50% of physical memory free
when calculating memory required for a runtime
server. Caveat: Maybe different on a large
server
● Add -XX:NativeMemoryTracking=summary
to wrapper.conf for an enhanced monitoring as
even Java Profilers do not show all allocations

Memory Distribution. JVM. Example

Common problems
Deploying Applications. On-premise
9
● Misconfigured conf/wrapper.conf. It is
recommended to keep 50-50 proportion,
i.e. 50% of memory for Heap and 50% of
Heap for New Generation area. Leave the
rest of native memory occupiers and OS
# Java Heap Size (in MB)
wrapper.java.initmemory=1024
wrapper.java.maxmemory=1024
wrapper.java.additional.11=-XX:NewRatio=1 or
wrapper.java.additional.12=-XX:NewSize=512m
wrapper.java.additional.13=-XX:MaxNewSize=512m
● System limits not configured. Especially
when running as a service. $ ulimit -a
$ cat /proc/RUNTIME_PID/limits
Max processes 65000 65000 processes
Max open file 65000 65000 files
● Shared resource domain does not consider
an increased number of applications
deployed
● The number of applications deployed per
runtime is too high. This leads to
complicated troubleshooting
The high limit of apps per runtime is 200. However, it is
recommended to keep this limit ~20 apps per runtime
● GC happens to often consuming high CPU Enable verbose GC tracking. Usually NewGeneration heap area are was
forgot to be reconfigured from default 512MB
● HEAP OOME # GC settings
wrapper.java.additional.<n>=-XX:+HeapDumpOnOutOfMemoryError

Common problems
Deploying Applications. CloudHub
10
● Application works fine in Studio
but crashes in CloudHub on a
fractional vCore worker
Memory fragmentation behaviour is different between running
from Studio and in CloudHub
● Custom code Memory and Thread leaks. Fine locally but reveals issues
when resources are limited, i.e. CloudHub
● Burstable resources of fractional
vCore workers
Unlike running on-premise, if an application runs for a
prolonged period of time at 100% CPU utilisation eventually,
performance will start suffering
● Verbose logging Was discussed on a previous meetup. Any verbose logging when
deploying to CloudHub needs to be omitted. Adaptive logging
using categories is advised
● Outdated Runtime Release CloudHub Runtimes are updated on
a regular basis to include all
improvements and bug fixes.
No runtime version upgrade occurs

Preparing
11
● Limit HEAP memory in Studio -Xms512MB
-Xmx512MB and
-NewRatio=1
● Native Memory Tracking Will allow to estimate the application size
-XX:NativeMemoryTracking=summary
● Limit logging verbosity Verbose logging plays significant role in fractional vCore
worker resource utilisation
● Consider limited disk space ~5GB free for the smallest workers
https://docs.mulesoft.com/runtime-manager/deploying-to-cloud
hub
● Use Java Profiler and Memory
Analyser Tools
VisualVM, YourKit, JConsole etc. Will allow to easily find
memory and thread leaks. Eclipse Memory Analyser Tool will
show most obvious leak suspects straight away

Alternative Approach
12
• Create a VM with 1 GB of memory and limited
number of CPU cores
• Add tiny ~256-512MB SWAP partition
• Configure your JVM to use 50% of memory available
for heap space
wrapper.java.maxmemory.percent=50 and a
half for New Generation -XX:NewRatio=1
• Enable native memory summary tracking
• Deploy an application and conduct testing there

Useful Tools and Commands. Linux
13
top -d1 -c update interval in seconds, -c – show full command, -d -
update interval, -H to show process threads, Shift+N to
change sorting mode
sar 1 1000 sysstat is powerful package to monitor all OS resources, e.g.
CPU utilisation breakdown (user, system, io wait, etc.), io,
swapping -W, paging -B, memory -r
vmstat 1 1000 virtual memory, io, swapping, CPU breakdown stats (similar to
sar)
lsof -p <PID> check open file descriptors for a process
/proc/meminfo memory statistics at a moment of time (next slides)
/proc/cpuinfo a quick way to check number of CPU cores
/proc/<PID>/limits a proper way of checking process limits, i.e. ulimit -a
/proc/<PID>/status running process status, memory, threads, etc. (next slides)
netstat -tulpn check network ports open
kill -3 <PID> generate thread dump in mule_ee.log
vm.swappiness by default is 60. Controls when kernel prefers swapping over
in-memory defragmentation
/proc/meminfo
MemFree: 82072 kB
MemAvailable: 234700 kB
Buffers: 100928 kB
Cached: 165108 kB
SwapCached: 0 kB
Active: 667976 kB
Inactive: 164084 kB
Active(file): 148080 kB
Inactive(file):117900 kB
SwapTotal: 262140 kB
SwapFree: 262140 kB
VmallocTotal: 34359738367 kB
/proc/<PID>/status
Name: java
State: S (sleeping)
VmPeak: 3301880 kB
VmSize: 3167708 kB
VmHWM: 849720 kB
VmRSS: 692832 kB
VmData: 1263096 kB
VmSwap: 294144 kB
Threads: 123

Useful Tools and Commands. Java
14
-Xms1024MB
-Xmx1024MB or
wrapper.java.initmemory=1024
wrapper.java.maxmemory=1024 or
wrapper.java.initmemory.percent=50
wrapper.java.maxmemory.percent=50
Controls minimum and maximum heap size
-NewRatio=1 or
-XX:NewSize=512m
-XX:MaxNewSize=512m
Controls New Generation size
-XX:+HeapDumpOnOutOfMemoryError Generated heap dump on out of memory error
-XX:NativeMemoryTracking=summary Allows to track native memory occupiers
jcmd <PID> VM.native_memory Shows native memory consumption
jcmd <PID> GC.run Perform full garbage collection
jstack -l <PID> Generate a thread dump
jmap -dump:format=b,file=<APP_NAME>.hprof <PID> Generate a heap dump
jps Java process list
JConsole, JMC, VisualVM, YourKit, Eclipse MAT Java management and monitoring visual tools (next slide)

Useful Tools and Commands. Java Visual Tools
15
YourKit
VisualVM
JConsole
Eclipse MAT

Support Information Collector and Memory Leaking Monitoring
Useful Tools and Commands. MuleSoft
16
https://support.mulesoft.com/s/article/Support-information-collector
https://support.mulesoft.com/s/article/scripts-for-memory-leaking-issues-linux

Takeaways
17
• Size the runtime properly. Leave space for native memory
occupiers. Follow a 50-50 thumb rule
• Check how much JVM spends on GC. Maybe it’s time to split
runtimes or add server resources
• Ensure no leaks in custom code. Use profilers
• Estimate application memory footprint by enabling native
memory tracking
• Reconsider logging volume in production
• Consider shared domain resources when adding applications
• Fine tune server vm.swappiness. Use sar -B or vmstat
• Ensure runtime starts with enough processes and open file
limits
• Always update connectors to the latest versions
• Always update CloudHub Runtime patch releases. No runtime
version upgrade occurs. Can be rolled back via ARM UI

Q & A

Additional Materials
19
Resources
• https://support.mulesoft.com/s/knowledge
• https://support.mulesoft.com/s/article/The-Impact-of-The-Excessive-Verbo
se-Logging-on-Application-Performance
• https://support.mulesoft.com/s/article/How-to-enable-GC-logs-without-a-r
untime-restart
• https://support.mulesoft.com/s/article/How-to-monitor-JVM-native-memor
y-usage-by-enabling-native-memory-tracking
• https://opensource.com/article/18/9/swap-space-linux-systems
• https://support.mulesoft.com/s/article/Support-information-collector
• https://support.mulesoft.com/s/article/scripts-for-memory-leaking-issues-li
nux
• http://www.differencebetween.info/difference-between-virtual-memory-an
d-swap-memory
• https://aticleworld.com/problems-with-dynamic-memory-allocation/
• https://www.kernel.org/doc/Documentation/sysctl/vm.txt
• https://unix.stackexchange.com/a/185255/291885
• https://ewx.livejournal.com/579283.html
• https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tool
descr007.html
• https://docs.oracle.com/javase/8/embedded/develop-apps-platforms/code
cache.htm
• https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html
Populated on
a daily basis

API Community Manager - Andrew Dent

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to API Community Manager - Andrew Dent

Similar to API Community Manager - Andrew Dent (20)

More from Royston Lobo

More from Royston Lobo (10)

Recently uploaded

Recently uploaded (20)

API Community Manager - Andrew Dent