ActiveMQ Performance Tuning

© 2012 FuseSource Corp.All rights reserved.1

Apache
Ac(veMQ
Performance
Tuning

Chris(an
Posta
Senior
Consultant
and
Architect

Blog:
h6p://chris9anposta.com/blog/

Twi6er:
@chris9anposta

GitHub:
h6ps://github.com/chris9an-‐posta


ceposta

§ Chris9an
Posta

§  Blog:
h6p://chris9anposta.com/blog/

§  Email:
ceposta@apache.org

§  Twi6er:
@chris9anposta

§  GitHub:
h6ps://github.com/chris9an-‐posta

§  Goolge+:
h6ps://plus.google.com/103810850748178017238

§ Senior
Consultant
and
Architect
at
Red
Hat
(formerly

FuseSource)

§ Commi6er
on:
Ac9veMQ,
Apollo

§ Author
of
Essen%al
Camel
Components
DZone
Refcard


What
Am
I
Talking
About?

§  What
is
Ac9veMQ?

§  Performance/Benchmarking
Tools

§  Tuning


Apache
Ac(veMQ

§  OpenSource
Messaging
Server

•  Queues,
Topics,
persistent
messaging,
high

availability,
etc

§  Apache
v2.0
License

§  Ac9ve
community,
Mature,
Stable

§  Used
at
top
companies

•  Insurance,
banking,
retail,
ecommerce,
health

care,
avia9on,
shipping,
et.
al!

§  5.8.0
latest
release


Apache
Ac(veMQ

§  High
performance

§  High
Availability

§  Light-‐weight

§  Mul9-‐protocol

§  JMS
compliant

§  Supported
in
Produc9on


Accessible!

§  Java

§  C/C++
h6p://ac9vemq.apache.org/cms/

§  .NET
h6p://ac9vemq.apache.org/nms/

§  PHP

§  Python

§  Ruby

§  PHP

§  JavaScript

§  Telnet!

§  Any
that
can
make
a
TCP
connec9on
and
send
Text!


Transports

§  TCP

§  UDP

§  SSL

§  NIO

§  HTTP/s

§  VM

§  WS
(Web
Sockets)

§  WSS
(Secure
Web
Sockets)


Wire
Protocols

§  Openwire

h6p://ac9vemq.apache.org/openwire.html

§  Simple
Text
Oriented
Messaging
Protocol

(STOMP)

h6p://stomp.github.io

§  AMQP
1.0

h6p://www.amqp.org/resources/speciﬁca9ons

§  MQTT

h6p://mq6.org

§  HTTP/REST


Ac(veMQ
Features

§  Master/Slave
fault
tolerance
and
High
Availability

§  Broker
“networking”
or
clustering

§  Pluggable
persistence
(KahaDB,
LevelDB,
JDBC)

§  Broker
interoperability
(RabbitMQ,
HornetQ,
etc)

§  Virtual
Topics

§  Mirrored
Queues

§  JMX
Monitoring


When
to
use
Messaging?

§  Asynchronous
communica9on/integra9on

§  Durability

§  Loose
coupling

§  Heterogenous
integra9on

§  Real-‐9me
data


Conﬁgura(on

Ac9veMQ
is
Highly
Conﬁgurable!!


Performance/Benchmarking

Tools


So
What
Are
We
Covering?

§  Tools

§  How
to
approach
performance
tuning

§  Areas
to
tweak!


What
are
Your
Objec(ves!?

§  Please,
please,
please..
Know

what
you’re
trying
to

accomplish!

§  Know
your
use
cases!

§  Know
your
hardware/OS

§  Understand
all
of
the
broker

conﬁg
changes
you
make!

§  VALIDATE
YOUR
CHANGES!


Broker
Benchmarking
Tools

§  Ac9veMQ
Performance
Module

§  jms-‐benchmark

§  JMSTester

§  Apache
Jmeter

§  Grinder

§  …?


Ac(veMQ
Performance
Module

§  Part
of
the
Ac9veMQ
Build

h6p://ac9vemq.apache.org/ac9vemq-‐performance-‐module-‐users-‐manual.html

§  Maven
plugin

§  Quick
and
easy
to
get
started

§  Flexible

§  Records
throughput,
basic
sta9s9cs,
and
CPU

metrics


jms-‐benchmark

§  Opensource
at
github,
from
Hiram
Chirino
h6p://github.com/chirino/jms-‐benchmark

§  Comprehensive
benchmarks

§  Conﬁgurable

§  All-‐on-‐one

§  Mul9ple
brokers

§  Pre6y
graphs!


JmsTester

§  From
FuseSource

h6p://jmstester.fusesource.org

§  My
branch:

h6ps://github.com/chris9an-‐posta/jms-‐tester

§  Allows
more
complicated
load
tes9ng

§  Distributed

§  Records
messaging
throughput
and
CPU,
Memory,

Network
IO,
Disk
IO,
et
al.

§  Thorough
documenta9on

§  My
blog
post:

h6p://www.chris9anposta.com/blog/?p=268


JMeter

§  Swiss
army
knife
of
load
tes9ng

§  From
Apache

h6p://jmeter.apache.org

§  JMS,
HTTP,
TCP,
SOAP,
JDBC,

and
many
others

§  Many
aggregators
and
repor9ng
features

§  Mature
product,
well
known


OS
Tools

§  Lest
we
forget…
rely
heavily
on
the
tools
your
Opera9ng

System
exposes!

§  Linux

•  top

•  iostat

•  netstat

•  vmstat

§  Windows

•  Built
in
process,
resource
monitors


Ac(veMQ
Performance
Module

§  Maven
plugin

§  Can
use
exis9ng
project:
h6ps://svn.apache.org/repos/asf/ac9vemq/sandbox/ac9vemq-‐persest

§  Spin
up
producers,
consumers

§  Control
number
of
threads,
test
dura9ons,
etc

§  Stats
and
Repor9ng

•  Throughput,
min/max,
broken
down
per
client

plugin

groupIdorg.apache.ac9vemq.tooling/groupId

ar9factIdac9vemq-‐perf-‐maven-‐plugin/ar9factId

version${ac9vemq-‐version}/version

/plugin


Using
Performance
Module

§  Example:

§  Out
of
the
box
examples:

§  Other
out
of
the
box
proﬁles:

•  Producer

–  AMQ-‐Prod-‐1-‐1-‐queue-‐nonpersistent.proper9es

–  AMQ-‐Prod-‐10-‐1-‐queue-‐nonpersistent.proper9es

–  AMQ-‐Prod-‐10-‐10-‐topic-‐persistent.proper9es

•  Consumer

–  AMQ-‐Cons-‐1-‐1-‐queue.proper9es

–  AMQ-‐Cons-‐10-‐1-‐topic-‐durable.proper9es

–  AMQ-‐Cons-‐10-‐10-‐queue.proper9es

user@computer mvn activemq-perf:producer –Dfactory.brokerURL=tcp://localhost:61616
user@computer mvn activemq-perf:producer  
–DsysTest.propsConfigFile=AMQ-prod-1-1-queue-persistent.properties


What
Broker
to
Use?

§  Load
up
an
external
broker

•  Exis9ng
broker

•  No
maven
access

•  Diﬀerent
machine
(good
idea!)

§  Load
one
from
the
perf
module

•  Tests
and
broker
conﬁg
all
in
one

•  Simple
maven
command

user@computer mvn activemq-perf:broker –Durl=tcp://localhost:61616


Defaults

§  Overall

•  tp,cpu
reports

•  5
min

•  30s
warm
up
30s
cool
down

•  1s
sample
rate

§  Producer

•  Non-‐persistent

•  Auto
ack

•  1K
message
size

§  Consumer

•  Non
durable
subscrip9on

•  Auto
ack

•  No
tx

Copyright
©
2011

Progress
Sosware
Corpora9on
and/or
its
subsidiaries
or
aﬃliates.
All
rights
reserved.

A
Progress
Sosware
Company
FuseSource
Conﬁden9al
25

Quick
Demo!


Results
explana(on


While
Running…

§  Use
JConsole!

§  A
more
powerful
toolkit?
YourKit?

§  Rely
on
OS
monitoring,
top,
iostat,
etc

Copyright
©
2011

Progress
Sosware
Corpora9on
and/or
its
subsidiaries
or
aﬃliates.
All
rights
reserved.

A
Progress
Sosware
Company
FuseSource
Conﬁden9al
28

Tuning
Ac(veMQ


Approach

§  Know
your
requirements

§  Use
real
hardware,
real
scenarios,
real
se}ngs

§  Know
what
conﬁg
changes
do!

§  Tune

§  Run
tests

§  Capture
results

§  Repeat

§  Change
only
one
se}ng
at
a
9me!


Where
are
the
BoPlenecks?

§  Overall
system

§  Network
latencies

§  Disk
IO!

§  Threading
overheads

§  JVM
op9miza9ons


Network

§  Mbits/Gbits
per
second
throughput

§  Topology

• Hops

• Switches

• Routes

• Gateways

§  Firewalls

§  Internal/External
clients


Disk

§  Read/Write
speeds

§  Mechanical
vs
SSD

§  Shared
disks,
SAN,
NAS,
NFS,
etc

§  Ac9veMQ
DiskBenchmark:

Benchmarking:
/Users/cposta/temp/test.dat

Writes:

159232
writes
of
size
4096
wri6en
in
10.244
seconds.

15543.928
writes/second.

60.718468
megs/second.

Sync
Writes:

33213
writes
of
size
4096
wri6en
in
10.001
seconds.

3320.968
writes/second.

12.972531
megs/second.

Reads:

5160332
reads
of
size
4096
read
in
10.001
seconds.

515981.6
writes/second.

2015.5531
megs/second.

user@computer java -classpath lib/activemq-kahadb-store-5.8.0.jar
org.apache.activemq.store.kahadb.disk.util.DiskBenchmark
/Users/cposta/temp/test.dat


Tune
What
Needs
Tuning!

The following aren’t things you must or need do…

they are things to consider and can be worth tuning.

Your
speciﬁc
use
cases
will
illuminate
what
needs
tuning


Transport
Tuning

§  TCP
Tuning

•  Timeouts
(Handshake,
Teardowns)

•  Conges9on
avoidance,
sliding
windows

•  Send/Receive
Buffers!!

•  Default
64K,
but
increase
to
Bandwidth
Delay
Product

–  Buffer
=
Bandwidth
*
RTT

h6p://www.speedguide.net/bdp.php

–  See
h6p://en.wikipedia.org/wiki/TCP_tuning

§  Configure
the
socket:

§  transport.*

§  socket.*

transportConnector
name=openwire

uri=tcp://0.0.0.0:61616?maximumConnec9ons=1000

wireformat.maxFrameSize=104857600transport.socketBufferSize=131072
/


Transport
Tuning…

§  Buffering
between
Socket
and
Protocol
Codec

§  IOBufferSize

•  Default
8k

transportConnector
name=openwire

uri=tcp://0.0.0.0:61616?maximumConnec9ons=1000

wireformat.maxFrameSize=104857600transport.ioBufferSize=65536
/


Wire
Format
Tuning

§  OpenWire
(default
protocol)

•  wireFormat.tcpNoDelayEnabled
–
whether
or
not
to
use
Nagle’s

batching
algorithm
(default:
true
..that
means
no
Nagle)

•  wireFormat.cacheSize
–
how
many
command
object
constants
to

cache
(default:
1024)

•  wireFormat.cacheEnabled
–
turn
cache
on/oﬀ
(default:
true)

•  wireFormat.9ghtEncoding
–
reduce
network
load
(default:
true)

•  wireFormat.maxFrameSize
–
message
sizes
(default:

Long.MAX_LONG)

•  wireFormat.maxInac9vityDura9on
–
monitors
period
of
inac9vity

(default:
30000ms,
set
to
0
to
disable))

transportConnector
name=openwire

uri=tcp://0.0.0.0:61616?wireFormat.tcpNoDelayEnabled=false
/


Tuning
Clients

§  Compression

•  ?jms.useCompression=true

•  jms.*
conﬁgures
underlying
connec9on
factory

•  Uses
java.u9l.zip.Deﬂater

§  Encoding

•  JMS
Message
types

– Object

– Bytes

– Text

– Stream

– Map

transportConnector
name=openwire”
uri=tcp://0.0.0.0:61616?jms.useCompression=true
/


Tuning
Producers

§  Producer
Flow
Control

•  Persistent
vs
non
persistent
messaging

•  Broker
vs
Client

•  Broker
memory,
client
producer
window

•  Default
sync
send
for
persistent,
async
send
for
non-‐persistent

and
transac9ons
(except
for
commit/rollback
opera9on)

•  En9re
connec9on
flow
control,
individual
des9na9on
flow
control,

TCP
flow
control

•  Disable
flow
control

§  jms.alwaysSyncSend

§  jms.useAsyncSend

§  Send
fail
if
no
space

policyEntry
queue=“queueWildcard”
producerFlowControl=“false”
/


Failover

§  Use
failover
transport
for
automa9c
reconnects

§  backup

§  priorityBackup

§  trackMessages
(default
false)

§  backOﬀMul9plier

§  maxReconnectA6empts

§  randomize

§  Can
automa9cally
get
updates
from
broker
when
cluster

situa9on
changes

failover:(tcp://primary:61616,tcp://secondary:61616)?randomize=false


Other
Producer
Parameters

§  jms.watchTopicAdvisories=false

§  jms.copyMessageOnSend

§  Batch
using
transac9ons


Tuning
Consumers

§  Prefetch

•  Queue
–
1000

•  Queue
Browser
–
500

•  Topic
–
32K

•  Durable
Topic
–
100

§  Broker
side
memory

§  Client
side
memory

§  Consumer
starva9on


Prefetch
SeXngs

§  Broker

§  Consumer

§  Des9na9on

broker
...

des9na9onPolicy

policyMap

policyEntries

policyEntry
queue=queue.
queuePrefetch=”1”/

policyEntry
topic=topic.
topicPrefetch=”1000”/

/policyEntries

/policyMap

/des9na9onPolicy

/broker

jms.prefetchPolicy.queuePrefetch=100

new
Ac9veMQQueue(TEST.QUEUE?consumer.prefetchSize=10)


Consumer
ACK
Modes

§  Session.AUTO_ACKNOWLEDGE

§  Session.CLIENT_ACKNOWLEDGE

§  Session.DUPS_OKAY_ACKNOWLEDGE

§  Session.SESSION_TRANSACTED

§  Ac9veMQSession.INDIVIDUAL_ACKNOWLEDGE

§  jms.op9mizedAcknowledge
with
AUTO_ACK

§  Note
the
diﬀerences
with
DUPS_OKAY_ACKNOWLEDGE

when
using
topics
or
queues.


Other
Consumer
Parameters

§  jms.alwaysSessionAsync
reduce
context
switching

§  MessageListener
faster

§  Spring
DMLC:
Cache
consumers
and
connec9ons!


Tuning
Broker
Core

§  Keep
an
eye
on
number
of
threads

•  UseDedicatedTaskRunner
–
use
a
dedicated
dispatch
thread
for

queues

•  op9mizedDispatch
–
don’t
take
the
penalty
of
context
switch,
just

let
the
transport
thread
do
the
dispatch

•  asyncDispatch
–
don’t
take
the
penalty
of
context
switch,
let
the

dispatching
thread
send
to
the
consumer’s
transport

•  NIO
connector
nio://localhost:61616


Broker
Resource
SeXngs

§  MemoryUsage
se}ngs
for
messages

§  Caching
for
persistent
vs
non-‐persistent

§  Policy
se}ngs,
memory
limits
for
des9na9ons,
cursor
high

water
mark,
etc

§  Temp
usage
for
oﬄining
non-‐persistent
messages

§  Use
the
right
combina9on
of
memory,
producer
ﬂow

control,
and
message
cursors


Conﬁgura(on
of
Resources

!–
defaults
-‐-‐

systemUsage

systemUsage

memoryUsage

memoryUsage
limit=64
mb
/

/memoryUsage

storeUsage

storeUsage
limit=100
gb
/

/storeUsage

tempUsage

tempUsage
limit=10
gb
/

/tempUsage

/systemUsage

/systemUsage

policyEntry
queue=“queueWildcard”
memoryLimit=“10M”
/


KahaDB

§  Current
messaging
database

§  Fast,
op9mized
for
messaging

§  Journal
+
Index
+
write
ahead
log

§  Fast
recovery

§  Fewer
ﬁle
descriptors


Tuning
KahaDB

§  What
levels
of
reliability
must
you
have?

§  Durability
vs
throughput

§  fsync
vs
ﬄush

§  Areas
of
tuning

•  Concurrent
store
and
dispatch

•  Index
paging,
caching,
page
size,
sync,
checkpoin9ng

•  Journal
sync,
ﬁle
length,
checkpoin9ng


KahaDB
Journal
SeXngs

§  enableJournalDiskSyncs
(default:
true)

§  cleanupInterval
(default:
30000ms)

§  checkForCorruptJournalFiles
(default:
false)

§  journalSize
(default:
32MB)

broker
brokerName=broker
...

persistenceAdapter

kahaDB
directory=ac9vemq-‐data
journalMaxFileLength=32mb”

cleanupInterval=“150000”
checkForCorruptJournalFiles=“true”/

/persistenceAdapter

...

/broker


KahaDB
Index
SeXngs

§  How
osen
the
index
writes
updates

•  checkpointInterval
(default:
5000),
indexWriteBatchSize
(default:

1000)

•  Can
amor9ze
cost
across
small
writes?
Or
batch
them
up
for

larger
writes

§  Size
of
the
index
in
memory

•  indexCacheSize
(default:
10000)

§  Sync
to
disk

•  enableIndexDiskSyncs
(default:
true)

broker
brokerName=broker
...

persistenceAdapter

kahaDB
directory=ac9vemq-‐data
journalMaxFileLength=32mb”

indexCacheSize=“40000”
enableIndexDiskSyncs=“false”/

/persistenceAdapter

...

/broker


Tradeoﬀs

§  Point
to
point,
publish
subscribe

§  Levels
of
durability

§  Duplicate
messages

§  Security

§  Throughput

§  Priority


Summary

§  Ac9veMQ
is
a
highly
tunable,
conﬁgurable
piece
of
server

sosware

§  Know
your
use
cases
and
requirements

§  Use
tools
to
not
only
load
the
broker,
but
monitor
the

bo6lenecks

§  Tune
for
producers/consumers,
broker
core,
network,
and

disk
IO

§  Ask
the
computer!


Useful
Links

§  Apache
Ac9veMQ

h6p://ac9vemq.apache.org/

§  JMS
Benchmarks

h6p://github.com/chirino/jms-‐benchmark

§  Ac9veMQ
Performance
module
tools:
h6p://svn.apache.org/repos/asf/ac9vemq/sandbox/
ac9vemq-‐persest

§  Lots
of
blogging
about
Ac9veMQ
J

h6p://chris9anposta.com/blog

ActiveMQ Performance Tuning

ActiveMQ Performance Tuning

Recommended

More Related Content

What's hot

What's hot(20)

Similar to ActiveMQ Performance Tuning

Similar to ActiveMQ Performance Tuning(20)

More from Christian Posta

More from Christian Posta(20)

Recently uploaded

Recently uploaded(20)

ActiveMQ Performance Tuning