The document discusses how Heroku leveraged Apache Kafka to realize the vision of an enterprise service bus (ESB). It defines what an ESB is according to analysts and vendors. Heroku defined the API as its ESB but faced bottlenecks and reliability issues. It transitioned to using Kafka with a pull-based architecture for independent development, scalability, and avoiding single points of failure. Heroku now uses Kafka for operational data pipelines and metrics aggregation. It provides examples of using Kafka topics and discusses next steps of implementing a schema registry and security.

1. The Enterprise
Kafka Summit 2016
Rimas Silkaitis

2. how Heroku leveraged Kafka to
realize the original vision of the
enterprise service bus (esb)

3. Rimas Silkaitis
Product

4. Heroku Postgres Heroku Redis

5. What is an ESB?

6. Analyst Definitions

7. ESBs receive service consumption requests,
route the requests to the correct service
provider, transform the requests to a format
compatible with the service provider, wait for
the results, and deliver them back to the
service consumer.
IDC
http://www.idc.com/2010st/esbcm.html

8. Uses a bus architecture for scalability and
reliability, establishes a unified communications
channel, provides integration, mediation and
communication services and is standards
based.
http://redmonk.com/public/OpenSourceESBs.pdf
Redmonk

9. Vendor Definitions

10. Oracle Service Bus transforms complex and
brittle architectures into agile integration
networks by connecting, virtualizing, and
managing interactions between services and
applications.
http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html
Oracle

11. Mule ESB is a lightweight Java-based
enterprise service bus (ESB) and integration
platform that allows developers to connect
applications together quickly and easily,
enabling them to exchange data.
https://www.mulesoft.com/resources/esb/what-mule-esb
Mulesoft

12. How should we define it?

13. ¯_(ツ)_/¯

14. App
App
App
App
Finance
HR
BizOps
Inventory

15. ESB
Routing
Mediation
Messaging
Event
Processing
Service
Orchestration

16. ESB
Push
Centralized

17. Architecture

18. API
CLI GIT Dashboard
Data
Runtime
BizOps

19. $ heroku addons:create heroku-postgresql:standard-0 -a sushi
Creating postgresql-reticulated-74861... done, ($50.00/month)
Adding postgresql-reticulated-74861 to neovintage... done
Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249
The database should be available in 3-5 minutes.
! The database will be empty. If upgrading, you can transfer
! data from another database with pg:copy.
Use `heroku pg:wait` to track status
Use `heroku addons:docs heroku-postgresql` to view documentation.

20. API
CLI GIT Dashboard
Data
Runtime
BizOps

21. API
CLI GIT Web
Data
Runtime
BizOps
THE API IS OUR ESB!

22. Problems Emerge
• Bottleneck in getting features released that touch API
• Difficult to get Analytics across the entire Org
• Hiring has become difficult
• API is down. So is the entire platform.

24. PUSH vs PULL
http://kafka.apache.org/documentation.html#design_pull

25. PUSH

26. PULL Producers Consumers

27. ESB Redefinition
• Pull over Push
• Consumers dictate their own consumption rate
• Independent development, scaling and scheduling
• Stop the world processing doesn’t exist

28. CLI GIT Dashboard
API Data Runtime BizOps

29. Other Use Cases
Around Heroku

30. Operational Experience

31. aggregation
router
aggregation
router
metrics-ingest
route
r
metrics-ingest
route
r
ZK
postgres
metrics-ingest
router
aggregation
router
aggregation
router
aggregation
router
sink
router
aggregation
router
aggregation
router
metrics-api
router

32. $ heroku kafka:list -a metaas-addons
=== Kafka Topics on KAFKA_URL
Name Messages Traffic
─────────────────────────────────── ──────────────
─────────────
ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec
DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec
DYNO_LOAD.v1 1,468.01/sec 153 kb/sec
ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec
SYSLOG.v1 691.9/sec 76.8 kb/sec
ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec
ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec
DYNO_ERROR.v1 354.06/sec 38.5 kb/sec
DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec
DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec
DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec
ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec
__consumer_offsets 4.1/sec 405 bytes/sec
ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec
ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec
ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec

33. What do we need to do
next?

34. Schema Registry

36. Security

38. as a Service

39. Creating kafka-lively-27896... done, (free)
Adding kafka-lively-27896 to sushi... done
Kafka cluster has been created and will be available shortly
Use `heroku addons:docs heroku-kafka` to view documentation.
=== KAFKA_URL
Name: kafka-silhouetted-79119
Created: 2016-03-24 21:21 UTC
Plan: Beta 3
Status: available
Version: 0.9.0.0
Topics: 0 topics (see heroku kafka:list)
Connections: 0 consumers (0 applications)
Messages: 0.0 messages/sec
Traffic: 0 Bytes/sec in / 0 Bytes/s out
$ heroku kafka:info -a sushi
$ heroku addons:create heroku-kafka:beta-3 -a sushi

40. Public Beta
https://heroku.com/kafka

41. Come See Us At Our Booth
We’re Hiring!