apidays Australia 2023 - Platforms, Products, and People: The Power of APIs October 11 & 12, 2023 https://www.apidays.global/australia/ The Playful Bond Between REST And Data Streams Warren Vella, Solutions Engineer at Confluent ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

1. The Playful Bond Between
REST And Data Streams
Warren Vella | Solutions Engineer, Melbourne | Confluent

3. About me.

4. “Hey, I am a microservice,
come and talk to me.”
REST API

5. “I need to share my data with
an external application.”
REST API

6. “I need to modernise my
database with a facade.”
REST API

7. API ≠ HTTP-based
RESTful APIs

8. Operational Databases Analytic Systems
Azure
Synapse
How do we integrate these systems together?
SaaS Applications
Applications
Microservices Monoliths
LEGACY INTEGRATION
REST APIs only work here

9. REST APIs
8
1. …have proliferated as a way to integrate
microservices.
2.…are often the default when exchanging
data between organisations.
3.…don’t meet all our integration needs.

10. Lets go on a journey…
The Stream Bean Cafe ☕

11. Early 2000s

12. Intra-Process Communication

13. 2011ish

14. Microservices.
Trend away from monolithic applications.

15. Distributed application architecture
POS Payment Gateway
DataWarehouse
Realtime Analytics
CRM

16. How many lines to
integrate 4 systems?

17. n2-n

18. App 2
App 3
App 4
App 1
42-4
= 12 lines

19. How many lines to
integrate 2,000 systems?

20. n2-n

21. 20002-2000
= 3,998,000 lines

23. The microservices death star…

24. 1. Physical
2. Data Link
3. Network
4. Transport
5. Session
6. Presentation
7. Application
REST API
POS DataWarehouse

25. Synchronous
request-response
REST APIS are
addictive.
Like an addiction, they give us a false sense of
comfort of being always-available.
The more we build, the more we want but we often
overlook the exponential costs associated with
building failure mechanisms for never ending point-
to-point integrations.
We end up relapsing to REST APIs.

26. Payment Gateway
Request
Response
POS
DataWarehouse
Realtime Analytics
CRM
Asynchronous
Synchronous

27. REST APIs
26
1. …have proliferated as a way to integrate
microservices.
2.…are often the default when exchanging
data between organisations.
3.…don’t meet all our integration needs.
But we have ended up with a death star.
But are not suited for high-throughput.
So we have lots of different integration tools.

28. Let’s meet

29. Govern
Process
Connect
Store
Kafka decouples the producers and consumers
Data Topic

30. Side note…
Why is Kafka not a HTTP protocol like
REST?

31. Old World New World
Event Processor Stream Processor
REST API

32. REST API
• Great for Synchronous Responses
• Tight-coupling
• No delivery guarantees
• Receiving systems can be
overwhelmed
• Emphasis on Requests/Responses
• No Replay
• True Decoupling
• Stronger delivery guarantees. Kafka is
a buffer.
• Receiving systems process at a rate
they can handle.
• Emphasis on events.
• Rebuild state by replay.

33. Payment Gateway
Data Topic
Can REST and Kafka co-exist?
DataWarehouse
Realtime Analytics
CRM
Inside Boundary
POS
Microservices
Outside Boundary
Request / Response

34. 1. Physical
2. Data Link
3. Network
4. Transport
5. Session
6. Presentation
7. Application
POS Realtime Analytics

35. Confluent Stream Sharing

36. Connect with Confluent

37. Operational Databases SaaS Applications Analytic Systems
Azure
Synaps
e
| DATA STREAMING PLATFORM
Today, a fourth estate is emerging that provides
a new foundation for your data strategy

38. “Hey, I am a microservice,
come and talk to me.”
REST API

39. “I need to share my data with
an external application.”
REST API

40. “I need to modernise my
database with a facade.”

41. The future of integration.

42. Key Takeaways
41
1. ✅ Reduce the amount of lines between
systems with Kafka.
2.✅ REST APIs are great for Request-
Response but also consider Kafka as a
pattern for exchanging data between
organisations.
3.✅ A Data Streaming Platform integrates
both application and data estates.

43. Let’s connect.
linkedin.com/in/warrenvella
github.com/wvella