This document discusses the future of application development and key topics in streaming data and AI. It begins with an overview of streaming concepts like topics, streams, and tables. It then covers the Kappa architecture for stream processing using tools like Kafka Streams, ksqlDB, and Flink. The document also discusses challenges with generative AI models like handling private data, long-term context and memory, and integration into businesses. It concludes with recommendations to simplify architectures and use streaming as smart pipes to process raw and enriched data.

1. The Future of Application Development
API Days — Melbourne 2023
David Peterson
Principal Solutions Engineer, APAC

2. Stream
Processing
ϰ
Kappa Generative
AI

4. Grady Booch
“No matter what future we may envision, it relies
on software that has not yet been written”

6. Big Data
More == Better
Data in Motion
Timely == Better
Value
of
Data
->
Time ->
Value
of
Data
->
Time ->

7. ϰ
Kappa Architecture

9. Publish /
Subscribe
Store Process

10. CDC
Connect
MongoDB
Connect
Kafka

11. MongoDB
Connect
CDC Events Batch Txns
Connect
Schema
Kappa
Schema

12. MongoDB
Connect
CDC Events Batch Txns
Connect
Kappa

13. CDC Events Batch Txns
Connect
MongoDB VectorDB
Postgres
Connect
Replay ^^
Kappa
Source of Truth
}

14. Stream Processing

15. Old world
Event processor
New world
Stream processor

16. Event == Fact
☹

17. Events == Story
"
#
$
% ☹

18. 1. TOPIC

21. {“actor”:”bear”,
“x”:410, “y”:20}
{“actor”:”racoon”,
“x”:380, “y”:20}

22. {“actor”:”bear”,
“x”:380, “y”:22}
{“actor”:”racoon”,
“x”:350, “y”:22}

23. 2.STREAM

24. unbounded, continuously updating data set, ordered,replayable, fault-tolerant

26. 3.TABLE

27. Exposure Sheet

28. Changelog stream – immutable events

29. Stream-table Duality

30. CDC Events Batch Txns
Connect
MongoDB VectorDB
Postgres
Connect
Kappa
Materialised Views
{

31. CDC Events Batch Txns
MongoDB VectorDB
Postgres
Connect
Connect
Stream Processing
Consumer and producer ^^
}
Kappa — Stream Processing
Processors
• Kafka Streams
• ksqlDB
• Flink

32. Context — Customer 360
32

33. Building our Customer Base

34. Just in time
Fast Batch

38. CREATE TABLE ORGANISATION_CUSTOMER_CUSTOMERS_360
WITH (
KAFKA_TOPIC = 'ORGANISATION_CUSTOMER_CUSTOMERS_360'
)
AS SELECT
CU.CUSTOMER_ID AS CUSTOMER_ID,
FIRST_NAME AS FIRST_NAME,
LAST_NAME AS LAST_NAME,
STRUCT(
STREET_ADDRESS := CU.ADDRESS->STREET_ADDRESS,
SUBURB := CU.ADDRESS->SUBURB,
POSTCODE := CU.ADDRESS->POSTCODE
) ADDRESS,
STRUCT(
VENDOR := L.VENDOR,
POINTS := L.POINTS,
TRANSACTION_ID := L.TRANSACTION_ID
) LATEST_LOYALTY_TRANSACTION,
STRUCT(
TOTAL_POINTS := L.TOTAL_POINTS,
AVERAGE_POINTS := L.AVERAGE_POINTS
) LOYALTY_POINTS,
STRUCT(
SCORE := CR.CREDIT_SCORE
) CREDIT_RATING
FROM ORGANISATION_CUSTOMER_CUSTOMERS_STATE CU
LEFT OUTER JOIN ORGANISATION_CREDIT_RATING_STATE CR ON ((CU.CUSTOMER_ID = CR.CUSTOMER_ID))
LEFT OUTER JOIN ORGANISATION_LOYALTY_TRANSACTIONS_STATE L ON ((CU.CUSTOMER_ID = L.CUSTOMER_ID))
PARTITION BY CU.CUSTOMER_ID
EMIT CHANGES;

41. Generative AI

47. Kurzweil - 2014
“I am more optimistic and put the date at
2029…”

48. Ilya Sutskever - 2022
“In the future, it will be obvious that the sole
purpose of science was to build AGI”

49. /imagine prompt:Street style photo of a woman shot on Kodak
July 2023
July 2022

50. LLM

51. Ilya Sutskever
“Shocking degree of understanding about the
world”

53. CDC Events Batch Txns
MongoDB VectorDB
Postgres
Connect
Connect
Stream Processing
Consumer and producer ^^
}
Microservice
Generate the prompt.
Powered by C360
Topic

54. Launch Day

55. • New / Private data
• Context
• Memory
• Integrating into business
Hard Problems with LLMs / Gen AI

56. Hard Problem — Private data
{"release_date": "2023-05-02", "shoe_name":"ActivExhale Ultra Light", "press_release": "We're proud to present a groundbreaking shoe crafted with world
class support: the ActivExhale Ultra Light. The shoe's ultra flexible design means your feet are relieved of any stress and adapted for the greatest
comfort. With reinforced soles, shock absorption technology, and its hi-tech mesh pattern, you can walk and move freely in this shoe, no matter the
terrain!"}
{"release_date":"2023-10-28", "shoe_name":"TailorSport Edge", "press_release": "Introducing TailorSport Edge, our newest range of shoes created for
performance and agility. The special woven upper materials adapt to the shape and size of your foot, while the cushioning system provides outstanding
comfort and responsiveness of your feet whilst running or exercising. Its cutting edge technology allows your foot to experience a new level of
performance stylishly.”}
{"release_date":"2023-03-29", "shoe_name":"EasyCloud OG", "press_release": "A fan favorite, the EasyCloud OG, is now available. The OG redefines the
ultimate leisure-style walking shoe thanks to its synthetic cushioning and responsive surface. The unobtrusive materials make walking in this shoe
fast, sleek, and comfortable. The sole hugs your foot perfectly, keeping each stride light yet sure. It's time for effortless style and energetic
running."}
{"release_date":"2023-06-11", "shoe_name":"Smart Grip Trailblazer","press_release": "Designed for trail hikes, the Smart Grip Trailblazer is here! This
all-in-one shoe offers unbeatable traction on uneven surfaces, perfect padding, and fair water resistance - offering you the ideal blend of versatility
and control. Step into a smarter hiking experience: Try the Trailblazer on all kinds of terrains and enjoy comfort from the first step!”}
{"release_date": "2023-07-24", "shoe_name":"Prince Stables Flexy", "press_release": "Welcome the Prince Stables Flexy, designed with comfort,
stability, and full range of motion in mind. Enjoy cornering soles which won’t let you down while relying on shock pads to provide keep your feet safe
from exhaustion. Slip into convenience and agility - With heat forming technology, the practice makes perfection philosophy takes on a whole new
meaning."}
{"release_date":"2023-11-15", "shoe_name":"New Skymax runners", "press_release": "Experience extreme comfort with every step with the new Skymax
runners. Specifically engineered with superior fit, this shoe has lightweight cushioning with ultra high energy return - keeps you powerfully pushing
forward all day. Step up your run as you breeze through long-distances — with the smoothest cushion system and breathable material, get the best of the
best this season and beyond!”}
{"release_date":"2023-02-06", "shoe_name":"FuturiZen FreedomLite", "press_release": "Fusing clean aesthetic with personalized functionary technology,
FuturiZen FreedomLite is a groundbreaking product in footwear with its contouring uppers, featherlight midsole, and laser molded outsole. Walk away in
comfort and confidence - experience extreme lightweight connectivity wherever when you take on everyday life thanks to Versatile Cushion crossovers
within the sole."}
{"release_date":"2023-09-12", "shoe_name":"AiraSaint OnLiners", "press_release": "Beeline into AiraSaint OnLiners for lightweight, supportive running
at all distances. Situated with signature mesh fabrics and forefoot structures, OnLiners is a hybrid style meet first choice that features extreme
breathability but also encourages toe-splaying. Benefit from optimal fit all around your feet as you fasten laces - Infinity Comfort Seams direct your
each movement towards your desired outcome with great bouncy response."}

57. Hard Problem — Private data

59. Hard Problem — Context

61. Hard Problem — Long-term Memory

63. Hard Problem — Integration

67. Yuval Noah Harari — Sapiens
“Consistency is the playground of dull minds”

69. • API != REST, depends on use-case
• Kappa > Kafka — simplify your architecture
• Schema — data contracts
• Source of Truth
• Raw / Enriched data pattern
• Stream Processing — smart pipes
• Events tell a story

70. Confluent Cloud

71. Thank you
Q&A