(Wanny Morellato, SAP Concur) Kafka Summit SF 2018 Monolithic architectures should become a thing of the past sooner or later—preferably, sooner, of course … However, as it usually is with shiny pictures of a perfect future outcome vs. sobering facts of the reality, moving from a monolith to microservices is sometimes easier said than done. This talk will cover many lessons we learned during this process and how Kafka, change tracking and KSQL were successfully leveraged to break down a SQL server monolith while at the same time allowing SAP Concur to scale its backends to billions of daily transactions, enabling several new features and functionalities. Learn how we: -Leveraged Kafka Connect change tracking to propagate data changes out of SQL Server -Used Kafka to provide a highly performant and horizontally scalable central nervous system for SAP Concur events -Implemented KStreams/KSQL to perform real-time joins, aggregations, windowing and webhook integrations

1. Wanny Morellato, SAP Concur
Director of Engineering
October 16, 2018
Breaking Down A SQL Monolith
with Change Tracking, Kafka and KStreams / KSQL

2. NOTE: Delete the yellow stickers when finished.
Samples of a title slide with a pictogram are also
available in this template.
See SAP Image Library for other available images
you can sort using the drop down menu and
selecting illustration options.
Wanny Morellato
Director of Engineering @ SAP Concur
@mrwanny
Wanny.Morellato@sap.com
#I am Hiring

3. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "3
Agenda

4. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "4
The Why
MTTRHealth Scale

5. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "5
The What
ReadsReads + Writes

6. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "6
The What
Reads + WritesReads

7. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "7
The How
Change
Tracking
Kafka
Connect
Kafka
Topic
Kafka
Streams
Micro
Service

8. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "8
MSSQL Change Tracking
SELECT CT.user_id,
U.uuid user_uuid,
U.is_active is_active,
CT.SYS_CHANGE_VERSION,
CASE CT.SYS_CHANGE_OPERATION
WHEN 'I' THEN 'UserCreated'
WHEN 'U' THEN 'UserUpdated'
WHEN 'D' THEN 'UserDeleted'
END
CT.SYS_CHANGE_COLUMNS
FROM CHANGETABLE( CHANGES [dbo]. [users], @last_version )
AS CT
LEFT OUTER JOIN [dbo].[users] AS U WITH (NOLOCK)
ON CT.user_id = U.user_id
WHERE CT.SYS_CHANGE_VERSION < @max_version
ORDER BY CT.SYS_CHANGE_VERSION

9. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "9
Kafka Connect
public class MssqlSourceTask extends SourceTask {
@Override
public List<SourceRecord> poll() {
ResultSet result = stmt.executeQuery();
return convert(result);
}

10. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "10
Kafka Topic
• One Topic per Table
• Partition Size aligns to the Table change rate
• Consistent Hashing partitioning based on a Table unique key
• Replication Factor : 3
• Min InSync Replica : 2

11. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "11
Kafka KSQL
CREATE TABLE users
(user_id VARCHAR,
user_uuid VARCHAR,
is_active BOOLEAN,
. . . )
WITH (KAFKA_TOPIC='users',
VALUE_FORMAT='JSON',
KEY = ‘user_uuid')
WHERE is_active = 'true' ;

12. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "12
Kafka Stream
Topology topology = new Topology();
topologyBuilder
.addSource("SOURCE", stringDeserializer, deserializer, getTopic())
.addProcessor("FILTER",getFilter(),"SOURCE")
.addProcessor("PROCESSOR", getWebhookProcessor(), "FILTER")
.addSink(“SINK",getDeadLetterTopic(),serializer,"PROCESSOR");
StreamsConfig config = new StreamsConfig(env.getStreamProperties());
KafkaStreams streaming = new KafkaStreams(topology, config);
streaming.start();

13. © 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ "13
Summary
Change
Tracking
Kafka
Connect
Kafka
Topic
Kafka
Streams
Micro
Service

14. Thank You.
Contact information:
Wanny Morellato
Director of Engineering
SAP Concur
@mrwanny
Wanny.Morellato@sap.com

15. © 2018 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP
SE or an SAP affiliate company.
The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its
distributors contain proprietary software components of other software vendors. National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or
warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials.
The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty
statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional
warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or
any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation,
and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platforms, directions, and
functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason
without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or
functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ
materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they
should not be relied upon in making purchasing decisions.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered
trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names
mentioned are the trademarks of their respective companies.
See www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.
Learn more at concur.com
Follow all of SAP Concur