At AO.com, our team have been focussing on how we can develop new capabilities delivering business value whilst decoupling from legacy monolithic architectures. The problem is, our existing applications contain a lot of data that we care about.

1. © 2018 AO World All Rights Reserved
Streaming data pipelines
with MongoDB and Kafka
Jon Vines, Software Engineer and Team Lead

2. • Jon Vines
• Software Engineer and Team Lead @ AO.com
• Get in touch:
• jonvines
• @_jonvines
• Loves reading, recommends:

5. Our
Approach
Our
Challenge
Data Driven
Applications
The Future

6. Data's
hidden
value

7. What is
Single
Customer
View?

8. A single customer view is an aggregated,
consistent and holistic representation of the data
held by an organisation about its customers that
can be viewed in one place, such as a single
page.
Wikipedia [https://en.wikipedia.org/wiki/Single_customer_view]

9. Get data
whilst it’s
hot

12. Why ?

13. SKETCH ONLY

14. SKETCH ONLY

15. {
"name" : "mongodb_europe_sql_connector",
"config" : {
"connector.class" : "io.confluent.connect.cdc.mssql.MsSqlSourceConnector",
"tasks.max" : "1",
"initial.database" : "MongoDBEurope2018",
"username" : "cdc",
"password" : "testing",
"server.name" : "sqlserver",
"server.port" : "1433",
"change.tracking.tables" : "dbo.tbl_Customer"
}
}
Kafka Connect SQL CDC

16. ~/kafka$ bin/kafka-topics.sh --list --zookeeper zookeeper:2181/kafka
cdc.MongoDBEurope18.tbl_Customer
cdc.MongoDBEurope18.tbl_Order
~/kafka$
Viewing Kafka Topics

17. ~/kafka$ bin/kafka-console-consumer.sh --bootstrap-server k1:9092 --topic cdc.MongoDBEurope18.tbl_Customer
--from-beginning
(sys_Change_operation|Mr Jon Vinesjon.vines@ao.comsys_change_creation_version 11862sys_change_version 11862
(sys_Change_operation|Mr Jonathan Vinesjon.vines@ao.comsys_change_creation_version 11874sys_change_version 11874
~/kafka$
Viewing Kafka Topics

18. {
"type": "record",
"name": "tbl_Customer",
"fields": [{
"name": "customerId",
"type": "int"
},
{
"name": "customerTitle",
"type": "string"
},
{
"name": "customerFirstName",
"type": "string"
},
{
"name": "customerSurname",
"type": "string"
},
{}
]
}
AVRO Serialisation

19. public void ConsumeMessages(string brokerList, IEnumerable<string> topics)
{
using (var consumer = new Consumer<byte[], byte[]>(ConstructConfig(brokerList, true),
new ByteArrayDeserializer(), new ByteArrayDeserializer()))
{
consumer.OnMessage += (_, message) => MessageReceived(message);
consumer.Subscribe(topics);
while (_consumeMessages)
{
consumer.Poll(TimeSpan.FromMilliseconds(100));
}
}
}
Consuming Kafka Messages

20. private void MessageReceived(Message<byte[], byte[]> message)
{
try
{
_processorStrategy.ProcessMessage(message.Topic, message);
}
catch (Exception exception)
{
throw;
}
}
Consuming Kafka Messages

21. public void ProcessMessage(Message<byte[], byte[]> message)
{
var cdcCustomer = _messageDeserializer.DeserializeMessage(message);
var customer = _scvCustomer.Add(cdcCustomer);
_customerRepository.Save(customer);
}
Processing Kafka Messages

22. {
"_id": "5af45b47db610c0001e222c1",
"scvId": "xLOuIoVo20uDMh1g+Y7gsQ==",
"customerSummaries": [
{
"firstName": "Jon",
"surname": "Vines",
"title": "Mr",
"landlineNumber": "01234125112"
}
],
"orders": [
{
"orderNumber": "SCV201415081",
"orderDate": "2018-10-15 17:31:02.000Z,
"orderTotal": 329.00
}
],
"lastActivity": "2018-06-15T10:00:47.470Z",
"addedDate": "2018-04-10T16:05:13.000Z",
"emailAddress": "jon.vines@ao.com",
"versionNumber": 12
}
Building up documents

23. ...
"customerSummaries": [
{
"firstName": "Jon",
"surname": "Vines",
"title": "Mr",
"landlineNumber": "01234125112"
},
{
"firstName": "Jonathan",
"surname": "Vines",
"title": "Mr",
"landlineNumber": "01234125112"
}
],
"orders": [
{
"orderNumber": "SCV201415081",
"orderDate": "2018-10-15 17:31:02.000Z",
"orderTotal": 329.00
},
{
"orderNumber": "SCV201417812",
"orderDate": "2018-10-15 17:31:02.000Z",
"orderTotal": 329.00
}
]
...
Building up documents

24. {
"name" : "mongodb_europe_oplog_connector",
"config" : {
"connector.class" : "io.debezium.connector.mongodb.MongoDbConnector",
"mongodb.hosts" : "rs0/192.168.99.27017",
"mongodb.name" : "SCV",
"collection.whitelist" : "SCV_Customer[.]*"
}
}
Kafka Connect MongoDB OpLog

25. Lessons learned

26. Generating batches
private void GenerateBatchJobs(int? firstEntityId, int lastEntityId, int maxBatchesGenerated)
{
var isFirstEntityIdProcessed = false;
var numberOfBatchesGenerated = 0;
_currentJobId = Guid.NewGuid();
while (!isFirstEntityIdProcessed && numberOfBatchesGenerated != maxBatchesGenerated)
{
var firstEntityIdOfBatch = lastEntityId;
var aoEntityBatch = _repository.Find(_limit, firstEntityIdOfBatch, _fieldName).ToList();
ProcessEntityBatch(aoEntityBatch);
lastEntityId = aoEntityBatch.Last().GetEntityId();
if (aoEntityBatch.Any(x => x.GetEntityId() == firstEntityId))
{
isFirstEntityIdProcessed = true;
}
lastEntityId--;
numberOfBatchesGenerated++;
}
}

27. MongoDB
Lessons

29. Use .explain()
public IEnumerable<T> Find(Expression<Func<T, bool>> expression, Expression<Func<T, int>> sortExpression,
int limit, string hint)
{
return _database
.GetCollection<T>(_collectionName)
.AsQueryable(new AggregateOptions() { Hint = new BsonDocument(hint, 1) })
.Where(expression)
.OrderBy(sortExpression)
.Take(limit);
}
...
db.order
.explain('executionStats')
.aggregate([
{ $match: { numberOfTimesProcessed: { $lt: 3 }, Id: { $gte: 19020680 } } },
{ $sort: { salesOrderId: 1 } },
{ $limit: 50 }
],
{hint: { Id: 1 }});

30. public IEnumerable<T> Find(string fieldName, int entityId, int maxNumberOfTimesProcessed, int limitToReturn)
{
var filter = new BsonDocument
{
{ Numberoftimesprocessed, new BsonDocument("$lt", maxNumberOfTimesProcessed) },
{ fieldName, new BsonDocument("$gte", entityId) }
};
var sort = new BsonDocument(fieldName, 1);
IAsyncCursor<T> query = _database.GetCollection<T>(_collectionName).Find(filter).Sort(sort)
.Limit(limitToReturn).ToCursor();
return query.ToList();
}
Use native query language

31. Use _id when possible

32. Watch for unbounded document growth

34. Lambda + VPC Peering + MongoDB Atlas
= Cold Starts

35. Cache your database object in your
lambdas

36. public LambdaConstructor()
{
var secretManager = new SecretManager(new ConfigurationManager());
var mongoDb =
JsonConvert.DeserializeObject<MongoConnectionString>(
secretManager.Get(Environment.GetEnvironmentVariable("SECRET_NAME")));
var mongoClient = new MongoClient(mongoDb.ConnectionString);
var mongoDatabase =
mongoClient.GetDatabase(Environment.GetEnvironmentVariable("MONGODB_GDPR_SUPPRESSION_COLLECTION"));
var pack = new ConventionPack {new CamelCaseElementNameConvention(), new IgnoreIfNullConvention(true)};
ConventionRegistry.Register("camel case",
pack,
t => true);
}
MongoDB once in Lambda
Serverless development with Node.js, AWS Lambda and MongoDB Atlas
https://bit.ly/2HrOGYX

37. Data access in Serverless architectures

43. People
And
culture

44. Are you ?