We adopted a serverless architecture to build a real-time analytics solution for tracking website usage. This involved using AWS Lambda functions triggered by events in Amazon Kinesis streams to index data from API requests in Amazon Elasticsearch. The serverless approach allowed us to focus on solving business problems rather than managing infrastructure, and provided built-in monitoring, auto-scaling, and pay-per-use billing. While some services like API Gateway could become expensive at high volumes, we optimized costs by batching requests and retrieving data in batches from Kinesis. The resulting solution met our goals of speed, cost-effectiveness, and reduced maintenance.

1. We’re AGILE
we embrace CHANGE
we automate EVERYTHING
we always value TOP TALENT

2. Serverless
Real-time Tracking & Analysis
@heisthedon

3. about://me
dev manager @carsales
head of retail platform
@heisthedon

4. Agenda
• Motivation behind this
• Serverless.. What and Why?
• The Architecture
• What We Learnt

5. Our Motivation

6. Our Inspiration

7. Feedback Loop

8. we want faster feedback

9. be autonomous and resourceful

10. take ownership and be accountable

11. so we build our own tracking solution

12. Our Success Metrics are
Speed of Implementation Must be Cost Effectives No Reinvent the Wheels Less Maintenance

13. What Is Serverless Architectures?
Serverless != No Server

14. Serverless is…
• Orchestration. Event-driven - the "if this, then that" trigger
• Microservices architecture
• In the Cloud

15. Why Serverless Architectures?
Focus on what matters

16. Serverless helps us to focus on what matters…
• Business problem to solve
• Infrastructure – less maintenance, auto-scale, capacity and
underutilization servers not a big concern
• Less code to write
• Logging and monitoring built-in

17. Our Lego Building Blocks
Amazon
CloudWatch
Amazon API
Gateway
Amazon
S3
Amazon
Elasticsearch
Service
Amazon
Kinesis
Streams
AWS
Lambda
Amazon
Redshift

18. Amazon
API Gateway
API Management Tools
• REST Endpoint
• Authentication
• Caching
• Monitoring
• Request throttling
• Cloudfront integration
• Cloudwatch Monitoring
• Stage features – dev vs prod
Maintenance
• No schedule downtime
Pricing Model
• Pay what you use

20. AWS
Lambda
Function as a Service (FAAS)
• Execute in response to events
• Deploy in millisecond
• Live in second (max 300s)
• Auto-scale by throughput (run 100 concurrent execution)
• Built-in retry mechanism
• Support versioning / aliasing
• Cloudwatch Log & Monitoring
Maintenance
• No schedule downtime
Pricing Model
• Pay what you use on execution and compute
(per execution on 100ms block)

22. Amazon
Kinesis Stream
Streaming Data Platform
• Scale in / out by Shard
• Data retention 1 – 7 days
• One shard has capacity of 1MB/sec in & 2MB/sec out,
and throughput of 1000 records / sec
• Cloudwatch Monitoring
Maintenance
• No schedule downtime
Pricing Model
• Pay what you use on data in
(PUT Payload Unit per 25KB block = $0.014 per million unit)
• Shard Hour (1 Shard = $11 per month)
• Data Retention

23. Amazon
Elasticsearch
Service
The Power of Elasticsearch and Cloud Combined
• The Full ELK Stacks
• High availability
• Failure detection, patch, snapshot are fully managed
• Cloudwatch Monitoring
Maintenance
• No schedule downtime
Pricing Model
• Pay what you use on compute hours & EBS storage

24. AWS
Lambda
Amazon
Elasticsearch
Service
Realtime Analytic
The Architecture
Amazon
CloudWatch
Amazon API
Gateway
Amazon
Kinesis
Streams

25. We are going to send…
100m requests / month

26. Amazon API
Gateway
Ajax /
img 1x1 Amazon
Kinesis
Streams
Integration with AWS Service Proxy
Map API Payload into Kinesis Stream Payload
Cost
$350
Cost
$18

27. Amazon
Kinesis
Streams
AWS
Lambda
Amazon
Elasticsearch
Service
POST to ES IndexPoll in batches
Cost
$94
Cost
$50

29. Kibana Dashboard

30. Few Cost Optimisation
• API Gateway and Lambda can be very
expensive..
• 100m requests = $350
• 200m requests = $700
• Batch requests to API Gateway
• Retrieve records in batch from Kinesis
• Lambda execution time

31. Deployment
Cloud Formation
for rest
Lambda utilises
Serverless Framework

32. Monitoring and Logging
For cost management
and visibility
CloudWatch
integration with all
services

33. CloudWatch Dashboard

34. CloudWatch Logs

35. Amazon
S3
AWS
Lambda
Amazon
Redshift
AWS
Lambda
Amazon
Elasticsearch
Service
Realtime Analytic
Extensibility
Amazon
CloudWatch
Amazon API
Gateway
Amazon
Kinesis
Streams

36. Our Success Metrics Revisited
Speed of Implementation Must be Cost Effectives No Reinvent the Wheels Less Maintenance

37. • Duplicate record on Kinesis Stream
• Cold-start on Lambda

38. Question?

39. Stay in touch
- careers.carsalesnetwork.com.au
- linkedin.com/carsales.com.au
- facebook.com/carsales.com.au
- Twitter: @carsales