A talk on MadStreetDen's serverless logging pipeline with live demo

1. BUILDING AND SCALING A LOG ANALYTICS
PLATFORM
- A SERVERLESS APPROACH
Narendran

2. A BIT ABOUT ME…
Naren
Product Engineer
Scaling A.I to millions
@ MadStreetDen
Python, Golang, FOSS, Cycling,
Travel

3. TWITTER & GITHUB
@DudeWhoCode
WEBSITE & BLOG
www.dudewho.codes

4. OUR DISTRIBUTED MICROSERVICES

5. ARCHITECTING A LOGGING FRAMEWORK

6. BEHAVIOURS OF DIFFERENT
MICROSERVICES
• Mostly Idle
• Horizontally scales up
• Rapid scale down
• Surges

7. EXPECTATIONS FROM ANY LOG
FRAMEWORK
• High Performance
• Secure
• No data Loss/overwriting
• Dynamically Scalable
• Highly Available
• Independent

8. KNOWYOUR LOGS
• Real time Analysis
• Historical Analysis

9. THE CONVENTIONAL ARCHITECTURE

10. PROBLEMS FACED

11. THETHING WITH ELK
•Threadpool queue size in Elastic Search
• Adding Logstash filters
• Burning more money, even for idle
time

12. WHO WILL MONITORTHE SYSTEMTHAT
MONITORS OTHER SYSTEMS?

13. WHO WILL MONITORTHE SYSTEMTHAT
MONITORS OTHER SYSTEMS?

14. WHATSTHE COMMONTHINGTHAT ALL
THE COMPONENTS HAVE?

15. WHATSTHE COMMONTHINGTHAT ALL
THE COMPONENTS HAVE?
Servers, Servers, Servers

16. Servers, Servers, Servers

17. GOING SERVERLESS
(buzzword alert!)
Serverless computing is a cloud computing execution model in which the cloud provider
dynamically manages the allocation of machine resources.
Pricing is based on the actual amount of resources consumed by an application, rather
than on pre-purchased units of capacity.
- Wikipedia

18. GOING SERVERLESS
No server is easier to manage than “no server”
- Werner Wogels, CTO,Amazon

19. GOING SERVERLESS

20. SERVERLESS ARCHITECTURE

21. AMAZON WEB SERVICES
(AWS)

22. AWS SERVERLESS

23. AWS SERVERLESS

24. AWS SERVERLESS

25. FAAS ARCHITECTURE

26. •The client application from
which we will get our logs

27. • Delivers real-time streaming data to other
services such as Amazon S3, Elastic Search.
• Configurable producers and consumers

28. • Run code without provisioning servers
• Event driven, Highly scalable

29. • Store data at massive scale
• Storage tier for many serverless applications

30. • Serverless interactive query service
• Point to your data source, define the schema,
start querying using standard SQL

31. • Cloud-powered visualization tool
• Perform ad-hoc analysis and get quick business
insights

32. • Process streaming data in real time with standard
SQL in real time
• Scales automatically to match throughput

33. • Real time log monitoring
• Collect and track metrics, log files, trigger alarms

34. MANAGINGTHE SERVERLESS MODEL
1. Web Console
2. Code (AWS SDK)
3. CLI

35. DEMO

36. YOU CAN WATCHTHE FOLLOWING
DEMO AS SCREENCAST AT
http://bit.ly/serverless-logging

45. SUMMARY
• Usual logging architecture: took us weeks and certain level of expertise
to implement
• Serverless architecture: Up and running in minutes
• No maintenance, nearly zero developer interventions after deployment
• This Faas architecture can scale infinitely for millions of logs from
thousands of microservices

46. SUMMARY
•
•
•
•
•

48. SLIDES :
www.dudewho.codes/talks
THANKYOU,
NAREN
@DudeWhoCode