http://www.meetup.com/vilnius-backend/events/221026855/

1. Backend & Cloud devs
kickoff meetup
2015-03-19
TRAFI
JURGIS@TRAFI.COM

2. Plan
1. About this meetup – why and what
2. TRAFI case
◦ Architecture overview
◦ Azure vs. AWS
◦ Queues and data streams (AWS Kinesis)
◦ Google BigQuery

3. Why meetup?
Community is good!
◦ Share knowledge
◦ Similar problems – someone else maybe solved it
◦ Different approaches – widens your view
◦ It’s fun

4. What is “backend developer”?
BEFORE NOW
ASP.NET
SQL
Mobile
API API
Load balancer
Queues
Cache
Logs
Metrics SQL NoSQL
Web JS
Worker
IAPI

5. Lots to learn
Can build a lot very easily… if you know how
Challenges
◦ Mobile, cloud, service-oriented architecture
◦ New tools every day (libraries, frameworks, cloud services)
◦ Agile, cross-functional teams
◦ No longer “C# devs”, “Java devs” and “SQL devs” – need to understand bigger picture

6. Meetup focus
Architecture & patterns
◦ How to build cloud (distributed) solutions
Tools & technologies
◦ Know what’s out there (right tool for the right job)

7. Possible topics
Architecture & patterns
◦ Service-oriented, queues, micro services
◦ Async, reactive programming
◦ Functional programming
◦ Domain driven design, hexagonal architecture
Languages
◦ C#, Scala, Go, Java8, C++11, Ruby, Node.js, …
NoSQL
◦ MongoDB, Redis, Firebase
Big data
◦ Hadoop, Spark, BigQuery
◦ Machine learning
Cloud services
◦ Amazon Web Services (38 services!)
◦ Microsoft Azure
◦ Google Cloud
Anything you’ve found out, you wished someone told you!

8. TRAFI
Use case
◦ Android, iOS, Web apps
◦ REST API (~100 reqs/sec)
◦ Memory & Compute intensive
◦ External data integrations
◦ Background jobs
Code
◦ C#, ASP.NET MVC
◦ Some Java
◦ Some Python

9. TRAFI architecture
Mobile
.NET API .NET API
Elastic Beanstalk
(load balancer)
Data streaming
(Kinesis)
ElastiCache
Logentries
(logs)
Librato
(metrics)
MS SQL BigQuery
Azure
Website
Background jobs
(Win VM)
Internal Java API
(Linux VM)
Infrastructure
◦ AWS Elastic Beanstalk
◦ AWS EC2 instances (c3.4xlarge)
◦ Azure websites
◦ Azure worker roles
Storage
◦ MS SQL server
◦ AWS Kinesis
◦ AWS ElastiCache
◦ Google BigQuery
Metrics & Logs
◦ Logentries
◦ Librato

10. Lesson #1: Azure vs AWS
Migrated .NET API
◦ Azure: Web roles + Azure SQL
◦ AWS: Elastic Beanstalk + RDS SQL
Advantages
◦ 2x faster CPU performance!
◦ Faster deployment (also “.NET-friendly”)
◦ More flexible load balancer, auto scaling
◦ Better metrics and logs
◦ SQL – faster, easier to manage, better metrics
Disadvantages
◦ No “staged rollout”, only swap

11. Lesson #2: use messaging
Use case: streaming data from external sources
Don’t reinvent the wheel – it’s not as easy as it looks!
External
integration
API 1
API 2
GPS
data DB
Background
External
integration
API 1
API 2
GPS
data
Background
Stream

12. Lesson #2: use messaging
Queues
◦ AWS SQS
◦ Azure storage queues
◦ RabbitMQ
Data streams (publish/subscribe)
◦ AWS Kinesis
◦ Azure Service bus
◦ Kafka
Queue
Worker 1
Worker 2
B
A
B
Stream
Listener 1
Listener 2
B
A
B
A
A
B
A

13. Lesson #3: storing logs
Use case: store insert-only data for later analysis
◦ Web server request logs
◦ App event logs
◦ GPS logs…

14. Storing logs attempt 1
Traditional SQL
◦ Good for small amount of data (<10 GB)
◦ Limited insert capacity
◦ Slow queries
◦ Expensive ($1.00 /GB/month)

15. Storing logs attempt 2
Azure table storage (NoSQL)
◦ Cheap ($0.07 /GB/month)
◦ Unlimited data (partitioning)
◦ Only simple queries, slow

16. Storing logs attempt 3
Google BigQuery (columnar store)
◦ Very Cheap ($0.02 /GB/month)*
◦ Unlimited data (partitioning)
◦ Fast advanced SQL queries!
◦ Structured data

17. Open questions
Websites
◦ Pretty nice in Azure – easy deployment, cheap
Background jobs
◦ No “worker roles” in AWS
◦ Running as “Windows Services” in custom EC2 VM
◦ Docker?

18. Thank you
Questions?
Your experience?
What’s next?
Beer time…