Building a minimum viable product in 3 months is easy. Scaling it towards a reactive system that can handle thousands of requests per second and deploying new versions without causing a denial of service is another challenge. Find out how at Crobox we scaled from a single machine (and point of failures) towards the high-available server cluster we are now running. On this journey you can also learn how we solved challenges with monitoring, logging and deployments.

1. Life of a startup
Sjoerd Mulder
AMSTERDAM 16 - 17 MAY 2017

3. Crobox uses
consumer psychology
and
machine learning to
find the
“why” behind the “buy”

4. How does it work?
2. Build persuasion profiles of each individual
visitor using eventsourcing and machine learning
1. Collect click behaviour on the website of
our customers through Javascript
3. Enrich website with predictions on what
to show to the visitor

5. Server
Let’s build it!
(summer 2015) Spring Boot
Nginx LB
MongoDB

6. Bigger Server
Let’s scale it up!
(Q4 2015)
Spring Boot
Nginx load balancer
DB Server 1
MongoDB
DB Server 2
MongoDB

7. Challenges
This “guy” These “people”

8. Challenges
● No automatic restarting when service OOM’s
● Manual deployments using CLI, e.g. error prone
● MongoDB just locks up on read-heavy operations
● Many logs on multiple servers
● Hard to handle peak traffic

9. Solution: Many logs on multiple servers
(Q4 2015)
ELK stack
+
Graylog

10. Solution: Hard to handle peak traffic
(Q1 2016)
1. Separate event
collection to service
outside of monolith
2. Added Kafka
in between of collector
and processing
New challenge
Service discovery
Spring Boot
Nginx Load Balancer
CollCollectorCollector
Kafka

11. Solution: MongoDB locking problem
(Q1 2016)
Use different datastore for
analytics and statistics queries
New challenge
Elasticsearch
Use different datastore for KV
collections larger than 100k rows
New challenge
Monitoring

12. Solution: Deployment and restarts
(Q1 2016)
Mesos and Marathon FTW!

13. Typical deployment
$ http PUT https://mesos-master/marathon/v2/apps/collector < marathon.json
{ "id": "collector",
"instances": 3,
"cpus": 2,
"mem": 2048,
"container": {
"type": "DOCKER",
"docker": {
"image": "crobox/collector:1.234",
...
}
},
"healthChecks": [{
"path": "/health",
"timeoutSeconds": 5
}],
...
}

14. Solution: Service discovery
(Q1 2016)
Consul
Service Discovery and Configuration Made Easy.
Consul-template
Creates configuration files based on services in Consul
Mesos-consul bridge
Automatically registers marathon jobs as services in Consul

15. Solution: Monitoring
(Q1 2016)
Prometheus / alertmanager
Open-source metrics and alerting
monitoring solution
Kamon
Open-source metric recording and trace
support JVM library
Grafana
The open platform for beautiful
analytics and monitoring

16. Future ready
(Q2 2016)
Nginxloadbalancer
Master server quorumMaster server quorum3 master servers
Monitoring / CIMonitoring & Logging
N worker servers
Node reqs
Service Service Service Service

17. Monitoring & logging
Grafana
Graylog

18. Now deploy any service with ease!
(Q2 2016 - now)
Platform services
● Collector API
● Public API
● Admin API
● Visitor service
● Product service
● Account service
● Reporting service
● Database replicators
● Prediction service
Auxiliary services
● Apache Zeppelin
● Kibana
● Superset
● Apache Spark
● Secor
● Device detector
● GeoIP lookup
● Webpack compiler
● Referrer parser

19. Challenge: Elasticsearch
● Really likes resources
● Single queries can down a
whole cluster
● Unreliable response times
● Really likes resources

20. Solution: Elasticsearch
(Q4 2016)
ClickHouse is an open source
column-oriented database management
system capable of real time generation of
analytical data reports using SQL queries.
https://clickhouse.yandex
SELECT
product_id,
uniq(profile_id) as value
FROM product_events
WHERE date >= '2017-05-01' AND
event_type = 'bought'
GROUP BY product_id
ORDER BY product_id DESC

21. Conclusion
● Container orchestration
● Aggregated logging
● Monitoring, monitoring, monitoring
● Use the database that fits your use-case

22. Thank you!
Questions?
Surprise: we are also hiring :-)
sjoerd@crobox.com