The document discusses techniques for building resiliency into real-time bidding systems. It describes monitoring the system through logging, heartbeats, and metrics collection. It also covers detecting and recovering from errors through techniques like circuit breakers and bulkheads. Rollbacks and retries are suggested for data errors, while circuit breakers and failovers can help handle system integration errors.

1. No bid left behind
My day to day handling a resilient real time bidding platform in a JVM environment.
Marc de Palol
Trovit

2. Hey hi,
• Studied here (good to be back)
• Some research on supercomputing
• Moved to London, discovered Hadoop & intensive
data systems.
• Came back, still in the ‘Data Engineering’ stuff.

3. A classified search engine for property, jobs, cars, products and holiday rentals
• 180 Million ads,
• 170 Tb in the cluster
• 65 Million uniques / 170 Million visits
• 10 apps (iOS, Android)
• Cool office in Barcelona.
have a look at http://www.trovit.es

4. Real Time Bidding
It’s about selling ads.
• Per impression basis.
• Programmatic instantaneous auction

5. We are using ‘DoubleClick Ad Exchange’ (Google)
• Response under 100 ms.
• If 15% of our responses are invalid or timed out,
we stop getting bid requests progressively

6. Currently 10.000 QPS.

7. This system, literally, spends money. So, it must be rock solid.
Our system is coded carefully, with love and tests.

8. Still, sh*t happens.*t Happens

9. Resiliency
The ability to recover from unexpected errors.
The ability to sleep at night.

13. Detect Recover Warn

14. Detect Recover Warn
Monitoring
Resiliency
Patterns
Notifications

15. Monitoring, in a sensible way

16. • Logging with ‘mailAppender’
log4j.appender.mail=org.apache.log4j.net.SMTPAppender
log4j.appender.mail.SMTPHost=localhost
log4j.appender.mail.From=Error <error-bla@trovit.com>
log4j.appender.mail.To=tech@trovit.com, ceo@trovit.com
log4j.appender.mail.Subject=[ERROR] WE ARE GOING TO DIE
log4j.appender.mail.layout=org.apache.log4j.PatternLayout
log4j.appender.mail.threshold=ERROR

17. • Logging with ‘mailAppender’
Probably, no e-mail when you’ve got an OOM.
log4j.appender.mail=org.apache.log4j.net.SMTPAppender
log4j.appender.mail.SMTPHost=localhost
log4j.appender.mail.From=Error <error-bla@trovit.com>
log4j.appender.mail.To=tech@trovit.com, ceo@trovit.com
log4j.appender.mail.Subject=[ERROR] WE ARE GOING TO DIE
log4j.appender.mail.layout=org.apache.log4j.PatternLayout
log4j.appender.mail.threshold=ERROR

18. Let’s talk about OOM for a
minute.

19. Let’s talk about OOM for a
minute.
ps ax | grep java

20. Let’s talk about OOM for a
minute.
ps ax | grep java
JVMOpts=“-
XX:OnOutOfMemoryError=
/usr/local/bin/slack-msg.sh"
🚫
👍

21. Some cool ideas for improving memory usage
• byte[] serialization in objects ❗
• Varying Memory Conditions ❗

22. • Logging with ‘mailAppender’
• Bad when OOM.

23. • Logging with ‘mailAppender’
• Bad when OOM.
• Heartbeat
• Doing some real work

24. • Logging with ‘mailAppender’
• Bad when OOM.
• Heartbeat
• Doing some real work
• Supervision with actors
• If you’re using Akka
• control flow != data flow

25. Our Monitoring:
• Nagios.
• Logging (to Sentry)
• Heartbeats with real work.
• graphite comparison

26. Our Monitoring:
• Nagios.
• Logging (to Sentry)
• Heartbeats with real work.
• graphite comparison

27. Have graphs

28. Now we know that something
is going wrong.

29. Recovery

30. Bad data in the system
or / and
Errors in the system

31. Data errors.
Roll back (when possible)
• Keeping different versions in the DB.
• Keep the old version around.
• Know how to do a rollback.

32. Data errors.
Roll back (when possible)
• Keeping different versions in the DB.
• Keep the old version around.
• Know how to do a rollback.

33. Checks & Asserts with google guava.
checkArgument(i >= 0,
"Argument was %s but expected nonnegative", i);
checkArgument(i < j,
"Expected i < j, but %s > %s", i, j);
checkNotNull(myList,
"List should not be null")
checkState(object.isValid(),
"Object is not valid")

34. System errors
These happen mostly between system integrations.
• Your code and the DB.
• Your code and the 3rd party library.
• Your code and the queue.

35. DBs, a necessary supervillain
• Lost connection.
• Timeouts
• Can give you corrupted data.
• Can give you 0 data.
• Can give you too much data.

36. Circuit Breaker and his friend,
the Bulkhead Pattern.

38. Circuit Breaker

39. Our Beloved
CircuitBreakers

40. Bulkhead

41. Once the circuit breaker is open,
• Notify
• Try again! maybe.
• Try to avoid DOS your own system.
• Exponential retry.
• Failover
• Restart

42. Some other bits and pieces:
• Tight coupling leads to fast propagation of errors.
• Event driven stuff
• Complete parameter checking
• Avoid SPF’s. Pretty please.
• Stateless is better.
• Bounded queues!

43. Your turn.
mdepalol@trovit.com
@lant
[]
http://www.maxisciences.com/destruction/wallpaper