Aleksandr Tavgen from Playtech, the world’s largest online gambling software supplier, will share how they are using InfluxDB 2.0, Flux, and the OpenTracingAPI to gain full observability of their platform. In addition, he will share how InfluxDB has served as the glue to cope with multiple sets of time series data, especially in the case of understanding online user activity — a use case that is normally difficult without the math functions now available with Flux.

1. Using InfluxDB for Full Observability of a
SaaS Platform
Aleksandr Tavgen Playtech

2. vAbout me
More than 19 years of
professional experience
FinTech and Data Science
background
From Developer to SRE Engineer
Solved and automated some
problems in Operations on scale

3. Overall problem
• Zoo of monitoring solutions in large enterprises often distributed over
the world
• M&A transactions or distributed teams make central managing
impossible or ineffective
• For small enterprises or startups the key question is about finding the
best solution
• A lot of companies have failed this way
• A lot of anti-patterns have developed

4. Managing a
Zoo
• A lot of independent teams
• Everyone has some sort of
solution
• It is hard to get overall picture
of operations
• It is hard to orchestrate and
make changes

5. QUITE OFTEN A ZOO LOOKS LIKE THIS

6. Common Anti-
patterns
It is tempting to keep everything
recorded just in case
Amount of metrics in monitoring
grows exponentially
Nobody understands such huge
bunch of metrics
Engineering complexity grows as
well

7. Uber case – 9 billion of metrics / 1000 + instances for monitoring

8. IF YOU NEED 9
BILLION OF
METRICS, YOU
ARE PROBABLY
WRONG

9. Dashboards problem
• Proliferating amount of metrics leads to unusable
dashboards
• How can one observe 9 billion metrics?
• Quite often it looks like spaghetti
• It is ok to pursue anti-pattern for approx. 1,5 years
• GitLab Dashboards are a good example

14. Actually not
• Dashboards are very useful when
you know where and when to watch
• Our brain can recognize and process
visual patterns more effectively
• But only when you know what you
are looking for and when

15. Queries
vs.
Dashboards
Querying your data requires more cognitive
effort than a quick look at dashboards
Metrics are a low resolution of your
system’s dynamics
Metrics should not replace logs
It is not necessary to have millions of them

16. What are
Incidents
• Something that has impact
on operational/business
level
• Incidents are expensive
• Incidents come with
credibility costs

17. COST OF AN
HOUR OF
DOWNTIME
2017-2018
https://www.statista.com/statistics/753938/worldwide-enterprise-server-hourly-downtime-cost/

18. • Change
• Network Failure
• Bug
• Human Factor
• Unspecified
• Hardware Failure
Causes of outage

19. Outage in dynamics

20. Timeline of
Outage
Detection
Investigation
Escalation
Fixing

21. What is it all about?
• Any reduction of
outage/incident timeline
results in significant positive
financial impact
• It is about credibility as well
• And your DevOps teams
feel less pain and toil on
their way

22. Focus on KPI metrics

23. Metrics
• It is almost impossible to operate on
billions of metrics
• In case of normal system behavior there
will always be outliers in real production
data
• Therefore, not all outliers should be
flagged as anomalous incidents
• Etsy Kale project case

25. Paradigm Shift
• The main paradigm shift comes from the fields of infrastructure and
architecture
• Cloud architectures, microservices, Kubernetes, and immutable
infrastructure have changed the way companies build and operate
systems
• Virtualization, containerization and orchestration frameworks abstract
infra level
• Moving towards abstraction from the underlying hardware and
networking means that we must focus on ensuring that our
applications work as intended in the context of our business
processes.

26. KPI monitoring
• KPI metrics are related to the core business
operations
• It could be logins, active sessions, any domain
specific operations
• Heavily seasoned
• Static thresholds can’t help here

27. Our Solution
• Narrowing down the
amount of metrics required
to defined KPI metrics
• We combined push/pull
model
• Local push
• Central pull
• And we created a ML-based
system, which learns your
metrics’ behavior

28. Predictive
Alerting System
Anomalies
combined with
rules
Based on dynamic
rules

29. Overwhelming
results
• Red area – Customer Detection
• Blue area – Own Observation (toil)
• Orange line – Central Grafana Introduced
• Green line – ML based solution in prod
Customer Detection has dropped to
low percentage points

30. General view
• Finding anomalies on metrics
• Finding regularities on a higher
level
• Combining events from
organization internals
(changes/deployments)
• Stream processing architectures

31. Why do we need time-series storage?
• We have unpredicted delay on networking
• Operating worldwide is a problem
• CAP theorem
• You can receive signals from the past
• But you should look into the future too
• How long should this window be in the future?

32. Why not Kafka and all those classical
streaming?
• Frameworks like Storm, Flink - oriented on tuples not time-ordered
events
• We do not want to process everything
• A lot of events are needed on-demand
• It is ok to lose some signals in favor of performance
• And we still have signals from the past

33. Why Influx v 2.0
• Flux
• Better isolation
• Central storage for metrics, events, traces
• Same streaming paradigm
• There is no mismatch between metaquering and quering

34. Taking a higher picture
• Finding anomalies on a lower level
• Tracing
• Event logs
• Finding regularities between them
• Building a topology
• We can call it AIOps as well

35. Open Tracing
• Tracing is a higher resolution of your
system’s dynamics
• Distributed tracing can show you unknown-
unknowns
• It reduces Investigation part of Incident
Timeline
• There is a good OSS Jaeger implementation
• Influx v 2.0 – the supported backend
storage

36. Jaeger with
Influxv2.0 as a
Backend storage
• Real prod case
• Every minute approx. 8000
traces
• Performance issue with
limitation on I/O ops
connections
• Bursts of context switches
on the kernel level

37. Impact on the particular
execution flow
• Db query is quite constant
• Processing time in normal case - 1-3 ms
• After a process context switch - more than 40 ms

38. Flux
• Multi-source joining
• Same functional composition paradigm
• Easy to test hypothesis
• You can combine metrics, event logs, and traces
• Data transformation based on conditions

39. Real incident
We need some statistical
models to operate on raw
data

40. Let’s check logins part

41. • Let’s check relations between them
• Looks more like stationary time – series
• Easier to model
• Let’s check relations between them
• Looks more like stationary time – series
• Easier to model

42. Random Walk
• Processes have a lot of random
factors
• Random Walk modelling
• X(t) = X(t-1) + Er(t)
• Er(t) = X(t) - X(t-1)
• Stationary time-series is very
easy to model
• Do not need statistical models
• Just reservoir with variance

43. Er(t) = X(t) - X(t-1)
Er(t) = discrete derivative of (X)

44. On a larger scale
• Simple to model
• Cheap memory reservoirs models
• Very fast

45. Security case
• Failed logins ratio is related to overall
statistical activity
• People make type-o’s
• Simple thresholds not working

46. One Flux transformation pipeline

47. Real Alerts related to attacks on Login Service

48. Combing all
together
Adding Traces and
Events can reduce
Investigation part
Can pinpoint to Root
Cause

49. •It is all about semantics
•Datacenters, sites, services
•Graph topology based on time-series data

50. Timetrix
• As a lot people involved in it from
different companies
• We decided to Open Source core
engine
• Integrations which are specific to
domain companies could be easily
added
• We plan to launch Q3/Q4 2019
• Core engine is written in Java
• Great Kudos to bonitoo.io team for
great drivers

51. Q&A
http://medium.com/@ATavgen/
www.timetrix.io