Laszlo Fogas - One does not simply build her own Heroku Software Circus Fest 2020 "Build us a Heroku" is the mandate for platform teams and consultants alike.. and frankly, who doesn't like to hear that?! Another undifferentiated internal application platform is forming. Cloud Native tools are outstanding, we are where we have never been before. Still, the edges are rough, developer experience is clunky. In this talk, Laszlo gathers the bill of material for an internal application platform, shows the horrific, ends with a glimmer of hope.

1. ONE DOES NOT SIMPLY MAKEONE DOES NOT SIMPLY MAKE
HER OWN HEROKUHER OWN HEROKU

2. CONTEXTCONTEXT
freelance DevOps consultant in the past 3,5 years
building ~ 4 PaaS like internal service platforms
active in the Cloud Native Nordics community

3. BUILD US HEROKUBUILD US HEROKU
on cloud X
using containers, Kubernetes
CNCF and opensource projects

4. MAKE USMAKE US
deploy faster
better at monitoring
spread the DevOps culture

5. BEEN GUILTY OF THIS MYSELFBEEN GUILTY OF THIS MYSELF
2017 fall, talk: The Self Service developer
via GIPHY

6. THIS JANUARYTHIS JANUARY
new project a er 12 months
had to start everything from scratch
via GIPHY

7. Kelsey Hightower
@kelseyhightower
So you want to roll your own application platform. All you need
is:
Linux
Docker
Kubernetes
Istio
Prometheus
Fluentd
Grafana
Jaeger
Harbor
Open Policy Agent
Vault
Spinnaker and Jenkins
Oh, almost forgot, you're also going to need servers, people,
and glue. Bring lots of glue.
7,541 3:32 AM - Apr 3, 2020
2,134 people are talking about this

8. Fintan Ryan
@fintanr
This. Far too many of my client inquiries are with orgs trying to
build their own platform without understanding what they are
getting into. The vast bulk of your requirements are not unique,
no matter how special you feel. twitter.com/kelseyhightowe…
Kelsey Hightower @kelseyhightower
So you want to roll your own application platform. All you need is:
Linux
Docker
Kubernetes
Istio
Prometheus
Fluentd
Grafana
Jaeger
Harbor
Open Policy Agent
Vault
Spinnaker and Jenkins
Oh, almost forgot, you're also going to need servers, people, and
glue. Bring lots of glue.
76 1:26 PM - Apr 3, 2020
21 people are talking about this

10. laszlo
@laszlocph
"while a service mesh can emit trace spans from the proxies, the
application will need to propagate headers in order for these
spans to be pulled together into traces"
twitter.com/benplotnick/st…
Ben Plotnick @benplotnick
How many times do I have to say it, service mesh/istio will not
magically get you distributed tracing. Why do people think this?
2 10:45 AM - Jan 23, 2020
See laszlo's other Tweets

11. Metrics?
=> Prometheus

12. (NOT) PICKING FIGHTS(NOT) PICKING FIGHTS

13. (NOT) PICKING FIGHTS(NOT) PICKING FIGHTS
happy as devops, the amount of metrics is staggering ,
I know promql, there are alerts 👍

14. (NOT) PICKING FIGHTS(NOT) PICKING FIGHTS
happy as devops, the amount of metrics is staggering ,
I know promql, there are alerts 👍
but then trying to explain promQL, Grafana
dashboards to devs 😬

15. anything meaningful is not trivial in promQL
even I spend 2 hours on a join Prom query, or label
replace

16. anything meaningful is not trivial in promQL
even I spend 2 hours on a join Prom query, or label
replace
then they ask about how to set up alerts 🤯

17. PROMETHEUS ALERTINGPROMETHEUS ALERTING

18. PROMETHEUS ALERTINGPROMETHEUS ALERTING
Prom, Alert Manager, Grafana

19. PROMETHEUS ALERTINGPROMETHEUS ALERTING
Prom, Alert Manager, Grafana
we manage alerts in a yaml file

20. PROMETHEUS ALERTINGPROMETHEUS ALERTING
Prom, Alert Manager, Grafana
we manage alerts in a yaml file
CI deploys the file

21. PROMETHEUS ALERTINGPROMETHEUS ALERTING
Prom, Alert Manager, Grafana
we manage alerts in a yaml file
CI deploys the file
restarts prom

23. AT THIS POINTAT THIS POINT
we built dashboards from scratch
alerts from scratch
we put all the automation in CI

24. AT THIS POINTAT THIS POINT
we built dashboards from scratch
alerts from scratch
we put all the automation in CI
have at least two workflows to create alerts

25. AT THIS POINTAT THIS POINT
we built dashboards from scratch
alerts from scratch
we put all the automation in CI
have at least two workflows to create alerts
we build dashboards for devs

27. GITOPSGITOPS
via GIPHY

28. GITOPSGITOPS

29. GITOPSGITOPS
You deploy Flux or ArgoCD and you are done

30. GITOPSGITOPS
You deploy Flux or ArgoCD and you are done
Not quite

31. GITOPSGITOPS
Flux is unopinionated about folder structure
Conventions and automation needed to promote code
between environments
Tooling to trigger promotion, CLI/UI/CI

32. Kelsey Hightower
@kelseyhightower
Replying to @kelseyhightower
You know you're building a PaaS when you wrap your current
tools with a custom API that provides a workflow to your users.
71 1:14 AM - Apr 12, 2017
17 people are talking about this
Kelsey Hightower
@kelseyhightower
Replying to @kelseyhightower
You know you're building a PaaS when you start stitching
together 1,000,000,000 other tools in order to get one click
deployments.
126 1:19 AM - Apr 12, 2017
52 people are talking about this

33. Kelsey Hightower
@kelseyhightower
Replying to @kelseyhightower
You know you're building a PaaS when you start work on that
custom templating engine for deployments and configuration
files.
73 1:16 AM - Apr 12, 2017
21 people are talking about this

34. Kelsey Hightower
@kelseyhightower
I'm convinced the majority of people managing infrastructure just
want a PaaS. The only requirement: it has to be built by them.
935 1:08 AM - Apr 12, 2017
459 people are talking about this

35. Lachlan Evenson
@LachlanEvenson
Options are wrapped IaaS = not opinionated enough. Off the
shelf PaaS = too opinionated for us. Conclusion. let's roll our
own
16 2:18 AM - Apr 12, 2017
See Lachlan Evenson's other Tweets
Lachlan Evenson @LachlanEvenson · Apr 12, 2017
Replying to @LachlanEvenson @kelseyhightower
That opinion equals PaaS

36. CNCF/OSS projects ⇔ off-the-shelf PaaS
build ⇔ buy

37. CNCF/OSS •••••••••••••••••••••••••• off-the-shelf PaaS

38. GRAFANAGRAFANA
stepping into the void of cloud native UX
prometheus explore UI
tracing UI for Zipkin and Jaeger traces in v7.0

39. Marcus Olsson
@marcusolsson
I can't give you a date, but this is something we're looking at.
Just like data sources, we'd like to support multiple alerting
backends, Alertmanager being one of them.
4 9:26 AM - May 4, 2020
See Marcus Olsson's other Tweets
laszlo @laszlocph · May 4, 2020
When can I manage Prometheus alerts from @grafana ? That's my
next wish. Prometheus, Alertmanager, Grafana.. such a broken
workflow. I feel bad even demoing it to developers.
twitter.com/laszlocph/stat…
laszlo @laszlocph
Tracing UI for Zipkin and Jaeger traces - Grafana is stepping
into the large usability void that exists with various cloud native
tools. Makes a lot of sense to me. twitter.com/grafana/status…

40. PROMLABSPROMLABS
Julius Volz
@juliusvolz
Replying to @juliusvolz
This was partly inspired by my question last year about what
frustrates people: twitter.com/juliusvolz/sta…
While there were many pain points, it was obvious that PromQL
was a challenge both for beginners and advanced users.
PromLens aims to at least ease some of those pains :)
Julius Volz @juliusvolz
As either a beginner or advanced @PrometheusIO user, what
are/were your biggest frustrations, missing features, or things that
were hard to understand and work with?
18 8:05 PM - May 7, 2020
See Julius Volz's other Tweets

41. GKE Prom/Grafana presets?
https://github.com/monitoring-mixins/docs

42. https://github.com/lunarway/release-manager

43. OSS projects are great
the glue is daunting
CNCF/OSS •••••••••••••••••••••••••• off-the-shelf PaaS
Jack Foy
@jack_foy
Buy/build false dichotomy; spectrum of appropriate integration
points, levels of abstraction, coupling 2/2
2 5:48 AM - Apr 12, 2017
See Jack Foy's other Tweets
Jack Foy @jack_foy · Apr 12, 2017
Replying to @kelseyhightower
One level deeper, we want a composed PaaS we can cheaply tune
and refine as our org's needs evolve. Choice is in where we put the
glue 1/2

44. @laszlcph
CNCF/OSS •••••••••••••••••••••••••• off-the-shelf PaaS
full disclosure