Versioning dashboards and alerts using “Infrastructure as code” makes the infrastructure more robust learn about them.

1. Dashboards as Code
Sanooj Mananghat RootConf
Data Engineer @ Intuit Hyderabad

4. Demo Day

5. Scenario I: Teamwork in Dashboard

6. Problems in Manually created Dashboards
- Human errors
- No Meaningful History
- No Rollbacks
- Incosistent Dashbaords

7. Scenario II : Concurrent Modification
- Difficult to Collaborate
- Introduces Conflicts.

8. Scenarios III: External Consumers

9. Problems in Manually created Dashboards
- Lacs Hygiene
- No separate test environment
- No/Minimal Peer Review System

10. DRY
- Effects productivity
- Zero value add
- Difficult to scale

11. Problem Statement
• Difficult to collaborate
• Difficult to Scale
• No/Minimal Peer Review
• Effects Productivity
• Introduces conflicts
• Inconsistent
• No Meaningful History
• Untracked Temporary changes
• No Rollback
• Lacks Hygiene

12. Process
• Process with Source control

13. Easiest Approach
• Dashboards store data in JSON files.

14. Bloaty JSON

15. Dashboards as JSON
- Bloaty huge JSON files
- PR review became impossible
- Duplicate blocks

16. How can we squeeze the elephant into the car trunk?

17. JSONNET Templates
• Reduce Boilerplate and redundant code blocks

18. JSONET Loops
• {
"person1": {
"name": "Alice",
"welcome": "Hello Alice!"
},
"person2": {
"name": "Bob",
"welcome": "Hello Bob!"
},
"person3": {
"name": "Jack",
"welcome": "Hello Jack!"
},
"person4": {
"name": "Jill",
"welcome": "Hello Jill!"
},
"person5": {
"name": "Pokemon",
"welcome": "Hello Pokemon!"
},
"person6": {
"name": "Dora",
"welcome": "Hello Dora!"
},
"person7": {
"name": "Jhony",
"welcome": "Hello Jhony!"
},
"person8": {
"name": "Humpty",
"welcome": "Hello Humpty!"
},
"person99": {
"name": "Dumpty",
"welcome": "Hello Dumpty!"
},
"person10": {
"name": "Tom",
"welcome": "Hello Tom!"
}
}
{
person1: {
name: "Alice",
welcome: "Hello " + self.name + "!",
},
person2: self.person1 { name: “Bob”},
person3: self.person1 { name: “Jack" },
person4: self.person1 { name: “Jill" },
person5: self.person1 { name: “Pokemon" },
person6: self.person1 { name: “Dora" },
person7: self.person1 { name: “Jhony" },
person8: self.person1 { name: “Humty" },
person9: self.person1 { name: “Dumpty" },
person10: self.person1 { name: “Tom" },
}

19. Architecture

20. Dashboards as JSONNET
- Simplest solution with less overhead.
- Learning a new language.
- Provisioning is still a challenge.
- Jsonnet for grafana: grafonnet-lib

21. Manual Provisioning
• Provisioning is still a challenge.

22. What?

23. Terraform
• Terraform providers/plugins
• terraform-provider-grafana
• terraform-provider-kibana
• terraform-provider-wavefront

24. Show me the code!
resource "wavefront_dashboard" "test_dashboard" {
name = "Terraform 12 Dashboard"
url = "tf12-test"
dynamic "section" {
for_each = var.sections
content {
name = section.key
dynamic "row" {
iterator = rows
for_each = section.value
content {
chart {
name = rows.value.name
description = rows.value.description
units = "unit"
source {
name = rows.value.source
query = rows.value.query
}
chart_setting {
type = "line"
}
summarization = "MEAN” } } } } } }
• terraform apply
For loop

25. Architecture

26. Dashboards as Code
- No dashboard specialist
- Introduced strict peer review concept
- Reviewing became meaningful
- Consistent and collaborative.
- Easily scalable.
- Rollbacks become easier.
- No repeated works, no clicks.

27. Dashboards as Code
- Oncall more easier with consistent dashboard.
- Less cognitive overhead.
- Visual Diff with test and prod environment
- Templatizing helped similar dashboard for similar services.
- Monitoring as code

28. Wrapping Up
• Initial Onboarding time.
• Moved majority of our dashboards to Terraform
• Moving further to Monitoring as Code.
• Grafana-Lib (JSONNET based solution)
• https://github.com/grafana/grafonnet-lib
• Grafana python module - from Weave works
• https://github.com/weaveworks/grafanalib
• Grafana dashboard generator - from Uber
• https://github.com/uber/grafana-dash-gen

29. And they lived happily ever after!!

30. Sanooj Mananghat
Data Engineer @ Intuit