Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Grafana and MySQL - Benefits and Challenges

15,728 views

Published on

A presentation on our experience at Ingram Content Group with Grafana and MySQL. In an enterprise environment it is sometimes necessary to keep data in a traditional, general purpose SQL database such as MySQL or PostgreSQL. These slides explore the challenges and benefits of using Grafana with an SQL database in a large enterprise production setting.

Published in: Data & Analytics
  • Have u ever tried external professional writing services like ⇒ www.HelpWriting.net ⇐ ? I did and I am more than satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! I do no use writing service very often, only when I really have problems. But this one, I like best of all. The team of writers operates very quickly. It's called ⇒ www.WritePaper.info ⇐ Hope this helps!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Did u try to use external powers for studying? Like ⇒ www.HelpWriting.net ⇐ ? They helped me a lot once.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ♥♥♥ http://bit.ly/369VOVb ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/369VOVb ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Grafana and MySQL - Benefits and Challenges

  1. 1. Grafana and MySQL Benefits and Challenges Aemelia Presentation Template from http://www.slidescarnival.com
  2. 2. About me Philip Wernersbach Software Engineer Ingram Content Group https://github.com/philip-wernersbach https://www.linkedin.com/in/pwernersbach 2
  3. 3. • I work in Ingram Content Group’s Automated Print On Demand division • We have an automated process in which publishers (independent or corporate) request books via a website, and we automatically print, bind, and ship those books to them • This process involves lots of hardware devices and software components 3
  4. 4. We collect: • Device Status (Online/Offline) • Tracking data for every piece of material for every book • Throughput metrics • Etc. This produces gigabytes of data an hour. 4
  5. 5. The Problem 5
  6. 6. The Problem “How do we aggregate and track metrics from our hardware and software sources, and display those data points in a graph format to the end user?”  Grafana! 6
  7. 7. Which data store should we use with Grafana? ▸ Out of the box, Grafana supports Elasticsearch, Graphite, InfluxDB, KairosDB, OpenTSDB 7
  8. 8. Which data store should we use with Grafana? ▸ We compared the options and tried InfluxDB ▸ There were several sticking points with InfluxDB, both technical and organizational, that caused us to rule it out 8
  9. 9. Which data store should we use with Grafana? ▸ We already have a MySQL cluster deployed, System Administrators and Operations know how to manage it ▸ Decided to go with MySQL as a data store for Grafana 9
  10. 10. The Solution: Ingram Content’s Grafana-MySQL Integration 10
  11. 11. The Integration ▸Written in Nim ▸Emulates an InfluxDB server ▸Connects to an existing MySQL server ▸Protocol compatible with InfluxDB 0.9.3 ▸Acts as a proxy that converts the InfluxDB protocol to the MySQL 11
  12. 12. The Integration 12 Grafana Integration Proxy (InfluxDB compatible) MySQL
  13. 13. The Challenges! 13
  14. 14. InfluxDB ▸ Time Series database ▸ “time” is a special value, has special meaning ▸ Applies special logic to “time” SMART! The Challenges! Database Engine Comparison MySQL ▸ Generic relational database ▸ “time” is a generic data value ▸ Doesn’t apply special logic DUMB! 14
  15. 15. The Challenges! Query Comparison InfluxDB “SELECT count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > now() - 7d GROUP BY time(1h)” 15
  16. 16. The Challenges! Query Comparison MySQL “SELECT time, count(bar1) FROM foo WHERE bar1 > 0 AND bar2 > 0 AND time > NOW(6) - INTERVAL 7 DAY GROUP BY YEAR(time), MONTH(time), DAY(time), HOUR(time) ORDER BY time ASC” 16
  17. 17. The Challenges! InfluxQL != SQL InfluxQL != SQL ▸ InfluxQL is SQL-like, but different enough that it can’t be passed through to MySQL 17
  18. 18. The Challenges! InfluxQL != SQL ▸ “time” is SELECT’d automatically implicitly in InfluxDB ▹ “SELECT bar FROM foo”  “SELECT time, bar FROM FOO” ▸ GROUP’ing on “time” is smart in InfluxDB, and dumb in MySQL ▹ See slides 16 and 17 ▸ “time” in epoch format with millisecond precision is a float in MySQL ▹ “FROM_UNIXTIME(1444667802.145)” 18
  19. 19. The Challenges! InfluxQL != SQL ▸ InfluxDB series are roughly equivalent to MySQL tables, but the management syntax is sometimes different ▹ “DROP SERIES FROM foo”  “DELETE FROM foo” ▸ InfluxDB allows quoted identifiers, MySQL is more restrictive ▹ “SELECT * FROM foo WHERE bar > ‘0’”  Silently returns garbage in MySQL ▹ Grafana quotes identifiers aggressively, so we had to patch it 19
  20. 20. The Challenges! InfluxQL != SQL There’s probably more InfluxQL- SQL incompatibilities, but these are the ones that we’ve run into with Grafana at Ingram Content Group. 20
  21. 21. The Benefits! 21
  22. 22. The Benefits! ▸ Allows us to leverage System Administrator’s and Operation’s existing knowledge of MySQL ▸ Allows us to process large amounts of data ▹ InfluxDB had issues when we tried to insert about a million points at a time, MySQL handles this with no problem 22
  23. 23. The Benefits! ▸ We can use full, regular SQL for our Grafana queries in addition to InfluxQL ▹ Think JOINs, etc. ▸ We’re doing massive batch inserts, and MySQL’s query cache makes Grafana queries significantly faster ▹ This is obviously use case specific, but shows how you can tune MySQL to your specific use case 23

×