SlideShare a Scribd company logo
1 of 43
Download to read offline
@zepouet#InfluxDB
:: InfluxDB ::
@zepouet
http://www.treeptik.fr
http://www.cloudunit.fr
http://www.labaixbidouille.com
@zepouet#InfluxDB
:: InfluxDB :: Time Series ::
• About Me
• What is a time serie ?
• State of the Art in 2015
• Why yet another product for time series ?
• Live Demo
• Q/A
@zepouet#InfluxDB
About Me
•Treeptik
•MarsJUG
•LabAixBidouille
What is a time series ?
Things happening in times…
@zepouet#InfluxDB
@zepouet#InfluxDB
Events, events… events
• Measurements (physical sensors…)
• Exceptions (applications)
• Page views
• User actions
• Commits Git
• Webapp Deployment
• Things appening in time
State of the Art :: 2015
@zepouet#InfluxDB
What we have to store ?
• At the moment, we have :
• Graphite
• OpenTSDB (events, Hadoop, HBase…)
• Kairos (events, rewrite from OpenTSBD)
• Ganglia (more present in BigData/Hadoop)
• And others…
@zepouet#InfluxDB
What we have to collect ?
• At the moment, we have :
• CollectD
• Sensu
• DropWizard/Metrics
• JMXTrans
• Jolokia
@YourTwitterHandle@YourTwitterHandle@zepouet#InfluxDB
Something missing…
@zepouet#InfluxDB
Because in 2015, we need
• Simple product to install and manage
• To store millions of points (IoT is here)
• HTTP native support (JSON)
• Build with API
• Automatically clear out old data
• Easy scalable : cloud is a buzzword
@YourTwitterHandle@YourTwitterHandle@zepouet#InfluxDB
UseCase : Fablab
@zepouet#InfluxDB
wiki.labaixbidouille.com/index.php?4tle=Domo4que
@zepouet#InfluxDB
Feedback
•Data volume :
•1 event / sensor / minute
•1 * 60 * 24 = 1440 events per day
•42.300 events per month
•518.400 events per year
•First error : use MYSQL
•Second error : bad pattern with InfluxDB
@zepouet#InfluxDB
1.21GIGAWATTS
@zepouet#InfluxDB
About InfluxDB
•An opensource distributed time series database
• ErrPlane
• MIT License
• Written in GO
• Young but awesome project
@zepouet#InfluxDB
InfluxDB :: design goals
• Simple to install and manage thank to Go.
• No external dependencies like Zookeeper and Hadoop.
• HTTP(s) interface for reading and writing data.
• Horizontally scalable.
• On disk and in memory. Most data is cold.
• Compute percentiles and others functions on the fly.
• Downsample data on different windows of time.
@zepouet#InfluxDB
InfluxDB :: installing
• MacOS : $ brew install influxdb
• Debian : $ sudo dpkg -i influxdb_latest_amd64.deb
• CentOS : $ sudo rpm -ivh influxdb-latest-1.x86_64.rpm
• Docker : $ docker run tutum/influxdb
• Soon ARM and Windows
@zepouet#InfluxDB
InfluxDB :: running
• $ influxdb -config=/usr/local/etc/influxdb.conf
• Ports
• 8083 : UI
• 8086 :API
• 8090 : Cluster management raft
• 8099 : Cluster management protobuf
@zepouet#InfluxDB
InfluxDB :: design
• Database (like in Mysql, Postgres…)
• Time Series (kind of like tables with time, sequence number and
columns)
• A timeserie is composed by points or events (kinds of like
rows)
• Primary index is always time
• Null values are not stored
• You can have millions of series
@zepouet#InfluxDB
InfluxDB :: security
• Cluster admins
• Database admins
• Database users
• Read permissions
• only certains series
• only queries with a column having a specific value (e.g. customer_id = 32)
• Write permissions
• only certains series
• only columns having a specific value
@zepouet#InfluxDB
InfluxDB :: create points
curl -X POST -d '[{"name":"temp","columns":
["celsius"],"points":[[23]]}]' ‘http://localhost:8086/db/
mydb/series?u=root&p=root
curl -G 'http://localhost:8086/db/mydb/series?
u=root&p=root' --data-urlencode "q=select * from temp"
@zepouet#InfluxDB
InfluxDB :: Pitfalls
• Schemaless Warning
• Data partinioning with one serie
Time Name Host Metrics
3236765 cpu web0 78
3236765 disk_io web0 98344
3236765 load db1 5
3236765 eth_0 ldap0 8755
@zepouet#InfluxDB
Time Name Host Metrics
3236765 disk_io web0 98344
3236766 disk_io web0 98354
3236767 disk_io web0 98224
3236768 disk_io web0 98994
Time Name Host Metrics
3236765 eth_0 ldap0 8755
3236766 eth_0 ldap0 8721
3236767 eth_0 ldap0 8734
3236768 eth_0 ldap0 8723
Time Name Host Metrics
3236765 cpu web0 78
3236766 cpu web0 77
3236767 cpu web0 79
3236768 cpu web0 76
Time Name Host Metrics
3236765 load db1 5
3236766 load db1 6
3236767 load db1 5
3236768 load db1 7
@zepouet#InfluxDB
InfluxDB :: Why so many series?
• To take advantage of the Storage engines
• Points are indexed by time, not by any other
columns
• Tricks : easily work with grafana
InfluxDB works best with large number of series with
fewer columns in each one
@zepouet#InfluxDB
:: Query Langage
• select * from /.*/ limit 1
• select val1, val2 from serverA
• select cpu from /server.*/
• select * from /.*/ where time > now() - 1h
• select * from /.*/ where time > ‘2013-08-12 23:32:00’
• select * from /.*/ group by time(10m)
• select count(val) from /.*/ group by time(10m)
• select percentile(val, 95) from /.*/ group by time(10m)
• select count(distinct(val)) from /.*/
@zepouet#InfluxDB
:: Query Langage
• DELETE
• delete from response_times where time < now() - 1h
• delete from /^stats.*/ where time < now() - 7d
• drop series response_times
• GROUP BY
• select count(type) from events group by time(10m);
• select count(type),type from events group by time(10m), type;
@zepouet#InfluxDB
:: Visualize and summarize
• Graphs
• Last 10 minutes
• Last 4 hours
• Last 24 hours
• Past week
• Past month
• All time
@zepouet#InfluxDB
:: Merging :: Series
• select count(type)
from user_events merge admin_events
group by time(10m)
• select mean(value)
from merge(/.*az.1.*.cpu/)
group by time(1h)
@zepouet#InfluxDB
:: Joining :: Series
• select hosta.value + hostb.value
from cpu_load as hosta inner join cpu_load as hostb
where hosta.host = 'hosta.influxdb.orb'
and hostb.host = ‘hostb.influxdb.org’;
• select errors_per_minute.value / page_views_per_minute.value
from errors_per_minute inner join page_views_per_minute
@zepouet#InfluxDB
:: Naming Strategy :: 0.8
• Tag versus Value
• Rule :
<tagName>.<tagValue>.serieName
• Examples :
arduino.uno.shield.ethernet.sensor.dht11.temperature
arduino.uno.shield.ethernet.sensor.dht11.temperature
arduino.uno.shield.wifi.sensor.dht22.humidity
arduino.uno.shield.wifi.sensor.dht22.humidity
@zepouet#InfluxDB
:: Naming Strategy :: 0.9+
• Migration processus
• Rule : serieName = serieName
• Tag are defined into JSON and indexed
{
"database" : "domotic",
"points": [
{
"name": "temperature_x",
"tags": {
"arduino": "uno",
"shield": "wifi",
"position": "indoor",
"sensor": "dht22",
},
"timestamp": "2015-03-28T14:50:00Z",
"fields": {
"celsius": 23.2,
"farenheit": 192
}
}
]
}
Continuous Queries
@zepouet#InfluxDB
:: Continuous Queries
• select count(type) from events
group by time(10m), type
into events.count_per_type.10m
DOWNSAMPLING
Next release
@zepouet#InfluxDB
Soon in april 2015
• New model Clustering
• Influx shell
• Tags indexed
• Backup
For Java Dev and Devops
@zepouet#InfluxDB
Libraries
• https://github.com/influxdb/influxdb-java
Official java client
• https://github.com/davidB/metrics-influxdb
A reporter for metrics which announces measurements to an InfluxDB server.
• https://github.com/vietj/vertx-influxdb-metrics
Proof of concept of reporting to InfluxDB
@zepouet#InfluxDB
davidb/metrics-influxdb
Non official plugin from https://github.com/dropwizard/metrics
@zepouet#InfluxDB
Carbon-influxdb
https://github.com/dropwizard/metrics
@YourTwitterHandle@YourTwitterHandle@zepouet#InfluxDB
Demo
@YourTwitterHandle@YourTwitterHandle@zepouet#InfluxDB
Q & A

More Related Content

What's hot

Debugging and Testing ES Systems
Debugging and Testing ES SystemsDebugging and Testing ES Systems
Debugging and Testing ES Systems
Chris Birchall
 

What's hot (20)

Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)Nodejs - A quick tour (v6)
Nodejs - A quick tour (v6)
 
Infrastructure as Code in Google Cloud
Infrastructure as Code in Google CloudInfrastructure as Code in Google Cloud
Infrastructure as Code in Google Cloud
 
Node.js - A Quick Tour II
Node.js - A Quick Tour IINode.js - A Quick Tour II
Node.js - A Quick Tour II
 
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutionsUsing Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
 
Terraform: Configuration Management for Cloud Services
Terraform: Configuration Management for Cloud ServicesTerraform: Configuration Management for Cloud Services
Terraform: Configuration Management for Cloud Services
 
Terraform: Cloud Configuration Management (WTC/IPC'16)
Terraform: Cloud Configuration Management (WTC/IPC'16)Terraform: Cloud Configuration Management (WTC/IPC'16)
Terraform: Cloud Configuration Management (WTC/IPC'16)
 
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLONPaul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
 
Workshop Infrastructure as Code - Suestra
Workshop Infrastructure as Code - SuestraWorkshop Infrastructure as Code - Suestra
Workshop Infrastructure as Code - Suestra
 
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
 
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et KibanaJournée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
Journée DevOps : Des dashboards pour tous avec ElasticSearch, Logstash et Kibana
 
Debugging and Testing ES Systems
Debugging and Testing ES SystemsDebugging and Testing ES Systems
Debugging and Testing ES Systems
 
Deploying Plone and Volto, the Hard Way
Deploying Plone and Volto, the Hard WayDeploying Plone and Volto, the Hard Way
Deploying Plone and Volto, the Hard Way
 
Infrastructure as Code: Introduction to Terraform
Infrastructure as Code: Introduction to TerraformInfrastructure as Code: Introduction to Terraform
Infrastructure as Code: Introduction to Terraform
 
Tim Panton - Presentation at Emerging Communications Conference & Awards (eCo...
Tim Panton - Presentation at Emerging Communications Conference & Awards (eCo...Tim Panton - Presentation at Emerging Communications Conference & Awards (eCo...
Tim Panton - Presentation at Emerging Communications Conference & Awards (eCo...
 
JavaScript Engines and Event Loop
JavaScript Engines and Event Loop JavaScript Engines and Event Loop
JavaScript Engines and Event Loop
 
Terraform 0.9 + good practices
Terraform 0.9 + good practicesTerraform 0.9 + good practices
Terraform 0.9 + good practices
 
JavaScript in 2015
JavaScript in 2015JavaScript in 2015
JavaScript in 2015
 
CPAN 模組二三事
CPAN 模組二三事CPAN 模組二三事
CPAN 模組二三事
 
Declarative & workflow based infrastructure with Terraform
Declarative & workflow based infrastructure with TerraformDeclarative & workflow based infrastructure with Terraform
Declarative & workflow based infrastructure with Terraform
 
ElasticSearch
ElasticSearchElasticSearch
ElasticSearch
 

Similar to Devoxx france 2015 influx db

Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
Serving Deep Learning Models At Scale With RedisAI: Luca AntigaServing Deep Learning Models At Scale With RedisAI: Luca Antiga
Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
Redis Labs
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
Wyatt Fang
 

Similar to Devoxx france 2015 influx db (20)

Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
 
Lessons Learned Running InfluxDB Cloud and Other Cloud Services at Scale by T...
Lessons Learned Running InfluxDB Cloud and Other Cloud Services at Scale by T...Lessons Learned Running InfluxDB Cloud and Other Cloud Services at Scale by T...
Lessons Learned Running InfluxDB Cloud and Other Cloud Services at Scale by T...
 
Running Airflow Workflows as ETL Processes on Hadoop
Running Airflow Workflows as ETL Processes on HadoopRunning Airflow Workflows as ETL Processes on Hadoop
Running Airflow Workflows as ETL Processes on Hadoop
 
Lessons Learned: Running InfluxDB Cloud and Other Cloud Services at Scale | T...
Lessons Learned: Running InfluxDB Cloud and Other Cloud Services at Scale | T...Lessons Learned: Running InfluxDB Cloud and Other Cloud Services at Scale | T...
Lessons Learned: Running InfluxDB Cloud and Other Cloud Services at Scale | T...
 
Cloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and LogsCloud Foundry Monitoring How-To: Collecting Metrics and Logs
Cloud Foundry Monitoring How-To: Collecting Metrics and Logs
 
Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.
 
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd についてKubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
 
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.comUsing Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
 
ETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetupETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetup
 
Introduction to NodeJS with LOLCats
Introduction to NodeJS with LOLCatsIntroduction to NodeJS with LOLCats
Introduction to NodeJS with LOLCats
 
Hydra - Getting Started
Hydra - Getting StartedHydra - Getting Started
Hydra - Getting Started
 
J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"
J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"
J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"
 
Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
Serving Deep Learning Models At Scale With RedisAI: Luca AntigaServing Deep Learning Models At Scale With RedisAI: Luca Antiga
Serving Deep Learning Models At Scale With RedisAI: Luca Antiga
 
Deploying to Ubuntu on Linode
Deploying to Ubuntu on LinodeDeploying to Ubuntu on Linode
Deploying to Ubuntu on Linode
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
 
Functional Hostnames and Why they are Bad
Functional Hostnames and Why they are BadFunctional Hostnames and Why they are Bad
Functional Hostnames and Why they are Bad
 
[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo
 
Server-Side JavaScript Developement - Node.JS Quick Tour
Server-Side JavaScript Developement - Node.JS Quick TourServer-Side JavaScript Developement - Node.JS Quick Tour
Server-Side JavaScript Developement - Node.JS Quick Tour
 

Recently uploaded

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
 
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAMWSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
WSO2CON 2024 - Unlocking the Identity: Embracing CIAM 2.0 for a Competitive A...
 
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
 
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next IntegrationWSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptx
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
 
WSO2CON 2024 - Building a Digital Government in Uganda
WSO2CON 2024 - Building a Digital Government in UgandaWSO2CON 2024 - Building a Digital Government in Uganda
WSO2CON 2024 - Building a Digital Government in Uganda
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 

Devoxx france 2015 influx db