You got database in my cloud (short version)

L
@stillinbeta / #Kubecon Seattle 2018
You got Database
in my Cloud!
Kubernetes Foreign Data Wrapper
for Postgres
@stillinbeta / #Kubecon Seattle 2018
Who am I?
Liz Frost
Twitter: @stillinbeta
slack.k8s.io: @liz
@stillinbeta / #Kubecon Seattle 2018
Who am I?
@stillinbeta / #Kubecon Seattle 2018
Who am I?
@stillinbeta / #Kubecon Seattle 2018
Who am I?
@stillinbeta / #Kubecon Seattle 2018
Who am I?
(Stickers are available)
@stillinbeta / #Kubecon Seattle 2018
@stillinbeta / #Kubecon Seattle 2018
...get it?
zéro un zéro
cero uno cero
零一零
@stillinbeta / #Kubecon Seattle 2018
What is a Foreign Data Wrapper?
● ODBC
● SQLite
● Cassandra
● Redis
● Hue Bulbs?
● FDWs??
● Kubernetes!
@stillinbeta / #Kubecon Seattle 2018
@stillinbeta / #Kubecon Seattle 2018
Success! Someone did the hard part for us!
github.com/dennwc/go-fdw
@stillinbeta / #Kubecon Seattle 2018
CREATE EXTENSION IF NOT EXISTS k8s_fdw;
CREATE SERVER IF NOT EXISTS kind
FOREIGN DATA WRAPPER k8s_fdw
OPTIONS (kubeconfig '/kubeconfig');
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
What kid of interface do we want?
@stillinbeta / #Kubecon Seattle 2018
CREATE EXTENSION IF NOT EXISTS k8s_fdw;
CREATE SERVER IF NOT EXISTS kind
FOREIGN DATA WRAPPER k8s_fdw
OPTIONS (kubeconfig '/kubeconfig');
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
What kid of interface do we want?
@stillinbeta / #Kubecon Seattle 2018
CREATE EXTENSION IF NOT EXISTS k8s_fdw;
CREATE SERVER IF NOT EXISTS kind
FOREIGN DATA WRAPPER k8s_fdw
OPTIONS (kubeconfig '/kubeconfig');
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
What kid of interface do we want?
@stillinbeta / #Kubecon Seattle 2018
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
Give it a run...
@stillinbeta / #Kubecon Seattle 2018
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
ERROR: couldn't get kubeconfig: couldn't
get resource mapper: could not get api
group resources: Get
https://ec2-54-215-202-190.compute-1.amazo
naws.com:6443/api?timeout=32s: net/http:
invalid header field value "postgres:
postgres postgres [local]
SELECTx00x00x00x00x00x00x00x00x00
x00/v0.0.0 (linux/amd64)
kubernetes/$Format" for key User-Agent
oh.
@stillinbeta / #Kubecon Seattle 2018
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
name | namespace
--------------------------------+-------------
coredns-6f685fffbf-7xrtp | kube-system
coredns-6f685fffbf-nzfn7 | kube-system
etcd-master | kube-system
kube-apiserver-master | kube-system
kube-controller-manager-master | kube-system
kube-proxy-lk2mq | kube-system
kube-scheduler-master | kube-system
That’s better!
@stillinbeta / #Kubecon Seattle 2018
What about more
complicated objects?
@stillinbeta / #Kubecon Seattle 2018
How about
JSONPATH?
@stillinbeta / #Kubecon Seattle 2018
ALTER FOREIGN TABLE pods
ADD COLUMN container text
OPTIONS (alias '{.spec.containers[0].image}')
;
Let’s do that!
@stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018
SELECT * from PODS;
name | namespace | container
--------------------------------+-------------+--------------------------------------------
coredns-6f685fffbf-7xrtp | kube-system | k8s.gcr.io/coredns:1.2.6
coredns-6f685fffbf-nzfn7 | kube-system | k8s.gcr.io/coredns:1.2.6
etcd-master | kube-system | k8s.gcr.io/etcd:3.2.24
kube-apiserver-master | kube-system | k8s.gcr.io/kube-apiserver:v1.12.1
kube-controller-manager-master | kube-system | k8s.gcr.io/kube-controller-manager:v1.12.1
kube-proxy-lk2mq | kube-system | k8s.gcr.io/kube-proxy:v1.12.1
kube-scheduler-master | kube-system | k8s.gcr.io/kube-scheduler:v1.12.1
@stillinbeta / #Kubecon Seattle 2018
How about a map?
@stillinbeta / #Kubecon Seattle 2018
ALTER FOREIGN TABLE pods
ADD COLUMN labels jsonb
OPTIONS (alias 'metadata.labels')
;
jsonb to the rescue
@stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018
SELECT name, labels FROM pods;
name | labels
--------------------------------+-----------------------------------------------------------------------------------------------------
coredns-6f685fffbf-7xrtp | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"}
coredns-6f685fffbf-nzfn7 | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"}
etcd-master | {"tier": "control-plane", "component": "etcd"}
kube-apiserver-master | {"tier": "control-plane", "component": "kube-apiserver"}
kube-controller-manager-master | {"tier": "control-plane", "component": "kube-controller-manager"}
kube-proxy-lk2mq | {"k8s-app": "kube-proxy", "pod-template-generation": "1", "controller-revision-hash": "6cbfff58bb"}
kube-scheduler-master | {"tier": "control-plane", "component": "kube-scheduler"}
(7 rows)
@stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018
SELECT name, container, labels->'component' AS component FROM pods;
name | container | component
--------------------------------+--------------------------------------------+-------------------------
--
coredns-6f685fffbf-7xrtp | k8s.gcr.io/coredns:1.2.6 |
coredns-6f685fffbf-nzfn7 | k8s.gcr.io/coredns:1.2.6 |
etcd-master | k8s.gcr.io/etcd:3.2.24 | "etcd"
kube-apiserver-master | k8s.gcr.io/kube-apiserver:v1.12.1 | "kube-apiserver"
kube-controller-manager-master | k8s.gcr.io/kube-controller-manager:v1.12.1 |
"kube-controller-manager"
kube-proxy-lk2mq | k8s.gcr.io/kube-proxy:v1.12.1 |
kube-scheduler-master | k8s.gcr.io/kube-scheduler:v1.12.1 | "kube-scheduler"
@stillinbeta / #Kubecon Seattle 2018
And for my
final trick:
@stillinbeta / #Kubecon Seattle 2018
CREATE FOREIGN TABLE IF NOT EXISTS replica_sets (
name text OPTIONS (alias 'metadata.name')
, replicas bigint OPTIONS (alias 'status.replicas')
, available bigint OPTIONS (alias 'status.availableReplicas')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'apps/v1'
, kind 'ReplicaSet'
);
CREATE FOREIGN TABLE IF NOT EXISTS deployments (
name text OPTIONS (alias 'metadata.name')
, replicas bigint OPTIONS (alias 'status.replicas')
, available bigint OPTIONS (alias 'status.availableReplicas')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'apps/v1'
, kind 'Deployment'
);
A few more tables
@stillinbeta / #Kubecon Seattle 2018
CREATE FOREIGN TABLE IF NOT EXISTS replica_sets (
name text OPTIONS (alias 'metadata.name')
, replicas bigint OPTIONS (alias 'status.replicas')
, available bigint OPTIONS (alias 'status.availableReplicas')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'apps/v1'
, kind 'ReplicaSet'
);
CREATE FOREIGN TABLE IF NOT EXISTS deployments (
name text OPTIONS (alias 'metadata.name')
, replicas bigint OPTIONS (alias 'status.replicas')
, available bigint OPTIONS (alias 'status.availableReplicas')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'apps/v1'
, kind 'Deployment'
);
Just the important bits
@stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018
SELECT "deployments"."name" AS deployment_name
, "replica_sets"."name" as replica_name
, "pods"."name" AS pod_name
FROM deployments
JOIN replica_sets on "replica_sets"."name" LIKE "deployments"."name" || '-%'
JOIN pods on "pods"."name" LIKE "replica_sets"."name" || '-%';
deployment_name | replica_name | pod_name
-----------------+--------------------+--------------------------
coredns | coredns-6f685fffbf | coredns-6f685fffbf-7xrtp
coredns | coredns-6f685fffbf | coredns-6f685fffbf-nzfn7
@stillinbeta / #Kubecon Seattle 2018
● Column types mostly ignored
What doesn’t work?
@stillinbeta / #Kubecon Seattle 2018
● Column types mostly ignored
● If it’s a not a number, string, or map...
...just kind of give up
What doesn’t work?
@stillinbeta / #Kubecon Seattle 2018
● Column types mostly ignored
● If it’s a not a number, string, or map...
...just kind of give up
● The codebase not being a knot of
spaghetti
What doesn’t work?
@stillinbeta / #Kubecon Seattle 2018
Questions?
Concerns?
Come find me!
I’m the one with pink hair!
Twitter: @stillinbeta
slack.k8s.io: @liz
Github: github.com/liztio/k8s-fdw
Docker: liztio/k8s_fdw:master
1 of 33

Recommended

Using Spinnaker to Create a Development Workflow on Kubernetes - Paul Czarkowski by
Using Spinnaker to Create a Development Workflow on Kubernetes - Paul CzarkowskiUsing Spinnaker to Create a Development Workflow on Kubernetes - Paul Czarkowski
Using Spinnaker to Create a Development Workflow on Kubernetes - Paul CzarkowskiVMware Tanzu
422 views118 slides
Scripting Embulk Plugins by
Scripting Embulk PluginsScripting Embulk Plugins
Scripting Embulk PluginsSadayuki Furuhashi
2.6K views14 slides
Embuk internals by
Embuk internalsEmbuk internals
Embuk internalsSadayuki Furuhashi
3.2K views9 slides
Automating Workflows for Analytics Pipelines by
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesSadayuki Furuhashi
1.9K views28 slides
Big data Lambda Architecture - Batch Layer Hands On by
Big data Lambda Architecture - Batch Layer Hands OnBig data Lambda Architecture - Batch Layer Hands On
Big data Lambda Architecture - Batch Layer Hands Onhkbhadraa
515 views25 slides
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020 by
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020
Five Great Ways to Lose Data on Kubernetes - KubeCon EU 2020Altinity Ltd
912 views37 slides

More Related Content

What's hot

GCPUG meetup 201610 - Dataflow Introduction by
GCPUG meetup 201610 - Dataflow IntroductionGCPUG meetup 201610 - Dataflow Introduction
GCPUG meetup 201610 - Dataflow IntroductionSimon Su
1.3K views37 slides
Data integration with embulk by
Data integration with embulkData integration with embulk
Data integration with embulkTeguh Nugraha
1.2K views21 slides
More kibana by
More kibanaMore kibana
More kibana琛琳 饶
5K views41 slides
CODAIT/Spark-Bench by
CODAIT/Spark-BenchCODAIT/Spark-Bench
CODAIT/Spark-BenchEmily Curtin
123 views78 slides
High Performance, High Reliability Data Loading on ClickHouse by
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
2.3K views37 slides
Embulk - 進化するバルクデータローダ by
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダSadayuki Furuhashi
4.9K views35 slides

What's hot(20)

GCPUG meetup 201610 - Dataflow Introduction by Simon Su
GCPUG meetup 201610 - Dataflow IntroductionGCPUG meetup 201610 - Dataflow Introduction
GCPUG meetup 201610 - Dataflow Introduction
Simon Su1.3K views
Data integration with embulk by Teguh Nugraha
Data integration with embulkData integration with embulk
Data integration with embulk
Teguh Nugraha1.2K views
High Performance, High Reliability Data Loading on ClickHouse by Altinity Ltd
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
Altinity Ltd2.3K views
Embulk - 進化するバルクデータローダ by Sadayuki Furuhashi
Embulk - 進化するバルクデータローダEmbulk - 進化するバルクデータローダ
Embulk - 進化するバルクデータローダ
Sadayuki Furuhashi4.9K views
Docker in Action by Simon Su
Docker in ActionDocker in Action
Docker in Action
Simon Su287 views
Autoscaling Your Kubernetes Workloads (Sponsored by Datadog) - AWS Summit Sydney by Amazon Web Services
Autoscaling Your Kubernetes Workloads (Sponsored by Datadog) - AWS Summit SydneyAutoscaling Your Kubernetes Workloads (Sponsored by Datadog) - AWS Summit Sydney
Autoscaling Your Kubernetes Workloads (Sponsored by Datadog) - AWS Summit Sydney
Monitoring Spark Applications by Tzach Zohar
Monitoring Spark ApplicationsMonitoring Spark Applications
Monitoring Spark Applications
Tzach Zohar13.9K views
Cncf event driven autoscaling with keda by JurajHantk
Cncf   event driven autoscaling with kedaCncf   event driven autoscaling with keda
Cncf event driven autoscaling with keda
JurajHantk97 views
Big data lambda architecture - Streaming Layer Hands On by hkbhadraa
Big data lambda architecture - Streaming Layer Hands OnBig data lambda architecture - Streaming Layer Hands On
Big data lambda architecture - Streaming Layer Hands On
hkbhadraa698 views
Kubernetes Service Catalog & Open Service Broker for Azure by Julien Corioland
Kubernetes Service Catalog & Open Service Broker for AzureKubernetes Service Catalog & Open Service Broker for Azure
Kubernetes Service Catalog & Open Service Broker for Azure
Julien Corioland188 views
How to Live in a Post-Spring-Cloud-Netflix World - Olga Maciaszek-Sharma & Ja... by VMware Tanzu
How to Live in a Post-Spring-Cloud-Netflix World - Olga Maciaszek-Sharma & Ja...How to Live in a Post-Spring-Cloud-Netflix World - Olga Maciaszek-Sharma & Ja...
How to Live in a Post-Spring-Cloud-Netflix World - Olga Maciaszek-Sharma & Ja...
VMware Tanzu1.4K views
Prometheus Monitoring Mixins (Berlin CNCB Meetup) by Kausal
Prometheus Monitoring Mixins (Berlin CNCB Meetup)Prometheus Monitoring Mixins (Berlin CNCB Meetup)
Prometheus Monitoring Mixins (Berlin CNCB Meetup)
Kausal1K views
MapReduce with Scalding @ 24th Hadoop London Meetup by Landoop Ltd
MapReduce with Scalding @ 24th Hadoop London MeetupMapReduce with Scalding @ 24th Hadoop London Meetup
MapReduce with Scalding @ 24th Hadoop London Meetup
Landoop Ltd2.5K views
FlutterでGraphQLを扱う by IgaHironobu
FlutterでGraphQLを扱うFlutterでGraphQLを扱う
FlutterでGraphQLを扱う
IgaHironobu2.9K views
Introducing Scylla Manager: Cluster Management and Task Automation by ScyllaDB
Introducing Scylla Manager: Cluster Management and Task AutomationIntroducing Scylla Manager: Cluster Management and Task Automation
Introducing Scylla Manager: Cluster Management and Task Automation
ScyllaDB1.9K views

Similar to You got database in my cloud (short version)

You got database in my cloud! by
You got database  in my cloud!You got database  in my cloud!
You got database in my cloud!Liz Frost
207 views80 slides
Kubernetes + Python = ❤ - Cloud Native Prague by
Kubernetes + Python = ❤ - Cloud Native PragueKubernetes + Python = ❤ - Cloud Native Prague
Kubernetes + Python = ❤ - Cloud Native PragueHenning Jacobs
3.7K views75 slides
使用 Prometheus 監控 Kubernetes Cluster by
使用 Prometheus 監控 Kubernetes Cluster 使用 Prometheus 監控 Kubernetes Cluster
使用 Prometheus 監控 Kubernetes Cluster inwin stack
2.5K views45 slides
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム by
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォームPivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォームMasayuki Matsushita
501 views36 slides
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup by
Kubernetes Architecture and Introduction – Paris Kubernetes MeetupKubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes MeetupStefan Schimanski
5.5K views45 slides
MySQL Audit using Percona audit plugin and ELK by
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKYoungHeon (Roy) Kim
4.2K views40 slides

Similar to You got database in my cloud (short version)(20)

You got database in my cloud! by Liz Frost
You got database  in my cloud!You got database  in my cloud!
You got database in my cloud!
Liz Frost207 views
Kubernetes + Python = ❤ - Cloud Native Prague by Henning Jacobs
Kubernetes + Python = ❤ - Cloud Native PragueKubernetes + Python = ❤ - Cloud Native Prague
Kubernetes + Python = ❤ - Cloud Native Prague
Henning Jacobs3.7K views
使用 Prometheus 監控 Kubernetes Cluster by inwin stack
使用 Prometheus 監控 Kubernetes Cluster 使用 Prometheus 監控 Kubernetes Cluster
使用 Prometheus 監控 Kubernetes Cluster
inwin stack2.5K views
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム by Masayuki Matsushita
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォームPivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup by Stefan Schimanski
Kubernetes Architecture and Introduction – Paris Kubernetes MeetupKubernetes Architecture and Introduction – Paris Kubernetes Meetup
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Stefan Schimanski5.5K views
MySQL Audit using Percona audit plugin and ELK by YoungHeon (Roy) Kim
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELK
YoungHeon (Roy) Kim4.2K views
Build Your Own CaaS (Container as a Service) by HungWei Chiu
Build Your Own CaaS (Container as a Service)Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)
HungWei Chiu726 views
GitOps - Operation By Pull Request by Kasper Nissen
GitOps - Operation By Pull RequestGitOps - Operation By Pull Request
GitOps - Operation By Pull Request
Kasper Nissen1.4K views
OSDN: Serverless technologies with Kubernetes by Provectus
OSDN: Serverless technologies with Kubernetes OSDN: Serverless technologies with Kubernetes
OSDN: Serverless technologies with Kubernetes
Provectus96 views
What is serveless? by Provectus
What is serveless? What is serveless?
What is serveless?
Provectus183 views
KubeFuse - A File-System for Kubernetes by Bart Spaans
KubeFuse - A File-System for KubernetesKubeFuse - A File-System for Kubernetes
KubeFuse - A File-System for Kubernetes
Bart Spaans1K views
Monitoring infrastructure with prometheus by Shahnawaz Saifi
Monitoring infrastructure with prometheusMonitoring infrastructure with prometheus
Monitoring infrastructure with prometheus
Shahnawaz Saifi188 views
Kube Your Enthusiasm by VMware Tanzu
Kube Your EnthusiasmKube Your Enthusiasm
Kube Your Enthusiasm
VMware Tanzu323 views
Kube Your Enthusiasm - Paul Czarkowski by VMware Tanzu
Kube Your Enthusiasm - Paul CzarkowskiKube Your Enthusiasm - Paul Czarkowski
Kube Your Enthusiasm - Paul Czarkowski
VMware Tanzu1.2K views
Istio Playground by QAware GmbH
Istio PlaygroundIstio Playground
Istio Playground
QAware GmbH528 views
[Hands-on] Kubernetes | Nov 18, 2017 by Oracle Korea
[Hands-on] Kubernetes | Nov 18, 2017[Hands-on] Kubernetes | Nov 18, 2017
[Hands-on] Kubernetes | Nov 18, 2017
Oracle Korea258 views

Recently uploaded

Neo4j y GenAI by
Neo4j y GenAI Neo4j y GenAI
Neo4j y GenAI Neo4j
42 views41 slides
Unleash The Monkeys by
Unleash The MonkeysUnleash The Monkeys
Unleash The MonkeysJacob Duijzer
7 views28 slides
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -... by
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...Deltares
6 views15 slides
Winter '24 Release Chat.pdf by
Winter '24 Release Chat.pdfWinter '24 Release Chat.pdf
Winter '24 Release Chat.pdfmelbourneauuser
9 views20 slides
Software testing company in India.pptx by
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptxSakshiPatel82
7 views9 slides
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023 by
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023Upgrading Incident Management with Icinga - Icinga Camp Milan 2023
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023Icinga
38 views17 slides

Recently uploaded(20)

Neo4j y GenAI by Neo4j
Neo4j y GenAI Neo4j y GenAI
Neo4j y GenAI
Neo4j42 views
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -... by Deltares
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
Deltares6 views
Software testing company in India.pptx by SakshiPatel82
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptx
SakshiPatel827 views
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023 by Icinga
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023Upgrading Incident Management with Icinga - Icinga Camp Milan 2023
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023
Icinga38 views
El Arte de lo Possible by Neo4j
El Arte de lo PossibleEl Arte de lo Possible
El Arte de lo Possible
Neo4j38 views
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut... by HCLSoftware
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...
HCLSoftware6 views
DSD-INT 2023 Baseline studies for Strategic Coastal protection for Long Islan... by Deltares
DSD-INT 2023 Baseline studies for Strategic Coastal protection for Long Islan...DSD-INT 2023 Baseline studies for Strategic Coastal protection for Long Islan...
DSD-INT 2023 Baseline studies for Strategic Coastal protection for Long Islan...
Deltares11 views
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko... by Deltares
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
Deltares11 views
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares9 views
A first look at MariaDB 11.x features and ideas on how to use them by Federico Razzoli
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use them
Federico Razzoli45 views
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove... by Deltares
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...
Deltares17 views
MariaDB stored procedures and why they should be improved by Federico Razzoli
MariaDB stored procedures and why they should be improvedMariaDB stored procedures and why they should be improved
MariaDB stored procedures and why they should be improved
SUGCON ANZ Presentation V2.1 Final.pptx by Jack Spektor
SUGCON ANZ Presentation V2.1 Final.pptxSUGCON ANZ Presentation V2.1 Final.pptx
SUGCON ANZ Presentation V2.1 Final.pptx
Jack Spektor22 views
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ... by marksimpsongw
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
marksimpsongw76 views

You got database in my cloud (short version)

  • 1. @stillinbeta / #Kubecon Seattle 2018 You got Database in my Cloud! Kubernetes Foreign Data Wrapper for Postgres
  • 2. @stillinbeta / #Kubecon Seattle 2018 Who am I? Liz Frost Twitter: @stillinbeta slack.k8s.io: @liz
  • 3. @stillinbeta / #Kubecon Seattle 2018 Who am I?
  • 4. @stillinbeta / #Kubecon Seattle 2018 Who am I?
  • 5. @stillinbeta / #Kubecon Seattle 2018 Who am I?
  • 6. @stillinbeta / #Kubecon Seattle 2018 Who am I? (Stickers are available)
  • 7. @stillinbeta / #Kubecon Seattle 2018
  • 8. @stillinbeta / #Kubecon Seattle 2018 ...get it? zéro un zéro cero uno cero 零一零
  • 9. @stillinbeta / #Kubecon Seattle 2018 What is a Foreign Data Wrapper? ● ODBC ● SQLite ● Cassandra ● Redis ● Hue Bulbs? ● FDWs?? ● Kubernetes!
  • 10. @stillinbeta / #Kubecon Seattle 2018
  • 11. @stillinbeta / #Kubecon Seattle 2018 Success! Someone did the hard part for us! github.com/dennwc/go-fdw
  • 12. @stillinbeta / #Kubecon Seattle 2018 CREATE EXTENSION IF NOT EXISTS k8s_fdw; CREATE SERVER IF NOT EXISTS kind FOREIGN DATA WRAPPER k8s_fdw OPTIONS (kubeconfig '/kubeconfig'); CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); What kid of interface do we want?
  • 13. @stillinbeta / #Kubecon Seattle 2018 CREATE EXTENSION IF NOT EXISTS k8s_fdw; CREATE SERVER IF NOT EXISTS kind FOREIGN DATA WRAPPER k8s_fdw OPTIONS (kubeconfig '/kubeconfig'); CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); What kid of interface do we want?
  • 14. @stillinbeta / #Kubecon Seattle 2018 CREATE EXTENSION IF NOT EXISTS k8s_fdw; CREATE SERVER IF NOT EXISTS kind FOREIGN DATA WRAPPER k8s_fdw OPTIONS (kubeconfig '/kubeconfig'); CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); What kid of interface do we want?
  • 15. @stillinbeta / #Kubecon Seattle 2018 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE Give it a run...
  • 16. @stillinbeta / #Kubecon Seattle 2018 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE ERROR: couldn't get kubeconfig: couldn't get resource mapper: could not get api group resources: Get https://ec2-54-215-202-190.compute-1.amazo naws.com:6443/api?timeout=32s: net/http: invalid header field value "postgres: postgres postgres [local] SELECTx00x00x00x00x00x00x00x00x00 x00/v0.0.0 (linux/amd64) kubernetes/$Format" for key User-Agent oh.
  • 17. @stillinbeta / #Kubecon Seattle 2018 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE name | namespace --------------------------------+------------- coredns-6f685fffbf-7xrtp | kube-system coredns-6f685fffbf-nzfn7 | kube-system etcd-master | kube-system kube-apiserver-master | kube-system kube-controller-manager-master | kube-system kube-proxy-lk2mq | kube-system kube-scheduler-master | kube-system That’s better!
  • 18. @stillinbeta / #Kubecon Seattle 2018 What about more complicated objects?
  • 19. @stillinbeta / #Kubecon Seattle 2018 How about JSONPATH?
  • 20. @stillinbeta / #Kubecon Seattle 2018 ALTER FOREIGN TABLE pods ADD COLUMN container text OPTIONS (alias '{.spec.containers[0].image}') ; Let’s do that!
  • 21. @stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018 SELECT * from PODS; name | namespace | container --------------------------------+-------------+-------------------------------------------- coredns-6f685fffbf-7xrtp | kube-system | k8s.gcr.io/coredns:1.2.6 coredns-6f685fffbf-nzfn7 | kube-system | k8s.gcr.io/coredns:1.2.6 etcd-master | kube-system | k8s.gcr.io/etcd:3.2.24 kube-apiserver-master | kube-system | k8s.gcr.io/kube-apiserver:v1.12.1 kube-controller-manager-master | kube-system | k8s.gcr.io/kube-controller-manager:v1.12.1 kube-proxy-lk2mq | kube-system | k8s.gcr.io/kube-proxy:v1.12.1 kube-scheduler-master | kube-system | k8s.gcr.io/kube-scheduler:v1.12.1
  • 22. @stillinbeta / #Kubecon Seattle 2018 How about a map?
  • 23. @stillinbeta / #Kubecon Seattle 2018 ALTER FOREIGN TABLE pods ADD COLUMN labels jsonb OPTIONS (alias 'metadata.labels') ; jsonb to the rescue
  • 24. @stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018 SELECT name, labels FROM pods; name | labels --------------------------------+----------------------------------------------------------------------------------------------------- coredns-6f685fffbf-7xrtp | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"} coredns-6f685fffbf-nzfn7 | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"} etcd-master | {"tier": "control-plane", "component": "etcd"} kube-apiserver-master | {"tier": "control-plane", "component": "kube-apiserver"} kube-controller-manager-master | {"tier": "control-plane", "component": "kube-controller-manager"} kube-proxy-lk2mq | {"k8s-app": "kube-proxy", "pod-template-generation": "1", "controller-revision-hash": "6cbfff58bb"} kube-scheduler-master | {"tier": "control-plane", "component": "kube-scheduler"} (7 rows)
  • 25. @stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018 SELECT name, container, labels->'component' AS component FROM pods; name | container | component --------------------------------+--------------------------------------------+------------------------- -- coredns-6f685fffbf-7xrtp | k8s.gcr.io/coredns:1.2.6 | coredns-6f685fffbf-nzfn7 | k8s.gcr.io/coredns:1.2.6 | etcd-master | k8s.gcr.io/etcd:3.2.24 | "etcd" kube-apiserver-master | k8s.gcr.io/kube-apiserver:v1.12.1 | "kube-apiserver" kube-controller-manager-master | k8s.gcr.io/kube-controller-manager:v1.12.1 | "kube-controller-manager" kube-proxy-lk2mq | k8s.gcr.io/kube-proxy:v1.12.1 | kube-scheduler-master | k8s.gcr.io/kube-scheduler:v1.12.1 | "kube-scheduler"
  • 26. @stillinbeta / #Kubecon Seattle 2018 And for my final trick:
  • 27. @stillinbeta / #Kubecon Seattle 2018 CREATE FOREIGN TABLE IF NOT EXISTS replica_sets ( name text OPTIONS (alias 'metadata.name') , replicas bigint OPTIONS (alias 'status.replicas') , available bigint OPTIONS (alias 'status.availableReplicas') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'apps/v1' , kind 'ReplicaSet' ); CREATE FOREIGN TABLE IF NOT EXISTS deployments ( name text OPTIONS (alias 'metadata.name') , replicas bigint OPTIONS (alias 'status.replicas') , available bigint OPTIONS (alias 'status.availableReplicas') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'apps/v1' , kind 'Deployment' ); A few more tables
  • 28. @stillinbeta / #Kubecon Seattle 2018 CREATE FOREIGN TABLE IF NOT EXISTS replica_sets ( name text OPTIONS (alias 'metadata.name') , replicas bigint OPTIONS (alias 'status.replicas') , available bigint OPTIONS (alias 'status.availableReplicas') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'apps/v1' , kind 'ReplicaSet' ); CREATE FOREIGN TABLE IF NOT EXISTS deployments ( name text OPTIONS (alias 'metadata.name') , replicas bigint OPTIONS (alias 'status.replicas') , available bigint OPTIONS (alias 'status.availableReplicas') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'apps/v1' , kind 'Deployment' ); Just the important bits
  • 29. @stillinbeta / #Kubecon Seattle 2018@stillinbeta / #Kubecon Seattle 2018 SELECT "deployments"."name" AS deployment_name , "replica_sets"."name" as replica_name , "pods"."name" AS pod_name FROM deployments JOIN replica_sets on "replica_sets"."name" LIKE "deployments"."name" || '-%' JOIN pods on "pods"."name" LIKE "replica_sets"."name" || '-%'; deployment_name | replica_name | pod_name -----------------+--------------------+-------------------------- coredns | coredns-6f685fffbf | coredns-6f685fffbf-7xrtp coredns | coredns-6f685fffbf | coredns-6f685fffbf-nzfn7
  • 30. @stillinbeta / #Kubecon Seattle 2018 ● Column types mostly ignored What doesn’t work?
  • 31. @stillinbeta / #Kubecon Seattle 2018 ● Column types mostly ignored ● If it’s a not a number, string, or map... ...just kind of give up What doesn’t work?
  • 32. @stillinbeta / #Kubecon Seattle 2018 ● Column types mostly ignored ● If it’s a not a number, string, or map... ...just kind of give up ● The codebase not being a knot of spaghetti What doesn’t work?
  • 33. @stillinbeta / #Kubecon Seattle 2018 Questions? Concerns? Come find me! I’m the one with pink hair! Twitter: @stillinbeta slack.k8s.io: @liz Github: github.com/liztio/k8s-fdw Docker: liztio/k8s_fdw:master