You got database in my cloud!

L
@stillinbeta // #postgresconf // NYC 2019
You got Database
in my Cloud!
Kubernetes Foreign Data Wrapper
for Postgres
Liz Frost
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
What is Kubernetes?
● Container Manager
● API- and Object-driven
● Highly Extensible
● Monumentally popular
@stillinbeta // #postgresconf // NYC 2019
Who am I?
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
...get it?
zéro un zéro
cero uno cero
零一零
@stillinbeta // #postgresconf // NYC 2019
What is a Foreign Data Wrapper (FDW)?
● ODBC
● SQLite
● Cassandra
● Redis
● Hue?
● FDWs?
● Kubernetes!
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
void
BeginForeignScan (ForeignScanState *node,
int eflags);
TupleTableSlot *
IterateForeignScan (ForeignScanState *node);
ForeignScan *
GetForeignPlan (PlannerInfo *root,
RelOptInfo *baserel,
Oid foreigntableid,
ForeignPath *best_path,
List *tlist,
List *scan_clauses,
Plan *outer_plan); That doesn’t look like Go...
@stillinbeta // #postgresconf // NYC 2019
How about Multicorn?
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
Why not Multicorn?
@stillinbeta // #postgresconf // NYC 2019
Why not Multicorn?
@stillinbeta // #postgresconf // NYC 2019
Native Extension?
@stillinbeta // #postgresconf // NYC 2019
Nope. No C.
@stillinbeta // #postgresconf // NYC 2019
Guess I’ll go
@stillinbeta // #postgresconf // NYC 2019
Success! Someone did the hard part for us!
@stillinbeta // #postgresconf // NYC 2019
github.com/dennwc/go-fdw
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
@stillinbeta // #postgresconf // NYC 2019
C <=> Go?
@stillinbeta // #postgresconf // NYC 2019
C <=> Go?
CGo!
@stillinbeta // #postgresconf // NYC 2019
// #include <stdio.h>
// #include <errno.h>
import "C"
How does cgo work?
@stillinbeta // #postgresconf // NYC 2019
//#cgo CFLAGS:
-I/usr/include/postgresql/9.6/server
-I/usr/include/postgresql/internal
//#cgo LDFLAGS: -Wl,-unresolved-symbols=ignore-all
//#include "postgres.h"
//#include "funcapi.h"
<snip>
Import “C”
How does cgo work?
@stillinbeta // #postgresconf // NYC 2019
// Property adds a key-value property to results of EXPLAIN query.
func (e Explainer) Property(k, v string) {
C.ExplainPropertyText(C.CString(k), C.CString(v), e.es)
}
So how do we call?
@stillinbeta // #postgresconf // NYC 2019
//static Datum cStringGetDatum(const char *str) {
// PG_RETURN_TEXT_P(CStringGetTextDatum(str));
//}
How do we use macros?
@stillinbeta // #postgresconf // NYC 2019
type Table interface {
Stats(opts *Options) TableStats
Scan(rel *Relation, opts *Options) Iterator
}
type Iterator interface {
Next() []interface{}
Reset()
Close() error
}
What’s our interface look like?
@stillinbeta // #postgresconf // NYC 2019
type Table interface {
Stats(opts *Options) (TableStats, error)
Scan(rel *Relation, opts *Options) (Iterator, error)
}
type Iterator interface {
Next() ([]interface{}, error)
Reset()
Close() error
}
Wouldn’t be Go without errors!
@stillinbeta // #postgresconf // NYC 2019
CREATE SERVER IF NOT EXISTS kind
FOREIGN DATA WRAPPER k8s_fdw
OPTIONS (kubeconfig '/kubeconfig');
What kind of interface do we want?
@stillinbeta // #postgresconf // NYC 2019
What’s a Kubeconfig?
@stillinbeta // #postgresconf // NYC 2019
apiVersion: v1
clusters:
- cluster:
server: http://localhost:8080
name: local-server
contexts:
- context:
cluster: local-server
namespace: the-right-prefix
user: myself
name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: myself
user:
password: secret
username: admin
Connection
Credentials
@stillinbeta // #postgresconf // NYC 2019
What does a Kubernetes
object look like?
@stillinbeta // #postgresconf // NYC 2019
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes!
&& sleep 3600']
- name: myapp-sidecar
image: stillinbeta/logging
status:
conditions:
- type: Ready
status: "True"
lastProbeTime: null
lastTransitionTime: 2018-01-01T00:00:00Z
Many fields, variable lengths, hierarchical...
@stillinbeta // #postgresconf // NYC 2019
11 fields
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes!
&& sleep 3600']
- name: myapp-sidecar
image: stillinbeta/logging
status:
conditions:
- type: Ready
status: "True"
lastProbeTime: null
lastTransitionTime: 2018-01-01T00:00:00Z
@stillinbeta // #postgresconf // NYC 2019
11 fields
...and this is a
small object
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes!
&& sleep 3600']
- name: myapp-sidecar
image: stillinbeta/logging
status:
conditions:
- type: Ready
status: "True"
lastProbeTime: null
lastTransitionTime: 2018-01-01T00:00:00Z
@stillinbeta // #postgresconf // NYC 2019
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/config.hash: 4b52d75cab61380f07c0c5a69fb371d4
kubernetes.io/config.mirror: 4b52d75cab61380f07c0c5a69fb371d4
kubernetes.io/config.seen: "2019-03-19T16:11:31.520506966Z"
kubernetes.io/config.source: file
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: "2019-03-19T16:12:57Z"
labels:
component: kube-scheduler
tier: control-plane
name: kube-scheduler-kate
namespace: kube-system
resourceVersion: "446"
selfLink: /api/v1/namespaces/kube-system/pods/kube-scheduler-kate
uid: dc176fb5-4a61-11e9-8e22-080027245760
spec:
containers:
- command:
- kube-scheduler
- --address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
image: k8s.gcr.io/kube-scheduler:v1.13.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
nodeName: kate
priority: 2000000000
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2019-03-19T16:11:32Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2019-03-19T16:11:36Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2019-03-19T16:11:36Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2019-03-19T16:11:32Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://aee2c6b8ed3a23a636f235183fbadaa6e827660ab49a7d8742a73f5143d376fd
image: k8s.gcr.io/kube-scheduler:v1.13.4
imageID: docker-pullable://k8s.gcr.io/kube-scheduler@sha256:e7b2f9b1dcfa03b0e43b891979075d62086fe14e169de081f9c23db378f5b2f7
lastState: {}
name: kube-scheduler
ready: true
restartCount: 0
state:
running:
startedAt: "2019-03-19T16:11:34Z"
hostIP: 10.0.2.15
phase: Running
podIP: 10.0.2.15
qosClass: Burstable
startTime: "2019-03-19T16:11:32Z"
Here’s a real-world
example
@stillinbeta // #postgresconf // NYC 2019
We probably don’t
care about every field!
@stillinbeta // #postgresconf // NYC 2019
CREATE SERVER IF NOT EXISTS kind
FOREIGN DATA WRAPPER k8s_fdw
OPTIONS (kubeconfig '/kubeconfig');
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
What kind of interface do we want?
@stillinbeta // #postgresconf // NYC 2019
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
Let’s try it out!
@stillinbeta // #postgresconf // NYC 2019
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
ERROR: couldn't get kubeconfig: couldn't get resource mapper:
could not get api group resources: Get
https://ec2-54-205-250-176.compute-1.amazonaws.com:6443/api?ti
meout=32s: net/http: invalid header field value "postgres:
postgres postgres [local]
SELECTx00x00x00x00x00x00x00x00x00x00/v0.0.0
(linux/amd64) kubernetes/$Format" for key User-Agent
oh.
@stillinbeta // #postgresconf // NYC 2019
$ psql --user postgres < test.sql
CREATE EXTENSION
CREATE SERVER
CREATE FOREIGN TABLE
name | namespace
--------------------------------+-------------
coredns-6f685fffbf-7xrtp | kube-system
coredns-6f685fffbf-nzfn7 | kube-system
etcd-master | kube-system
kube-apiserver-master | kube-system
kube-controller-manager-master | kube-system
kube-proxy-lk2mq | kube-system
kube-scheduler-master | kube-system
That’s better!
@stillinbeta // #postgresconf // NYC 2019
That’s boring. What
else can you do?
@stillinbeta // #postgresconf // NYC 2019
How about
JSONPATH?
@stillinbeta // #postgresconf // NYC 2019
...what’s JSONPATH?
@stillinbeta // #postgresconf // NYC 2019
text the plain text kind is {.kind} kind is List
@ the current object {@} the same as input
. or [] child operator {.kind} or {[‘kind’]} List
.. recursive descent {..name} 127.0.0.1 127.0.0.2 myself e2e
* wildcard. Get all objects {.items[*].metadata.name} [127.0.0.1 127.0.0.2]
[start:end :step] subscript operator {.users[0].name} myself
[,] union operator {.items[*][‘metadata.name’,
‘status.capacity’]}
127.0.0.1 127.0.0.2 map[cpu:4]
map[cpu:8]
?() filter {.users[?(@.name==“e2e”)].user.
password}
secret
range, end iterate list {range
.items[*]}[{.metadata.name},
{.status.capacity}] {end}
[127.0.0.1, map[cpu:4]]
[127.0.0.2, map[cpu:8]]
“ quote interpreted string {range
.items[*]}{.metadata.name}{’t’}{
end}
127.0.0.1 127.0.0.2
@stillinbeta // #postgresconf // NYC 2019
text the plain text {.kind}
. or [] child operator {.kind} or {[‘kind’]}
[start:end :step] subscript operator {.users[0].name}
@stillinbeta // #postgresconf // NYC 2019
Why JSONPath?
@stillinbeta // #postgresconf // NYC 2019
Why JSONPath?
$ kubectl get pods -o=jsonpath='{@}'
$ kubectl get pods -o=jsonpath='{.items[0]}'
$ kubectl get pods -o=jsonpath='{.items[0].metadata.name}'
$ kubectl get pods -o=jsonpath='{range.items[*]}{.metadata.name}{"t"}{.status.startTime}{"n"}{end}'
@stillinbeta // #postgresconf // NYC 2019
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
, container text OPTIONS (alias '{.spec.containers[0].image}')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
Let’s do that!
@stillinbeta // #postgresconf // NYC 2019
SELECT * from PODS;
name | namespace | container
--------------------------------+-------------+--------------------------------------------
coredns-6f685fffbf-7xrtp | kube-system | k8s.gcr.io/coredns:1.2.6
coredns-6f685fffbf-nzfn7 | kube-system | k8s.gcr.io/coredns:1.2.6
etcd-master | kube-system | k8s.gcr.io/etcd:3.2.24
kube-apiserver-master | kube-system | k8s.gcr.io/kube-apiserver:v1.12.1
kube-controller-manager-master | kube-system | k8s.gcr.io/kube-controller-manager:v1.12.1
kube-proxy-lk2mq | kube-system | k8s.gcr.io/kube-proxy:v1.12.1
kube-scheduler-master | kube-system | k8s.gcr.io/kube-scheduler:v1.12.1
@stillinbeta // #postgresconf // NYC 2019
That’s fine and dandy,
but not everything’s
a string!
@stillinbeta // #postgresconf // NYC 2019
Labels
object
Map of string keys and values that can be used to organize and
categorize (scope and select) objects. May match selectors of replication
controllers and services.
@stillinbeta // #postgresconf // NYC 2019
How about a map?
@stillinbeta // #postgresconf // NYC 2019
CREATE FOREIGN TABLE IF NOT EXISTS pods (
name text OPTIONS (alias 'metadata.name')
, namespace text OPTIONS (alias 'metadata.namespace')
, container text OPTIONS (alias '{.spec.containers[0].image}')
, labels jsonb OPTIONS (alias 'metadata.labels')
)
SERVER kind
OPTIONS (
namespace 'kube-system'
, apiVersion 'v1'
, kind 'Pod'
);
What kid of interface do we want?
@stillinbeta // #postgresconf // NYC 2019
SELECT name, labels FROM pods;
name | labels
--------------------------------+-----------------------------------------------------------------------------------------------------
coredns-6f685fffbf-7xrtp | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"}
coredns-6f685fffbf-nzfn7 | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"}
etcd-master | {"tier": "control-plane", "component": "etcd"}
kube-apiserver-master | {"tier": "control-plane", "component": "kube-apiserver"}
kube-controller-manager-master | {"tier": "control-plane", "component": "kube-controller-manager"}
kube-proxy-lk2mq | {"k8s-app": "kube-proxy", "pod-template-generation": "1", "controller-revision-hash": "6cbfff58bb"}
kube-scheduler-master | {"tier": "control-plane", "component": "kube-scheduler"}
(7 rows)
@stillinbeta // #postgresconf // NYC 2019
SELECT name, container, labels->'component' AS component FROM pods;
name | container | component
--------------------------------+--------------------------------------------+---------------------------
coredns-6f685fffbf-7xrtp | k8s.gcr.io/coredns:1.2.6 |
coredns-6f685fffbf-nzfn7 | k8s.gcr.io/coredns:1.2.6 |
etcd-master | k8s.gcr.io/etcd:3.2.24 | "etcd"
kube-apiserver-master | k8s.gcr.io/kube-apiserver:v1.12.1 | "kube-apiserver"
kube-controller-manager-master | k8s.gcr.io/kube-controller-manager:v1.12.1 | "kube-controller-manager"
kube-proxy-lk2mq | k8s.gcr.io/kube-proxy:v1.12.1 |
kube-scheduler-master | k8s.gcr.io/kube-scheduler:v1.12.1 | "kube-scheduler"
@stillinbeta // #postgresconf // NYC 2019
And for my
final trick:
@stillinbeta // #postgresconf // NYC 2019
Pod Pod Pod PodPod
@stillinbeta // #postgresconf // NYC 2019
Deployment
ReplicaSet ReplicaSet ReplicaSet
Pod Pod Pod Pod Pod Pod
@stillinbeta // #postgresconf // NYC 2019
coredns
coredns-6fec coredns-adc1 coredns-14ac
coredns
-6fec
-add7
coredns
-6fec
-7ae4
coredns-
adc1
-bbd5
coredns-
adc1
-1aed
coredns-
14ac
-86ef
coredns-
14ac
-d4ec
@stillinbeta // #postgresconf // NYC 2019
coredns
coredns-6fec coredns-adc1 coredns-14ac
coredns
-6fec
-add7
coredns
-6fec
-7ae4
coredns-
adc1
-bbd5
coredns-
adc1
-1aed
coredns-
14ac
-86ef
coredns-
14ac
-d4ec
deployments
replica_sets
pods
@stillinbeta // #postgresconf // NYC 2019
SELECT "deployments"."name" AS deployment_name
, "replica_sets"."name" as replica_name
, "pods"."name" AS pod_name
FROM deployments
JOIN replica_sets on "replica_sets"."name" LIKE "deployments"."name" || '-%'
JOIN pods on "pods"."name" LIKE "replica_sets"."name" || '-%';
deployment_name | replica_name | pod_name
-----------------+--------------------+--------------------------
coredns | coredns-6f685fffbf | coredns-6f685fffbf-7xrtp
coredns | coredns-6f685fffbf | coredns-6f685fffbf-nzfn7
@stillinbeta // #postgresconf // NYC 2019
● Column types mostly ignored
What doesn’t work?
@stillinbeta // #postgresconf // NYC 2019
● Column types mostly ignored
● If it’s a not a number, string, or map...
...just kind of give up
What doesn’t work?
@stillinbeta // #postgresconf // NYC 2019
● Column types mostly ignored
● If it’s a not a number, string, or map...
...just kind of give up
● The codebase not being a knot of
spaghetti
What doesn’t work?
@stillinbeta // #postgresconf // NYC 2019
● Column types mostly ignored
● If it’s a not a number, string, or map...
...just kind of give up
● The codebase not being a knot of
spaghetti
● Read-only
What doesn’t work?
@stillinbeta // #postgresconf // NYC 2019
Lessons from
Production
@stillinbeta // #postgresconf // NYC 2019
Lessons from
Production
The week I wrote this
@stillinbeta // #postgresconf // NYC 2019
Debugging is Hard!
● Lots of Segfaults
● Postgres codebase large and intimidating
● Go Problem? C Problem? Both?
@stillinbeta // #postgresconf // NYC 2019
PGXS?
● Makefile based
● C Only :(
@stillinbeta // #postgresconf // NYC 2019
“For more
complicated
packages, you might
need to write your
own build system.”
@stillinbeta // #postgresconf // NYC 2019
Bazel + Docker
● Bazel builds C + Go
● Docker container runs postgres
bazel run //:k8s_fdw_image
docker run 
-v /tmp/config:/kubeconfig 
--rm --name=k8s_fdw 
bazel:k8s_fdw_image
@stillinbeta // #postgresconf // NYC 2019
Questions?
Concerns?
Come find me! I’m the one with
pink hair!
Twitter: @stillinbeta
Github: github.com/liztio/k8s-fdw
K8s.slack.io: @liz
1 of 80

Recommended

Nodejs Native Add-Ons from zero to hero by
Nodejs Native Add-Ons from zero to heroNodejs Native Add-Ons from zero to hero
Nodejs Native Add-Ons from zero to heroNicola Del Gobbo
427 views39 slides
Server Development Workflow For PicCollage by
Server Development Workflow For PicCollageServer Development Workflow For PicCollage
Server Development Workflow For PicCollageLin Jen-Shin
783 views11 slides
FOSDEM 2020 Presentation : Precise, cross-project code navigation at GitHub s... by
FOSDEM 2020 Presentation : Precise, cross-project code navigation at GitHub s...FOSDEM 2020 Presentation : Precise, cross-project code navigation at GitHub s...
FOSDEM 2020 Presentation : Precise, cross-project code navigation at GitHub s...Fasten Project
15 views40 slides
Mete Atamel - Serverless with Knative - Codemotion Amsterdam 2019 by
Mete Atamel - Serverless with Knative - Codemotion Amsterdam 2019Mete Atamel - Serverless with Knative - Codemotion Amsterdam 2019
Mete Atamel - Serverless with Knative - Codemotion Amsterdam 2019Codemotion
142 views21 slides
2008-12-21 Rubinius by
2008-12-21 Rubinius2008-12-21 Rubinius
2008-12-21 RubiniusLin Jen-Shin
589 views135 slides
The Architecture of PicCollage Server by
The Architecture of PicCollage ServerThe Architecture of PicCollage Server
The Architecture of PicCollage ServerLin Jen-Shin
602 views126 slides

More Related Content

Similar to You got database in my cloud!

You got database in my cloud (short version) by
You got database  in my cloud (short version)You got database  in my cloud (short version)
You got database in my cloud (short version)Liz Frost
287 views33 slides
Kubernetes from a Box Mix by
Kubernetes from a Box MixKubernetes from a Box Mix
Kubernetes from a Box MixLiz Frost
304 views74 slides
Containerize vs Virtualize? NGDC 2009 by
Containerize vs Virtualize? NGDC 2009Containerize vs Virtualize? NGDC 2009
Containerize vs Virtualize? NGDC 2009Andy d
428 views43 slides
carrow - Go bindings to Apache Arrow via C++-API by
carrow - Go bindings to Apache Arrow via C++-APIcarrow - Go bindings to Apache Arrow via C++-API
carrow - Go bindings to Apache Arrow via C++-APIYoni Davidson
452 views41 slides
Functional IoT: Hardware and Platform by
Functional IoT: Hardware and PlatformFunctional IoT: Hardware and Platform
Functional IoT: Hardware and PlatformKiwamu Okabe
1.6K views16 slides
Istio Playground by
Istio PlaygroundIstio Playground
Istio PlaygroundQAware GmbH
528 views53 slides

Similar to You got database in my cloud!(20)

You got database in my cloud (short version) by Liz Frost
You got database  in my cloud (short version)You got database  in my cloud (short version)
You got database in my cloud (short version)
Liz Frost287 views
Kubernetes from a Box Mix by Liz Frost
Kubernetes from a Box MixKubernetes from a Box Mix
Kubernetes from a Box Mix
Liz Frost304 views
Containerize vs Virtualize? NGDC 2009 by Andy d
Containerize vs Virtualize? NGDC 2009Containerize vs Virtualize? NGDC 2009
Containerize vs Virtualize? NGDC 2009
Andy d428 views
carrow - Go bindings to Apache Arrow via C++-API by Yoni Davidson
carrow - Go bindings to Apache Arrow via C++-APIcarrow - Go bindings to Apache Arrow via C++-API
carrow - Go bindings to Apache Arrow via C++-API
Yoni Davidson452 views
Functional IoT: Hardware and Platform by Kiwamu Okabe
Functional IoT: Hardware and PlatformFunctional IoT: Hardware and Platform
Functional IoT: Hardware and Platform
Kiwamu Okabe1.6K views
Istio Playground by QAware GmbH
Istio PlaygroundIstio Playground
Istio Playground
QAware GmbH528 views
Serverless Security: A How-to Guide @ SnowFROC 2019 by James Wickett
Serverless Security: A How-to Guide @ SnowFROC 2019Serverless Security: A How-to Guide @ SnowFROC 2019
Serverless Security: A How-to Guide @ SnowFROC 2019
James Wickett719 views
Three Years of Lessons Running Potentially Malicious Code Inside Containers by Ben Hall
Three Years of Lessons Running Potentially Malicious Code Inside ContainersThree Years of Lessons Running Potentially Malicious Code Inside Containers
Three Years of Lessons Running Potentially Malicious Code Inside Containers
Ben Hall235 views
Measuring Software development with GrimoireLab by Valerio Cosentino
Measuring Software development with GrimoireLabMeasuring Software development with GrimoireLab
Measuring Software development with GrimoireLab
Valerio Cosentino112 views
How to Create the API Document from Real API and Localization by Pronovix
How to Create the API Document from Real API and Localization How to Create the API Document from Real API and Localization
How to Create the API Document from Real API and Localization
Pronovix156 views
Getting your Hooks into Cordova by Gavin Pickin
Getting your Hooks into CordovaGetting your Hooks into Cordova
Getting your Hooks into Cordova
Gavin Pickin300 views
Getting Your Hooks Into Cordova by devObjective
Getting Your Hooks Into CordovaGetting Your Hooks Into Cordova
Getting Your Hooks Into Cordova
devObjective285 views
Building Serverless applications with Python by Andrii Soldatenko
Building Serverless applications with PythonBuilding Serverless applications with Python
Building Serverless applications with Python
Andrii Soldatenko781 views
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC... by Jian-Hong Pan
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...
Jian-Hong Pan85 views
Optimizing Spring Boot apps for Docker by Graham Charters
Optimizing Spring Boot apps for DockerOptimizing Spring Boot apps for Docker
Optimizing Spring Boot apps for Docker
Graham Charters765 views
Révolution eBPF - un noyau dynamique by Raphaël PINSON
Révolution eBPF - un noyau dynamiqueRévolution eBPF - un noyau dynamique
Révolution eBPF - un noyau dynamique
Raphaël PINSON199 views

Recently uploaded

Copilot Prompting Toolkit_All Resources.pdf by
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdfRiccardo Zamana
6 views4 slides
SAP FOR TYRE INDUSTRY.pdf by
SAP FOR TYRE INDUSTRY.pdfSAP FOR TYRE INDUSTRY.pdf
SAP FOR TYRE INDUSTRY.pdfVirendra Rai, PMP
23 views3 slides
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptxanimuscrm
13 views19 slides
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...Deltares
9 views32 slides
What Can Employee Monitoring Software Do?​ by
What Can Employee Monitoring Software Do?​What Can Employee Monitoring Software Do?​
What Can Employee Monitoring Software Do?​wAnywhere
21 views11 slides
Software evolution understanding: Automatic extraction of software identifier... by
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...Ra'Fat Al-Msie'deen
7 views33 slides

Recently uploaded(20)

Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana6 views
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm13 views
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares9 views
What Can Employee Monitoring Software Do?​ by wAnywhere
What Can Employee Monitoring Software Do?​What Can Employee Monitoring Software Do?​
What Can Employee Monitoring Software Do?​
wAnywhere21 views
Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker by Deltares
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
Deltares9 views
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko... by Deltares
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
DSD-INT 2023 Simulation of Coastal Hydrodynamics and Water Quality in Hong Ko...
Deltares11 views
Software testing company in India.pptx by SakshiPatel82
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptx
SakshiPatel827 views
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... by Donato Onofri
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Donato Onofri711 views
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller38 views
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -... by Deltares
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
Deltares6 views
Advanced API Mocking Techniques by Dimpy Adhikary
Advanced API Mocking TechniquesAdvanced API Mocking Techniques
Advanced API Mocking Techniques
Dimpy Adhikary19 views
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ... by marksimpsongw
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
marksimpsongw76 views
Cycleops - Automate deployments on top of bare metal.pptx by Thanassis Parathyras
Cycleops - Automate deployments on top of bare metal.pptxCycleops - Automate deployments on top of bare metal.pptx
Cycleops - Automate deployments on top of bare metal.pptx
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t... by Deltares
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
Deltares9 views
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports by Ra'Fat Al-Msie'deen
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug ReportsBushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports

You got database in my cloud!

  • 1. @stillinbeta // #postgresconf // NYC 2019 You got Database in my Cloud! Kubernetes Foreign Data Wrapper for Postgres Liz Frost
  • 2. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 3. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 4. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 5. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 6. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 7. @stillinbeta // #postgresconf // NYC 2019 What is Kubernetes? ● Container Manager ● API- and Object-driven ● Highly Extensible ● Monumentally popular
  • 8. @stillinbeta // #postgresconf // NYC 2019 Who am I?
  • 10. @stillinbeta // #postgresconf // NYC 2019 ...get it? zéro un zéro cero uno cero 零一零
  • 11. @stillinbeta // #postgresconf // NYC 2019 What is a Foreign Data Wrapper (FDW)? ● ODBC ● SQLite ● Cassandra ● Redis ● Hue? ● FDWs? ● Kubernetes!
  • 14. @stillinbeta // #postgresconf // NYC 2019 void BeginForeignScan (ForeignScanState *node, int eflags); TupleTableSlot * IterateForeignScan (ForeignScanState *node); ForeignScan * GetForeignPlan (PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid, ForeignPath *best_path, List *tlist, List *scan_clauses, Plan *outer_plan); That doesn’t look like Go...
  • 15. @stillinbeta // #postgresconf // NYC 2019 How about Multicorn?
  • 19. @stillinbeta // #postgresconf // NYC 2019 Why not Multicorn?
  • 20. @stillinbeta // #postgresconf // NYC 2019 Why not Multicorn?
  • 21. @stillinbeta // #postgresconf // NYC 2019 Native Extension?
  • 22. @stillinbeta // #postgresconf // NYC 2019 Nope. No C.
  • 23. @stillinbeta // #postgresconf // NYC 2019 Guess I’ll go
  • 24. @stillinbeta // #postgresconf // NYC 2019 Success! Someone did the hard part for us!
  • 25. @stillinbeta // #postgresconf // NYC 2019 github.com/dennwc/go-fdw
  • 28. @stillinbeta // #postgresconf // NYC 2019 C <=> Go?
  • 29. @stillinbeta // #postgresconf // NYC 2019 C <=> Go? CGo!
  • 30. @stillinbeta // #postgresconf // NYC 2019 // #include <stdio.h> // #include <errno.h> import "C" How does cgo work?
  • 31. @stillinbeta // #postgresconf // NYC 2019 //#cgo CFLAGS: -I/usr/include/postgresql/9.6/server -I/usr/include/postgresql/internal //#cgo LDFLAGS: -Wl,-unresolved-symbols=ignore-all //#include "postgres.h" //#include "funcapi.h" <snip> Import “C” How does cgo work?
  • 32. @stillinbeta // #postgresconf // NYC 2019 // Property adds a key-value property to results of EXPLAIN query. func (e Explainer) Property(k, v string) { C.ExplainPropertyText(C.CString(k), C.CString(v), e.es) } So how do we call?
  • 33. @stillinbeta // #postgresconf // NYC 2019 //static Datum cStringGetDatum(const char *str) { // PG_RETURN_TEXT_P(CStringGetTextDatum(str)); //} How do we use macros?
  • 34. @stillinbeta // #postgresconf // NYC 2019 type Table interface { Stats(opts *Options) TableStats Scan(rel *Relation, opts *Options) Iterator } type Iterator interface { Next() []interface{} Reset() Close() error } What’s our interface look like?
  • 35. @stillinbeta // #postgresconf // NYC 2019 type Table interface { Stats(opts *Options) (TableStats, error) Scan(rel *Relation, opts *Options) (Iterator, error) } type Iterator interface { Next() ([]interface{}, error) Reset() Close() error } Wouldn’t be Go without errors!
  • 36. @stillinbeta // #postgresconf // NYC 2019 CREATE SERVER IF NOT EXISTS kind FOREIGN DATA WRAPPER k8s_fdw OPTIONS (kubeconfig '/kubeconfig'); What kind of interface do we want?
  • 37. @stillinbeta // #postgresconf // NYC 2019 What’s a Kubeconfig?
  • 38. @stillinbeta // #postgresconf // NYC 2019 apiVersion: v1 clusters: - cluster: server: http://localhost:8080 name: local-server contexts: - context: cluster: local-server namespace: the-right-prefix user: myself name: default-context current-context: default-context kind: Config preferences: {} users: - name: myself user: password: secret username: admin Connection Credentials
  • 39. @stillinbeta // #postgresconf // NYC 2019 What does a Kubernetes object look like?
  • 40. @stillinbeta // #postgresconf // NYC 2019 apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp-container image: busybox command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'] - name: myapp-sidecar image: stillinbeta/logging status: conditions: - type: Ready status: "True" lastProbeTime: null lastTransitionTime: 2018-01-01T00:00:00Z Many fields, variable lengths, hierarchical...
  • 41. @stillinbeta // #postgresconf // NYC 2019 11 fields apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp-container image: busybox command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'] - name: myapp-sidecar image: stillinbeta/logging status: conditions: - type: Ready status: "True" lastProbeTime: null lastTransitionTime: 2018-01-01T00:00:00Z
  • 42. @stillinbeta // #postgresconf // NYC 2019 11 fields ...and this is a small object apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp-container image: busybox command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'] - name: myapp-sidecar image: stillinbeta/logging status: conditions: - type: Ready status: "True" lastProbeTime: null lastTransitionTime: 2018-01-01T00:00:00Z
  • 43. @stillinbeta // #postgresconf // NYC 2019 apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/config.hash: 4b52d75cab61380f07c0c5a69fb371d4 kubernetes.io/config.mirror: 4b52d75cab61380f07c0c5a69fb371d4 kubernetes.io/config.seen: "2019-03-19T16:11:31.520506966Z" kubernetes.io/config.source: file scheduler.alpha.kubernetes.io/critical-pod: "" creationTimestamp: "2019-03-19T16:12:57Z" labels: component: kube-scheduler tier: control-plane name: kube-scheduler-kate namespace: kube-system resourceVersion: "446" selfLink: /api/v1/namespaces/kube-system/pods/kube-scheduler-kate uid: dc176fb5-4a61-11e9-8e22-080027245760 spec: containers: - command: - kube-scheduler - --address=127.0.0.1 - --kubeconfig=/etc/kubernetes/scheduler.conf - --leader-elect=true image: k8s.gcr.io/kube-scheduler:v1.13.4 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10251 scheme: HTTP initialDelaySeconds: 15 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 15 name: kube-scheduler resources: requests: cpu: 100m terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/kubernetes/scheduler.conf name: kubeconfig readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true nodeName: kate priority: 2000000000 priorityClassName: system-cluster-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute operator: Exists volumes: - hostPath: path: /etc/kubernetes/scheduler.conf type: FileOrCreate name: kubeconfig status: conditions: - lastProbeTime: null lastTransitionTime: "2019-03-19T16:11:32Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2019-03-19T16:11:36Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2019-03-19T16:11:36Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2019-03-19T16:11:32Z" status: "True" type: PodScheduled containerStatuses: - containerID: docker://aee2c6b8ed3a23a636f235183fbadaa6e827660ab49a7d8742a73f5143d376fd image: k8s.gcr.io/kube-scheduler:v1.13.4 imageID: docker-pullable://k8s.gcr.io/kube-scheduler@sha256:e7b2f9b1dcfa03b0e43b891979075d62086fe14e169de081f9c23db378f5b2f7 lastState: {} name: kube-scheduler ready: true restartCount: 0 state: running: startedAt: "2019-03-19T16:11:34Z" hostIP: 10.0.2.15 phase: Running podIP: 10.0.2.15 qosClass: Burstable startTime: "2019-03-19T16:11:32Z" Here’s a real-world example
  • 44. @stillinbeta // #postgresconf // NYC 2019 We probably don’t care about every field!
  • 45. @stillinbeta // #postgresconf // NYC 2019 CREATE SERVER IF NOT EXISTS kind FOREIGN DATA WRAPPER k8s_fdw OPTIONS (kubeconfig '/kubeconfig'); CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); What kind of interface do we want?
  • 46. @stillinbeta // #postgresconf // NYC 2019 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE Let’s try it out!
  • 47. @stillinbeta // #postgresconf // NYC 2019 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE ERROR: couldn't get kubeconfig: couldn't get resource mapper: could not get api group resources: Get https://ec2-54-205-250-176.compute-1.amazonaws.com:6443/api?ti meout=32s: net/http: invalid header field value "postgres: postgres postgres [local] SELECTx00x00x00x00x00x00x00x00x00x00/v0.0.0 (linux/amd64) kubernetes/$Format" for key User-Agent oh.
  • 48. @stillinbeta // #postgresconf // NYC 2019 $ psql --user postgres < test.sql CREATE EXTENSION CREATE SERVER CREATE FOREIGN TABLE name | namespace --------------------------------+------------- coredns-6f685fffbf-7xrtp | kube-system coredns-6f685fffbf-nzfn7 | kube-system etcd-master | kube-system kube-apiserver-master | kube-system kube-controller-manager-master | kube-system kube-proxy-lk2mq | kube-system kube-scheduler-master | kube-system That’s better!
  • 49. @stillinbeta // #postgresconf // NYC 2019 That’s boring. What else can you do?
  • 50. @stillinbeta // #postgresconf // NYC 2019 How about JSONPATH?
  • 51. @stillinbeta // #postgresconf // NYC 2019 ...what’s JSONPATH?
  • 52. @stillinbeta // #postgresconf // NYC 2019 text the plain text kind is {.kind} kind is List @ the current object {@} the same as input . or [] child operator {.kind} or {[‘kind’]} List .. recursive descent {..name} 127.0.0.1 127.0.0.2 myself e2e * wildcard. Get all objects {.items[*].metadata.name} [127.0.0.1 127.0.0.2] [start:end :step] subscript operator {.users[0].name} myself [,] union operator {.items[*][‘metadata.name’, ‘status.capacity’]} 127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8] ?() filter {.users[?(@.name==“e2e”)].user. password} secret range, end iterate list {range .items[*]}[{.metadata.name}, {.status.capacity}] {end} [127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]] “ quote interpreted string {range .items[*]}{.metadata.name}{’t’}{ end} 127.0.0.1 127.0.0.2
  • 53. @stillinbeta // #postgresconf // NYC 2019 text the plain text {.kind} . or [] child operator {.kind} or {[‘kind’]} [start:end :step] subscript operator {.users[0].name}
  • 54. @stillinbeta // #postgresconf // NYC 2019 Why JSONPath?
  • 55. @stillinbeta // #postgresconf // NYC 2019 Why JSONPath? $ kubectl get pods -o=jsonpath='{@}' $ kubectl get pods -o=jsonpath='{.items[0]}' $ kubectl get pods -o=jsonpath='{.items[0].metadata.name}' $ kubectl get pods -o=jsonpath='{range.items[*]}{.metadata.name}{"t"}{.status.startTime}{"n"}{end}'
  • 56. @stillinbeta // #postgresconf // NYC 2019 CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') , container text OPTIONS (alias '{.spec.containers[0].image}') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); Let’s do that!
  • 57. @stillinbeta // #postgresconf // NYC 2019 SELECT * from PODS; name | namespace | container --------------------------------+-------------+-------------------------------------------- coredns-6f685fffbf-7xrtp | kube-system | k8s.gcr.io/coredns:1.2.6 coredns-6f685fffbf-nzfn7 | kube-system | k8s.gcr.io/coredns:1.2.6 etcd-master | kube-system | k8s.gcr.io/etcd:3.2.24 kube-apiserver-master | kube-system | k8s.gcr.io/kube-apiserver:v1.12.1 kube-controller-manager-master | kube-system | k8s.gcr.io/kube-controller-manager:v1.12.1 kube-proxy-lk2mq | kube-system | k8s.gcr.io/kube-proxy:v1.12.1 kube-scheduler-master | kube-system | k8s.gcr.io/kube-scheduler:v1.12.1
  • 58. @stillinbeta // #postgresconf // NYC 2019 That’s fine and dandy, but not everything’s a string!
  • 59. @stillinbeta // #postgresconf // NYC 2019 Labels object Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services.
  • 60. @stillinbeta // #postgresconf // NYC 2019 How about a map?
  • 61. @stillinbeta // #postgresconf // NYC 2019 CREATE FOREIGN TABLE IF NOT EXISTS pods ( name text OPTIONS (alias 'metadata.name') , namespace text OPTIONS (alias 'metadata.namespace') , container text OPTIONS (alias '{.spec.containers[0].image}') , labels jsonb OPTIONS (alias 'metadata.labels') ) SERVER kind OPTIONS ( namespace 'kube-system' , apiVersion 'v1' , kind 'Pod' ); What kid of interface do we want?
  • 62. @stillinbeta // #postgresconf // NYC 2019 SELECT name, labels FROM pods; name | labels --------------------------------+----------------------------------------------------------------------------------------------------- coredns-6f685fffbf-7xrtp | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"} coredns-6f685fffbf-nzfn7 | {"k8s-app": "kube-dns", "pod-template-hash": "6f685fffbf"} etcd-master | {"tier": "control-plane", "component": "etcd"} kube-apiserver-master | {"tier": "control-plane", "component": "kube-apiserver"} kube-controller-manager-master | {"tier": "control-plane", "component": "kube-controller-manager"} kube-proxy-lk2mq | {"k8s-app": "kube-proxy", "pod-template-generation": "1", "controller-revision-hash": "6cbfff58bb"} kube-scheduler-master | {"tier": "control-plane", "component": "kube-scheduler"} (7 rows)
  • 63. @stillinbeta // #postgresconf // NYC 2019 SELECT name, container, labels->'component' AS component FROM pods; name | container | component --------------------------------+--------------------------------------------+--------------------------- coredns-6f685fffbf-7xrtp | k8s.gcr.io/coredns:1.2.6 | coredns-6f685fffbf-nzfn7 | k8s.gcr.io/coredns:1.2.6 | etcd-master | k8s.gcr.io/etcd:3.2.24 | "etcd" kube-apiserver-master | k8s.gcr.io/kube-apiserver:v1.12.1 | "kube-apiserver" kube-controller-manager-master | k8s.gcr.io/kube-controller-manager:v1.12.1 | "kube-controller-manager" kube-proxy-lk2mq | k8s.gcr.io/kube-proxy:v1.12.1 | kube-scheduler-master | k8s.gcr.io/kube-scheduler:v1.12.1 | "kube-scheduler"
  • 64. @stillinbeta // #postgresconf // NYC 2019 And for my final trick:
  • 65. @stillinbeta // #postgresconf // NYC 2019 Pod Pod Pod PodPod
  • 66. @stillinbeta // #postgresconf // NYC 2019 Deployment ReplicaSet ReplicaSet ReplicaSet Pod Pod Pod Pod Pod Pod
  • 67. @stillinbeta // #postgresconf // NYC 2019 coredns coredns-6fec coredns-adc1 coredns-14ac coredns -6fec -add7 coredns -6fec -7ae4 coredns- adc1 -bbd5 coredns- adc1 -1aed coredns- 14ac -86ef coredns- 14ac -d4ec
  • 68. @stillinbeta // #postgresconf // NYC 2019 coredns coredns-6fec coredns-adc1 coredns-14ac coredns -6fec -add7 coredns -6fec -7ae4 coredns- adc1 -bbd5 coredns- adc1 -1aed coredns- 14ac -86ef coredns- 14ac -d4ec deployments replica_sets pods
  • 69. @stillinbeta // #postgresconf // NYC 2019 SELECT "deployments"."name" AS deployment_name , "replica_sets"."name" as replica_name , "pods"."name" AS pod_name FROM deployments JOIN replica_sets on "replica_sets"."name" LIKE "deployments"."name" || '-%' JOIN pods on "pods"."name" LIKE "replica_sets"."name" || '-%'; deployment_name | replica_name | pod_name -----------------+--------------------+-------------------------- coredns | coredns-6f685fffbf | coredns-6f685fffbf-7xrtp coredns | coredns-6f685fffbf | coredns-6f685fffbf-nzfn7
  • 70. @stillinbeta // #postgresconf // NYC 2019 ● Column types mostly ignored What doesn’t work?
  • 71. @stillinbeta // #postgresconf // NYC 2019 ● Column types mostly ignored ● If it’s a not a number, string, or map... ...just kind of give up What doesn’t work?
  • 72. @stillinbeta // #postgresconf // NYC 2019 ● Column types mostly ignored ● If it’s a not a number, string, or map... ...just kind of give up ● The codebase not being a knot of spaghetti What doesn’t work?
  • 73. @stillinbeta // #postgresconf // NYC 2019 ● Column types mostly ignored ● If it’s a not a number, string, or map... ...just kind of give up ● The codebase not being a knot of spaghetti ● Read-only What doesn’t work?
  • 74. @stillinbeta // #postgresconf // NYC 2019 Lessons from Production
  • 75. @stillinbeta // #postgresconf // NYC 2019 Lessons from Production The week I wrote this
  • 76. @stillinbeta // #postgresconf // NYC 2019 Debugging is Hard! ● Lots of Segfaults ● Postgres codebase large and intimidating ● Go Problem? C Problem? Both?
  • 77. @stillinbeta // #postgresconf // NYC 2019 PGXS? ● Makefile based ● C Only :(
  • 78. @stillinbeta // #postgresconf // NYC 2019 “For more complicated packages, you might need to write your own build system.”
  • 79. @stillinbeta // #postgresconf // NYC 2019 Bazel + Docker ● Bazel builds C + Go ● Docker container runs postgres bazel run //:k8s_fdw_image docker run -v /tmp/config:/kubeconfig --rm --name=k8s_fdw bazel:k8s_fdw_image
  • 80. @stillinbeta // #postgresconf // NYC 2019 Questions? Concerns? Come find me! I’m the one with pink hair! Twitter: @stillinbeta Github: github.com/liztio/k8s-fdw K8s.slack.io: @liz