Layer 7 Observability and Centralized Configuration with Consul Service Mesh

Copyright © 2019 HashiCorp
Layer 7 Observability
with Consul Service
Mesh

Layer 7 Observability with Consul Service Mesh.
Erik Veld
Developer Advocate at HashiCorp
@erikveld
eveld@hashicorp.com

Networking
Connect infrastructure
and applications.
Development
Run applications.
Security
Secure infrastructure and
applications.
Operations
Provision infrastructure.
INTRODUCTION © 2019 HashiCorp 4
The 4 essential
elements of
dynamic
infrastructure

Multi-Cloud
Service
Networking
INTRODUCTION © 2019 HashiCorp 5
Networking
Connect infrastructure
and applications.
Development
Run applications.
Security
Secure infrastructure and
applications.
Operations
Provision infrastructure.

Consul
Use Cases
Distributed service
networking layer to
connect and secure
any services
6© 2019 HashiCorpINTRODUCTION
Service
Registry &
Discovery
Dynamically locate any
application or
infrastructure service to
simplify network
connectivity
Service Mesh
A distributed
networking layer to
connect, secure and
observe services across
any runtime platform
and cloud

Market
trend from
monoliths to
microservices Single,
Physical
Server
Dynamic Virtual
Machines
Smaller,
Ephemeral
Containers

INTRODUCTION 8© 2019 HashiCorp
The shift
from static
to dynamic
networking
Dynamic Infrastructure
Service-based networking
Static Infrastructure
Host-based networking

The shift
from static
to dynamic
networking
Dynamic Infrastructure
Service-based networking
Static Infrastructure
Host-based networking
Private datacenters with static IPs,
primarily north-south traffic, protected by
perimeter security and coarse-grained
network segments.
TRADITIONAL APPROACH
● Static connectivity between services
● A fleet of load balancers to route traffic
● Ticket driven processes to update
network middleware
● Firewall rule sprawl to constrict access
and Insecure flat network zones
Multiple clouds and private datacenters
with dynamic IPs, dominated by east-west
traffic, no clear network perimeters.
CONSUL APPROACH
● Centralized registry to locate any
service
● Services discovered and connected
with centralized policies
● Network automated in service of
applications
● Zero trust network enforced by
identity-based security policies

Reduced Productivity
Waiting for manual updates to load balancers and
ﬁrewalls blocks development throughput.
Business
challenges
of dynamic
infrastructure
Increased Risk
Firewall rule sprawl is complex to manage and
mistakes create security vulnerabilities.
Increased Cost
Load balancers and ﬁrewalls are expensive and costly
to maintain.

Networks are not 100% stable and
often experience transient failure.

Gray Failure
The Achilles' Heel of Cloud-Scale Systems
"The major availability breakdowns and performance anomalies we see in
cloud environments tend to be caused by subtle underlying faults, i.e.
gray failures rather than fail-stop failure"
Paper from Microsoft research

Gray Failure
The Achilles' Heel of Cloud-Scale Systems
● Performance degradation
● Random packet loss
● Flaky I/O
● Memory pressure
● Non-fatal exceptions

Common reliability patterns
● Service discovery
● Load balancing
● Retries
● Circuit breaking
● Rate limiting

Components
Service Mesh
Control Plane
▪ Service to service communication
policy.
▪ Service Catalog
▪ CA and x509 certiﬁcate generation.
▪ Conﬁguration and proxy management.
16© 2019 HashiCorpSERVICE MESH

Components
Service Mesh
Data Plane
▪ Authorization
▪ Request tracing
▪ Traﬃc shaping
▪ Load balancing
▪ Service discovery
▪ Circuit breaking
▪ Retry logic

Configure Proxies
SERVICE MESH 18© 2019 HashiCorp
● A proxy is co-located with a single service
instance which it represents and proxy all
inbound traffic to
● The Client agent instantiates the proxy
and registers it as a service.
● Configure proxy with a port on which is
used for the service and upstream
destination that the service wants to
connect to zz
connect = {
proxy = {
config = {
upstreams = [
{
destination_name = "mongodb",
local_bind_port = 8001
}
]
}
}
}

Get certificate for application
● Upon its instantiation, the proxy requests
from local agent both leaf and root
certificates for the service instance it
represents.
● If the local agent can not find the
certificates of this service instance in its
cache, it will generate a new unique
private key and send a certificate signing
request (CSR) up to Consul server.
● Server returns the signed certificate to
local agent. Agent then caches the
certificates the return to the proxy.

Retrieve Intentions
● A subset of intentions which are relevant
to destination services are locally cached
at the destination Consul agent
● Connection attempts require only local
agent communication for authorization
● Updates to intentions are propagated
instantly to agents
zz
$ consul intention create -allow web db
Created: web => db (allow)

Establish the connection 1/2
● The proxy of the web service uses Consul
service discovery APIs to request the
location of the DB.
● The local agent returns the proxy’s IP
address/Port of a healthy DB instance.
● The local agent also returns the URI for
the expected identity of the service it is
connected to
● Proxies between web and database start
TLS handshake to authenticate the
identity

Establish the connection 2/2
● The DB proxy sends the authorization
request to its local agent
● The local agent authorizes the connection
based on locally cached intention
● Mutual TLS is established

You can’t do Reliability without
Observability.

Observe
Check Implement

Observability is a measure of how well
internal states of a system can be
inferred from knowledge of
external outputs.

▪ System metrics (e.g. network, node, etc.)
▪ Health checks
▪ Tracing
▪ Access logs
▪ Application metrics
▪ Application logs
▪ Business analytics
Observability
27© 2019 HashiCorpOBSERVABILITY

▪ Full system failure (easy to spot)
▪ Global Service failure
▪ Service instance failure
▪ Node failure
Types of
failures
In order to pinpoint
the cause of a failure,
we need metadata to
locate the source.

▪ Connections vs Requests
▪ Failed requests don't always cause a
disconnect
▪ Descriptive status codes
▪ gRPC method names
Layer 7
In order to pinpoint
the cause of a failure,
we need metadata to
locate the source.

▪ Originally created by Etsy
▪ Push based metrics
▪ Lightweight UDP protocol
▪ No support for metadata
StatsD
CODE EDITOR
myservice.service1.mymethod.called

▪ Created by DataDog based on StatsD protocol
▪ Push based metrics
▪ Lightweight UDP protocol
▪ Support for metadata through tags
DogStatsD
CODE EDITOR
myservice.mymethod.called tags[serviceid:service1]

▪ Pull based approach from central server
▪ Service implements HTTP endpoint exposing
metrics
▪ Supports metadata by default
Prometheus
CODE EDITOR
envoy_http_downstream_rq_completed{envoy_http_conn_manager_preﬁx="ingress_cache"}

Architecture
Conﬁguration
34© 2019 HashiCorpCONFIGURATION
Service A
Sidecar proxy
StatsD exporter Prometheus
Service B
Sidecar proxy
Pod A Pod B
Grafana
emits metrics to StatsD
scrapes the StatsD
exporter
pulls metrics from
Prometheus
Consul
injects and
conﬁgures the
sidecar proxy

CODE EDITOR
- job_name: "pods"
scrape_interval: 1s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_pod_container_name]
regex: "prometheus-statsd"
Scrape conﬁguration
Scrape conﬁguration for kubernetes pods.

Scrape conﬁguration
Scrape conﬁguration for VMs using Consul Service Discovery.
CODE EDITOR
- job_name: "pods"
scrape_interval: 1s
kconsul_sd_configs:
- server: "localhost:8500"
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: job

CODE EDITOR
kind = "proxy-defaults"
name = "global"
config {
envoy_dogstatsd_url = "udp://127.0.0.1:9125"
# envoy_prometheus_bind_addr = "0.0.0.0:9102"
}
Configuration file
Enable and configure the metrics endpoint.

Write the configuration
Write the configuration to the centralized configuration.
TERMINAL
$ consul config write proxy-defaults.hcl

CODE EDITOR
metadata:
labels:
app: emojify-api
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service-upstreams": "emojify-facebox:8003,emojify-cache:8005"
Conﬁgure the service
Injecting the sidecar proxy using consul-k8s.

Bootstrap
Applying the kubernetes conﬁguration will automatically bootstrap the sidecar proxy.
TERMINAL
$ kubectl apply -f emojify-api.yml

CODE EDITOR
services {
name = "emojify-api"
port = 9090
connect {
sidecar_service {
port = 20000
proxy {
local_service_address = "127.0.0.1"
local_service_port = 9090
upstreams {
destination_name = "emojify-facebox"
local_bind_port = 8003
}
upstreams {

Bootstrap
Conﬁguring the Envoy sidecar proxy for the service.
TERMINAL
$ consul connect envoy -sidecar-for emojify-api

Thank You
eveld@hashicorp.com
www.hashicorp.com
44

HashiCorp
Presentation Template
IMPORTANT:
Please read through the
“How to use this template” slides

How to use this template………………………………………………………………..………………………..……………………………….…………
Splash slides……………………………………………………………………………………………………………………………………………….…………
Presenter slides…………………………………………………………………………………………………………….………………………………………
Cover & section slides…………………………………………..………………………………………..……………...……………………………………
Text only slides……………………………………………………..……………………………..……………..…….….……………………………………..
Image & diagram slides…..……………………………………………………..……………..……………..……….……………………………………
Screenshot slides…………………………………………………………………..……………..……………..……….……………………………………..
Code block slides…………………………………………………………………..……………..…………..……….….……………………………………..
Sequence slides………………………………………………………………..…………….………………………..…..……………………………………..
Callout slides………….…………………………………………………………..…………….…………….…..……………………………………………..
Collection slides…………………………………..……………………………..…………….………….……………..…………………………………….
Security indicators and footer content……………………….……………………………………………………..……………………………
Assets…………………………………..………….………………………………..………………………….………….…………………………………………
Table of Contents
3
11
18
23
34
52
70
75
88
101
126
155
160
Start Here

Create a new document
“From a template.”
From Google Drive, click the “New” button or
right click in the blank space in any folder.
Choose “Google Slides” > “From a template”
and select the Default template.
The template will be duplicated into your
current folder. The ﬁrst thing you’ll want to do
is rename your new ﬁle to avoid confusion!

How to use sample slides.
Take some time browsing through each section to review
the slide template options. It may be easier to use the
Grid view (found below the your slide navigator).
Copy/Paste or Duplicate sample slides, then modify as
needed to build your presentation.

The grid.
In order to maintain visual consistency across
slides it’s imperative you keep content and
objects aligned to the HashiCorp
Presentation grid.
To turn on guides click view > Guides >
“Show Guides”

Select the image you want to replace, then choose
“Replace Image…” from the Google Slides top menu.
How to edit / replace
placeholder images.
To reposition and resize the placed image, double-click it.
This should bring you into adjustment mode where you
can move and resize both the cropping frame and the
image itself.

Avoid when presenting
to an audience
Content best practices.
Dense information is diﬃcult to digest. An example of a
dense slide may include a slide title, overview paragraph text,
bullet point, and an image.

Content best practices.
For higher impact, consider distributing key content across
multiple slides. In this example, some content was removed to avoid
visual clutter, and is instead delivered as part of the talk track.
“The large community supporting Terraform Open Source ranges from independent developers to teams within large
technology companies. Community activity provides constant improvements to our platform and to provider plugins.”

Security indicators.
Be sure to use the correct security indicator
for your content on a per slide basis. If you
have any questions as to which category your
content falls within, please reference:
https://policypoint.hashicorp.services/
standards/risk-classiﬁcation-standard
To learn how security indicators would be
displayed. review the footer content
template.
Public Content (Low Risk)
No indicator needed.
Restricted Content (Medium Risk)
For customers under NDA.
Internal Content (High Risk)
For internal use only.
INTERNAL USE ONLY
RESTRICTED USE ONLY

Help us improve this template.
Any and all feedback is welcome.
Get in touch in #proj-slide-template on Slack.
You can also check out additional documentation on Conﬂuence:
https://hashicorp.atlassian.net/wiki/spaces/MKT/pages/357531677/Presentations+Workﬂow

Thank You
hello@hashicorp.com
www.hashicorp.com
62

Jonathan Appleseed
Placeholder at HashiCorp

Presentation title goes here
Jonathan Appleseed

Presentation title goes here
Jonathan Appleseed
Jonathan Appleseed

Jonathan
Appleseed
Placeholder at Client
Jonathan
Appleseed
Placeholder at Client
Jonathan
Appleseed
Jonathan
Appleseed
PREPARED FOR PREPARED BY

General
presentation title.
3 line max.
Secondary headline here /
keep this short and concise.
3 Line max.

Slide title / 2 line max.
Secondary headline / 1 line max. Delete if slide title is 2 lines.
An organization is driven by its people and culture. What follows are the
principles of HashiCorp, which together describe the foundational
characteristics of the company.

Section title.
Every action we take moves us in some direction. Vision is a point much farther
than a single action can take us. Having a vision allows us to judge if an action
moves us closer or further from our vision.
Section title.
By having a vision, we try to move in some direction, rather than moving in no
singular direction at all. Vision requires you to reﬂect on the big picture; to
understand the greater goal behind the smaller actions.

Section title.
Text box placement is dependent on the
amount of content. If your content is 8 lines
or less, please move text box down one
guide. Multiple text boxes on each slide
should align with each other, based on the
longest column.
Section title.
Text box placement is dependent on the
amount of content. If your content is 9 lines
or more, your text box should align with this
guide.
HashiCorp will always be focused on
innovating and pushing the boundaries in an
attempt to deeply impact the status quo.
However, forward progress requires strong
grounding in reality.

Section title.
Long after we forget the
details of an interaction, we
remember how we felt. This
extends to our impressions of
people, websites, tools, and
products.
Section title.
HashiCorp will always be
focused on innovating and
pushing the boundaries in an
attempt to deeply impact the
status quo. However, forward
progress requires strong
Section title.
Beauty can exist in any job well
done. A job well done requires
applying a sense of purpose
and thoughtfulness, a
consideration for the
consumer of our work.

▪ Bullet One
▪ Bullet Two
▪ Bullet Three
▪ Bullet Four
▪ Bullet Five
▪ Bullet Six

1. Bullet One
2. Bullet Two
3. Bullet Three
4. Bullet Four
5. Bullet Five
6. Bullet Six

Slide title
/ 4 line max.
Secondary headline
/ 4 line max.
Section title. Content aligned to slide title.
An organization is driven by its people and culture. What
follows are the principles of HashiCorp, which together
describe the foundational characteristics of the company.
▪ Body Level One
– Body Level Two
– Body Level Two
▪ Body Level One
– Body Level Two

Section title. Content centered vertically.
Use this layout only when the length of your content exceeds
the space allowed for the “TL - Content Aligned to Slide Title”
Template.
Section two title.
Beauty can exist in any job well done. A job well done requires
applying a sense of purpose and thoughtfulness, a
consideration for the consumer of our work. In this way, we
must treat our work as a craft to be practiced and perfected.
This attention to detail should be applied to everything we
produce internally and externally.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Section title.
Section title.
products.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

2,712+
Paragraph title, 1 line max.
Body copy, 1 line max.
2,712+
2,712+
2,712+
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

▪ Bullet One
▪ Bullet Two
▪ Bullet Three
▪ Bullet Four
▪ Bullet Five
▪ Bullet Six
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

1. Bullet One
2. Bullet Two
3. Bullet Three
4. Bullet Four
5. Bullet Five
6. Bullet Six
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Section title.
Every action we take moves us in some direction.
Section title.
Vision is a point much farther than a single action
can take us.
Section title.
Having a vision allows us to judge if an action moves
us closer or further from our vision.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

1. Bullet One
2. Bullet Two
3. Bullet Three
4. Bullet Four
5. Bullet Five
6. Bullet Six

1. Bullet One
2. Bullet Two
3. Bullet Three
4. Bullet Four
5. Bullet Five

Section title.
Every action we take moves us in some
direction. Vision is a point much farther than
a single action can take us. Having a vision
allows us to judge if an action moves us
closer or further from our vision.

Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Section title.
extends to our impressions
of people, websites, tools,
and products.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

and products.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Section title.
One or two line description
of above title here.
Section title.
One or two line description
of above title here.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Section title.
status quo. However,
forward progress requires
strong grounding in reality.
Section title.
and products.
Slide title
/ 4 line max.
Secondary headline
/ 4 line max.

Clarence Howard
CEO, Founder
Essie Fitzpatrick
CEO, Founder
Cleve Balistreri
CEO, Founder
Amina Lendgren
CEO, Founder

Callout or descriptive copy / 1 line max.

Callout or
descriptive copy
/ 4 line max.

Slide title
/ 4 line max.
Body / 4 line max.

CODE EDITOR
{
“foo”: “bar”,
“baz”: 42
}

Slide title
/ 4 line max.
Body / 4 line max.
CODE EDITOR
{
“baz”: 42
}

Slide title
/ 4 line
max.
Body / 4 line max.
CODE EDITOR
{
“baz”: 42
}

CODE EDITOR
{
“baz”: 42
}

TERMINAL
> lorem ipsum -d -s amet

TERMINAL
Slide title
/ 4 line max.
Body / 4 line max.

Slide title
/ 4 line max.
Body / 4 line max.
TERMINAL

TERMINAL

!
Slide title
/ 4 line max.
Body / 4 line max.

1 2 3
Slide title
/ 4 line max.
Body / 4 line max.

Slide title
/ 4 line max.
Body / 4 line max.
1 2 3

Body copy / 4 line max.

Callout or quote text /
4 line max.
Body copy / 4 line max.

Callout Text
Beauty works best.

Collection item
products.
Collection item
Collection item
Beauty can exist in any job well
done. A job well done requires
applying a sense of purpose
and thoughtfulness, a
consideration for the

Collection item
and products.
Collection item
status quo. However,
forward progress requires
strong grounding in reality.
Collection item
Beauty can exist in any job
well done. A job well done
requires applying a sense of
purpose and thoughtfulness,
a consideration for the

Collection item
details of an interaction,
we remember how we
felt. This extends to our
impressions of people,
websites, tools, and
products.
Collection item
we remember how we
products.
Collection item
we remember how we
products.

Collection item
Body / 1 line max.
Collection item
Body / 1 line max.
Collection item
Body / 1 line max.

Collection item
Collection item
Collection item
Collection item
Collection item

Collection item
Collection item
Collection item

Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.

Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.
Collection item
Body / 3 line max.

Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.

Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.
Collection item
Body /
2 line max.

Collection item / 1 line max.
Large Body / 2 line max.

Body Copy
Body Copy

Collection item
Body / 2 line max.
Collection item
Body / 2 line max.
Collection item
Body / 2 line max.

Collection item
This extends to our
products.
Collection item
remember how we felt.
Collection item
remember how we felt.

Client Logo Slide
TITLETITLE TITLE TITLE TITLE
TITLE

Security Indicator
and Footer Content

Footer Content
Light background:
If you need a running footer, security indicator, copyright info,
or page number please copy the footer content from this slide.
⁄ 201Copyright © 2019 HashiCorp⁄ SECTION OR PRESENTATION TITLE RESTRICTED USE ONLY

Key Messaging
Cloud Infrastructure
Automation
Consistent workﬂows to provision, secure,
connect, and run any infrastructure for
any application.

Text styles
Display
Slide Title
Section Title
BodyParagraph Title
PAGE FOOTER
Caption
Small BodySmall Paragraph Title
Large Body
LABEL
Code Code Small

Layer 7 Observability and Centralized Configuration with Consul Service Mesh

More Related Content

What's hot

Similar to Layer 7 Observability and Centralized Configuration with Consul Service Mesh

More from Mitchell Pronschinske

Recently uploaded

Layer 7 Observability and Centralized Configuration with Consul Service Mesh