Shortly describe how Consul could help us and propose some minor changes in department for future continuous delivery.

1. Consul
-­‐
Service
discovery
and
others
By
Walter
Liu
2014/11/11

2. Similar
services
• Consul
• SkyDNS
• Others
• Zookeeper
• etcd
• Serf

3. What’s
Consul?
• Service
Discovery
• DNS
• HTTP
(JSON)
• Health
Checking
(Service
Discovery
need
this)
• Key/Value
Store
• dynamic
configuraMon
• feature
flagging
• coordinaMon
• leader
elecMon
• (Very
like
the
funcMonality
of
Zookeeper
we
used)
• MulM
Datacenter
• Network
parMMon
• If
one
DC
off,
no
impact
to
another
DC.
• Data
duplicate
to
another
DC
automaMcally.

5. User
Scenarios
in
Our
Infra
• Simplify
config
(even
with
SaltStack)
• Dynamic
server
cluster
• Faster
failure
recovery
• Dynamic
cluster
configuraMon
• Launching
a
node
in
a
container
cloud.
(Future
maybe?)

6. Scenario:
Simplify
con>ig
• ENV
(from
hostname
or
specified
in
Salt)
• $TYPE=“prod”
• $GROUP=“gg”
• $DATACENTER=“mydc1”
• Config
could
be
• HOST_POSTFIX
=
$GROUP+$TYPE+$DATACENTER
=
• “prod.gg.mydc1.consul”
• Config
in
Group
• “db.prod.gg.mydc.consul”
• Config
for
a
datacenter
• “redis.prod.mydc1.consul”
• So,
every
config
for
prod/stg/dev
could
be
the
same.
• Do
config
in
service
provider
instead
of
every
client.
• Service
provider
could
automaMcally
register
if
supported.

7. Scenario:
Dynamic
server
cluster
• Background
• A
server
cluster
without
load
balancer
(Cannot
apply
LB)
• An
applicaMon
that
use
this
service
• Sequence
• Query
Consul
will
return
a
list
of
servers.
• When
a
server
failed,
Consul
will
aware
that
very
quickly
and
remove
it
from
the
list.
• An
applicaMon
that
use
this
cluster
will
be
interrupted
and
need
to
re-­‐query
Consul.
Consul
will
return
the
new
list
of
servers.

8. Scenario:
Faster
failure
recovery
• Background
• A
master/slave
DB
without
automaMc
failover
• Sequence
• The
master
DB
is
dead.
• Nagios
found
it.
• OPS
aware
it
and
log-­‐in
slave
DB.
• OPS
restart
slave
DB
as
master
and
run
script
to
register
new
DB
• All
applicaMons
that
use
this
DB
will
change
to
new
DB.

9. Scenario:
Dynamic
cluster
con>iguration
• If
a
service
cluster
needs
dynamic/automaMc
change
configuraMon
during
run-­‐Mme.

10. Scenario:
Launching
a
node
in
a
container
cloud.
• Assume
the
container
image
of
the
service
is
ready.
• Specify
correct
hostname
of
the
new
container.
• Launch
the
new
container.
• The
new
container
start
the
new
service.
• If
it
is
a
simple
service
provider,
it
registers
itself
to
Consul.
• If
it
uses
other
services,
it
queries
these
services
from
Consul.
• If
the
configuraMon
is
all
the
same,
the
new
node
is
ready.

11. OTHERS

12. Use
pip
+
virtualenv
• Why,
• Reduce
package
conflict.
(Use
egg
is
not
very
friendly.)
• Some
Python
packages
doesn’t
have
RPM.
• Install
using
requirement.txt
(pip
install
–r
requirement.txt)
• (?)
private
pip
repository
• Below
for
dev
• Use
virtualenvwrapper
to
be
more
convenient.
• How
to
produce
requirement
more
easily?
• pip
freeze
>
stable-­‐req.txt
• Pip
freeze
–r
dev-­‐req.txt
>
stable-­‐req.txt
#
Keep
old
comments

13. Example:
requirement.txt
• Beaker==1.3.1
• Chameleon==2.11
• DecoratorTools==1.7
• DistuMls2==1.0a3
• Django==1.5.1
• M2Crypto==0.20.2
• Magic-­‐file-­‐extensions==0.1
• Mako==0.8.1
• MarkupSafe==0.9.2
• MySQL-­‐python==1.2.3c1
• PasteDeploy==1.5.0
• PyXML==0.8.4
• Pygments==1.1.1
• SOAPpy==0.11.6
• SSSDConfig==1.11.6
• Scrapy==0.18.4

14. Store
SHOULD-­‐BE-­‐CHANGED
con>ig
in
environment
var
• Why,
• Leak
secrets
in
version
control
system.
• Check-­‐in
always
config
to
version
control
system
accidentally.
• Example,
• DB
username/password,
AWS
API
Keys
• Secret-­‐token
(web
framework
like
Django)
• URI
resource
string,
Used
service
host,
port
…..

15. Explicitly
store
prod/stg/dev
settings
in
version
control
• Why,
• Check-­‐in
always
config
to
version
control
system
accidentally.
• All
sevngs
in
all
environment
are
visible
to
everyone.
• For
example,
• DEBUG=True
in
Django
• Log
level
• How
(Django
for
example)
• sevngs/
• prod.py
• staging.py
• dev.py
• ENV[“DJANGO_SETTINGS_MODULE”]
=
“sevngs.prod”
• “sevngs.prod”
could
automaMcally
come
from
hostname.
• If
this
env
variable
is
not
set,
service
will
print
log
and
exit.

16. Consul
vs.
SkyDNS
• Pros
• Beyer
health
check
(SkyDNS
uses
heartbeat
+
TTL)
• MulMple
datacenter
support
by
using
“region”.
• A
beyer
web
view.
(hyp://demo.consul.io/ui/)
• Beyer
RAFT
implementaMon
(unverified)
• Cons
• lesser
contributors
(1/3)
• More
immature
(0.41
now)
and
young.

17. Problem
needs
attention
• For
only
one
server
service
like
DB,
what
if
a
second
server
register
the
service
accidentally?
• Use
DNS+DNS
forwarding
approach
may
not
be
good
for
DNS
heavy
service.
(Like
Odin)

18. Other
References
• Service
registrator
for
Docker
• hyps://github.com/progrium/registrator
• Set
env
by
Consul
• hyps://github.com/hashicorp/envconsul