Since the introduction of the Berkshelf API and subsequent integration with the Chef Community Supermarket, organizations have increasingly wished for a private supermarket to showcase their in-house creations. In this talk, you will learn how to configure and deploy your own private supermarket with test-kitchen and Vagrant. You will also learn a bit about how a private supermarket can work with your CI/CD environment, and how it integrates into your organization’s infrastructure. https://youtu.be/U-34dQbgsfM

1. Building
a
Private
Supermarket
for
your
Organization
Justin
Dossey

2. Who
I
am
(and
why
you
should
listen
to
me)
• I’m
Justin
Dossey.
• Twenty
years
as
a
professional
sysadmin
• Ten
years
as
a
Rubyist
• Seven
years
with
CM
• I
just
built
a
bunch
of
supermarkets
a
couple
of
months
ago
• We’re
hiring!
Talk
to
me
after.

3. A
Brief
History
of
the
Supermarket

4. A
Brief
History
of
the
Supermarket
• Built
for
Chef
as
a
simple
Ruby
on
Rails
app
• Hosted
in
AWS
• First
commit
(on
Github):
October
2013
• Originally
designed
as
something
to
be
hosted
for
Chef,
by
Chef,
as
a
public
thing
• This
means
that
stuff
like
CLA
signing
was
baked
in
early
• It
also
means
some
things
about
private
are
funky

5. Why
Build
Private?
• The
organization
makes
a
lot
of
its
own
cookbooks
• Public
supermarket
may
provide
cookbooks
of
suspect
quality
• The
organization
is
worried
that
the
public
supermarkets
could
be
closed
when
it
needs
to
shop
• Worries
that
the
public
supermarket
could
be
robbed
or
infiltrated

6. The
Public
Supermarket
Code
makes
the
private
supermarket
possible

7. The
Supermarket
is
Open-­‐Source
• Chef
makes
the
supermarket
code
available
to
everyone
• If
they
can
run
one,
so
can
we!
• …but
we’re
not
Chef,
so
we
might
have
to
change
some
things
about
how
it
works.
• There
are
an
awful
lot
of
build
dependencies
• …
but
there’s
a
cookbook,
so
it’s
easy,
right?

8. Moving
Parts
how
the
supermarket
works

9. Supermarket
Architecture
• Supermarket
is
a
Ruby
on
Rails
application.
• It
talks
to
a
Postgres
Database
• It
uses
dotenv
to
store
environment
variables
(usually
related
to
config)
• It
uses
omniauth
+
devise
• Redis
is
in
there
too
9

10. Supermarket
Architecture
• Lots
of
integrations
with
external
services:
• oc-­‐id
on
the
Chef
server
• Github
• StatsD
• Sentry
• Amazon
S3
• Fieri
• Curry
• Pubsubhubbub
• The
SMTP
relay
• New
Relic
• Google
Analytics
10

11. Our
Minimum
Working
Set
• Lots
of
integrations
with
external
services:
• oc-­‐id
on
the
Chef
server
• Github
• StatsD
• Sentry
• Amazon
S3
• Fieri
• Curry
• Pubsubhubbub
• The
SMTP
relay
• New
Relic
• Google
Analytics
11

12. Configuring
Fun
with
Data
Bags

13. Yes,
Data
Bags.
• For
whatever
reason,
Chef’s
Supermarket
developers
decided
that
the
best
place
to
put
the
application
configuration
was
a
data
bag.
• Back
in
2013,
this
probably
made
sense.
• That
decision
remains
with
us
today
and
feels
weird,
because
we
all
write
wrapper
cookbooks
to
accomplish
the
same
thing
nowadays.
• The
data
bag
we
need
is
called
“apps/
supermarket.”

14. Stuff
in
the
Data
Bag
• redis
URL
• SMTP
server
info
• github
info
• oauth2
tokens
for
oc-­‐id
• curry
info
• sentry
url
• which
revision
of
supermarket
to
clone
• a
bunch
of
URLs

15. Stuff
not
in
the
Data
Bag
• SSL
stuff
• supermarket
hostname
• deploy
location
• some
database
config
stuff
• some
sidekiq
stuff

16. Doing
this
Ourselves

17. Write
a
Wrapper
Cookbook
• Put
attributes
and
a
working
data
bag
in
place.
• Get
the
.kitchen.yml
written
and
the
thing
building
in
dev,
then
build
the
production
system.
• Read
the
docs—
this
stuff
is
changing
pretty
frequently.
• When
stuff
breaks,
look
carefully
at
the
production.log,
the
.env
file,
and
the
data
bag.

18. Authentication
to
the
Chef
Server
• The
Chef
server
needs
to
know
who
is
allowed
to
register
apps
for
oc-­‐id
auth.
• Last
I
checked,
that
was
a
line
in
/etc/opscode/chef-server.rb
like
this:
• Once
that’s
in
and
we
run
chef-server-ctl
reconfigure,
we
can
get
the
oauth
stuff
we
need
for
that
data
bag.
oc_id[‘administrators’] = [‘chef_username’]

19. Continuous
Integration
and
Continuous
Deployment
for
Cookbooks
A
brief
overview
of
pipeline
design
for
infrastructure
code

20. How
it’s
Usually
Done

21. Doing
it
with
Supermarket

22. Sticky
Situations

23. Cookbook
Naming
• Berkshelf
has
a
flat
namespace.
• We
can
specify
multiple
sources
for
Berkshelf.
• If
our
(in-­‐house)
cookbook
is
named
the
same
as
one
on
the
public
supermarket,
the
version
is
the
only
differentiator.
This
can
also
collide.
• Switching
to
a
single
origin
means
someone
has
to
upload
community
cookbooks
to
the
private
supermarket
• Doing
this
trades
namespace
collisions
for
pipeline
and
maintenance
overhead

24. Cookbook
Ownership
• With
the
pipeline
proposed,
the
only
uploader
of
cookbooks
is
a
role
user—
Jenkins.
• Therefore,
connecting
supermarket
accounts
to
github
accounts
is
weird.
• We
wind
up
assigning
ownership
outside
Supermarket.

25. The
Future
of
Private
Supermarkets

26. Supermarket
Omnibus
Installer
• Increasingly,
Chef
is
pushing
installation
via
pre-­‐built
packages
instead
of
the
Supermarket
cookbook.
• This
means
that
instead
of
the
data
bag
and
such,
we
will
write
a
simple
wrapper
cookbook
to
install
a
package
and
configure
components
as
necessary.
• It
is
unlikely
that
this
newer
method
will
affect
any
of
the
integration
points.

27. Review

28. Review
• Running
a
private
supermarket
means
adapting
the
public
code
(which
performs
tasks
relevant
to
the
public
supermarket)
to
our
needs
• We
need
our
own
chef
server
to
host
a
private
supermarket
• The
supermarket
cookbook
uses
a
data
bag
for
some
configuration
data
• There
are
many
external
integration
points
which
we
can
select
if
we
like,
but
most
are
not
required
for
basic
operation
• We
have
to
decide
how
to
approach
namespace
and
ownership
issues
early
on
28

29. Questions

30. Thank
you!
(Did
I
mention
we
are
hiring?)
Find
me!
blog:
www.justindossey.com
twitter:
@justindossey
irc:
jdossey
on
#chef
(freenode)
email:
justin@newcontext.com