Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A New Model for
Image Distribution
Stephen Day
Distribution, Tech Lead
Docker, Inc.
stephen@docker.com
@stevvooe
github.com/stevvooe
Overview
• Why does this matter?
• History
• Docker Registry API V2
• Implementation
• The Future
What is Docker?
What is an Image?
What is an Image?
• Identified by a name
• ubuntu
• redis
• stevvooe/myapp
• docker run ubuntu
- Runs a container, created...
What is an Image?
• Containers, the runtime of docker, are created from images
• Filesystem made up with “layers”
- Just t...
What is the Docker
Registry?
What is the Docker Registry?
• A central place to store and distribute
docker images
• Stores the layers and the descripti...
What is the Docker Registry?
• Several Implementations
• A simple web server to make images available
• A complete web app...
History
Docker Registry API V1: History
• Layer Oriented
• Layer IDs are randomly assigned
• JSON object corresponding to each lay...
Registry API V1 URL Layout
Methods URL
GET /v1/_ping
GET, PUT /v1/images/(image_id)/layer
GET, PUT /v1/images/(image_id)/j...
Docker Registry API V1: Problems
• Abstraction
- Exposes Internals of Image to distribution mechanism
• Security
- Image I...
Docker Registry API V1: Problems
• Implementation in Python
- Affected ease of deployment
- Reduced sharing with main Dock...
Docker Registry API V2
Docker Registry API V2: Goals
• Simplicity
- Easy to implement
- Works with static host
• Security
- Verifiable Images
- S...
Docker Registry API V2: Goals
• Distribution
- Separate location of content from naming
• Performance
- Remove the single ...
Docker Registry API V2: Content Addressable
• Layers are treated as content-addressable blobs
- Much better for security
-...
Docker Registry API V2: Digests
• Uniquely identifies content
• A cryptographically strong hash
- Chose a name, digest, th...
Docker Registry API V2: Manifests
• Describes the components of an image in a single object
- Layers can be fetched immedi...
Docker Registry API V2: Manifests
{
"name": <name>,
"tag": <tag>,
"fsLayers": [
{
"blobSum": <digest>
},
...
]
],
"history...
Docker Registry API V2: Manifest
• Content-addressable:
- docker pull
ubuntu@sha256:8126991394342c2775a9ba4a843869112da815...
Docker Registry API V2: Repositories
• All content is now part of a named repository
- Image IDs are no longer a secret
- ...
Registry API V2 URL Layout
Methods URL
GET /v2/
GET /v2/<name>/tags/list
GET, PUT, DELETE /v2/<name>/manifests/<reference>...
Docker Registry API V2: Design
• Shared-nothing
- “Backend” ties a cluster of registries together
- Allows scaling by addi...
Docker Registry API V2: Differences with V1
• Content addresses (digests) are primary identifier
• Unrolled image descript...
Docker Registry
2.0
–Earl Milford
“[A registry] should be
neither seen nor heard. ”
Handlers
Docker Registry 2.0: Architecture
Repository Repository
Storage
Access Control
Notifications
Docker Engine
Auth
A...
Docker Registry 2.0: An Ingredient
• Move away from monolithic architecture
• Narrower scope
- Distribute content
• Extens...
Docker Registry 2.0
• Full support released with Docker 1.6
- Minimal bugs
- Most problems are common to version upgrades
...
Docker Registry 2.0: Should you use it?
• Are you on Docker 1.6+?
- Yes.
• Evaluate it
• Test it
• Break it (and file bugs...
Docker Registry 2.0: Deploying
• Internal deployments
- Use the filesystem driver — it is really fast
- Backup with rsync
...
Docker Registry 2.0: Docker Hub
• Running the Hub
- S3 backend
• Having some trouble with round trips to s3 :(
- Decent pe...
Monitoring culture
Docker Hub Adoption
0%
50%
100%
Last Three Months
V1
(1.5-)
V2
(1.6+)
Docker Hub Adoption
• Overall usage increasing
• A V2 world and growing
V1/V2 Protocol Overall Comparison
0
25
50
75
100
Requests Bandwidth
V1
V2
80% Fewer Requests 60% Less Bandwidth
V1/V2 Protocol HTTP Errors
Peak Average
V1
V2
5
Exceptional Panicking
• 1 Panic in Three Months of Production
• 4000 protocol level errors per 30 minutes in V1
• 5 protoc...
Docker Registry
2.1
Docker Registry 2.1
• Key Changes
- Documentation
- Pull-through Caching
- Soft-Deletion
- Native Basic Auth Support
- Sta...
Docker
Distribution
Docker Distribution: Goals
• Goals
- Improve the state of image distribution in Docker
- Build a solid and secure foundati...
Docker Distribution: Future
• Ingredients
- From the start, we have targeted solid packages
- Provide Lego to build image ...
Thank you
Stephen Day
Google Group: distribution@dockerproject.org
GitHub: https://github.com/docker/distribution
IRC on F...
Upcoming SlideShare
Loading in …5
×

Docker Registry V2

20,079 views

Published on

Slides from presentation at DockerCon SF 2015

Published in: Technology

Docker Registry V2

  1. 1. A New Model for Image Distribution
  2. 2. Stephen Day Distribution, Tech Lead Docker, Inc. stephen@docker.com @stevvooe github.com/stevvooe
  3. 3. Overview • Why does this matter? • History • Docker Registry API V2 • Implementation • The Future
  4. 4. What is Docker?
  5. 5. What is an Image?
  6. 6. What is an Image? • Identified by a name • ubuntu • redis • stevvooe/myapp • docker run ubuntu - Runs a container, created from image ubuntu
  7. 7. What is an Image? • Containers, the runtime of docker, are created from images • Filesystem made up with “layers” - Just tar files - Layers can be shared between images • Includes a description organizing layers into an image A runnable component with a filesystem
  8. 8. What is the Docker Registry?
  9. 9. What is the Docker Registry? • A central place to store and distribute docker images • Stores the layers and the description of how they make up an image • Implements a common API agreed upon by Docker clients
  10. 10. What is the Docker Registry? • Several Implementations • A simple web server to make images available • A complete web application • Services • Docker Hub • Docker Trusted Registry • Documentation: https://docs.docker.com/registry/ A central place to store and distribute docker images
  11. 11. History
  12. 12. Docker Registry API V1: History • Layer Oriented • Layer IDs are randomly assigned • JSON object corresponding to each layer referencing a parent • Naming accomplished through tags Layer Layer Layer Layer JSON JSON JSON JSONFetch(ID) {
  13. 13. Registry API V1 URL Layout Methods URL GET /v1/_ping GET, PUT /v1/images/(image_id)/layer GET, PUT /v1/images/(image_id)/json GET /v1/images/(image_id)/ancestry GET /v1/repositories/(namespace)/(repository)/tags GET, PUT, DELETE /v1/repositories/(namespace)/(repository)/tags/(tag*) DELETE /v1/repositories/(namespace)/(repository)/ GET /v1/search 13 https://docs.docker.com/reference/api/hub_registry_spec/
  14. 14. Docker Registry API V1: Problems • Abstraction - Exposes Internals of Image to distribution mechanism • Security - Image IDs must be kept secret - Who assigns the layer IDs? - Hard to audit, verify • Performance - Fetch a layer, fetch the parent, fetch the parent, …
  15. 15. Docker Registry API V1: Problems • Implementation in Python - Affected ease of deployment - Reduced sharing with main Docker Project • More information: • https://github.com/docker/docker/issues/8093
  16. 16. Docker Registry API V2
  17. 17. Docker Registry API V2: Goals • Simplicity - Easy to implement - Works with static host • Security - Verifiable Images - Straightforward access control
  18. 18. Docker Registry API V2: Goals • Distribution - Separate location of content from naming • Performance - Remove the single track • Implementation - Use Go to increase code sharing with Docker Engine
  19. 19. Docker Registry API V2: Content Addressable • Layers are treated as content-addressable blobs - Much better for security - Permits safe-distribution through untrusted channels • All data can be verified • De-duplication • Improved cache-ability • Content address is known as the “digest”
  20. 20. Docker Registry API V2: Digests • Uniquely identifies content • A cryptographically strong hash - Chose a name, digest, that does not conflict with other concepts (map, dict, crc, etc.) - Simply using sha256(bytes) • Independently Verifiable - By agreeing on common algorithm, IDs chosen for content without coordination • Strongly-typed with tools to parse and verify - http://godoc.org/github.com/docker/distribution/digest
  21. 21. Docker Registry API V2: Manifests • Describes the components of an image in a single object - Layers can be fetched immediately, in parallel LayerLayer Layer Layer JSONFetch(ID) {
  22. 22. Docker Registry API V2: Manifests { "name": <name>, "tag": <tag>, "fsLayers": [ { "blobSum": <digest> }, ... ] ], "history": [<v1 image json>, ... ] }
  23. 23. Docker Registry API V2: Manifest • Content-addressable: - docker pull ubuntu@sha256:8126991394342c2775a9ba4a843869112da815 6037451fc424454db43c25d8b0 • Leverages Merkle DAG - Because the digests of the layers are in the manifest, if any bit in the layer changes, the digest of the manifest changes - Similar to git, ipfs, camlistore and a host of other projects • Tags are in the manifest - This will going away
  24. 24. Docker Registry API V2: Repositories • All content is now part of a named repository - Image IDs are no longer a secret - Simplified authorization model • repository + operation (push, pull) - Clients must “prove” content is available to another repository by providing it • Opened up namespace to allow more than two components - No reason to have registry enforce “<user>/<image>” - API “reversed” to make static layout easier
  25. 25. Registry API V2 URL Layout Methods URL GET /v2/ GET /v2/<name>/tags/list GET, PUT, DELETE /v2/<name>/manifests/<reference> GET /v2/<name>/blobs/<digest> POST /v2/<name>/blobs/uploads/ GET, PUT, PATCH, DELETE /v2/<name>/blobs/uploads/<uuid> https://docs.docker.com/registry/spec/api/
  26. 26. Docker Registry API V2: Design • Shared-nothing - “Backend” ties a cluster of registries together - Allows scaling by adding instances - Performance limited by backend • Make backend faster, registry gets faster • Pull-optimized - Most important factor when distributing software - May hurt certain use cases • Resumable Pull and Push (specified but not implemented) - Resumable pull already available with http Range requests - Two-step upload start for resumable push - Built into the protocol for future support • A living specification - Meant to be used and modified - Always backwards compatible
  27. 27. Docker Registry API V2: Differences with V1 • Content addresses (digests) are primary identifier • Unrolled image description model • Multi-step upload - Provides flexibility in failure modes - Options for future alternative upload location (redirects) • No Search API - In V1, this API does everything - Replacing with something better • No explicit tagging API - This will change: https://github.com/docker/distribution/pull/173
  28. 28. Docker Registry 2.0
  29. 29. –Earl Milford “[A registry] should be neither seen nor heard. ”
  30. 30. Handlers Docker Registry 2.0: Architecture Repository Repository Storage Access Control Notifications Docker Engine Auth API
  31. 31. Docker Registry 2.0: An Ingredient • Move away from monolithic architecture • Narrower scope - Distribute content • Extensible - Authentication - Index - Ponies • Strong core - Docker Hub - Docker Trusted Registry
  32. 32. Docker Registry 2.0 • Full support released with Docker 1.6 - Minimal bugs - Most problems are common to version upgrades • Header required to declare support for 2.0 API • Validated most concepts in 1.3, 1.4 with V2 preview - Much faster pull performance - You’ve probably already used it with Docker Hub • There are some edge cases - push-heavy workflows - disk IO when verifying large images - We are mitigating these
  33. 33. Docker Registry 2.0: Should you use it? • Are you on Docker 1.6+? - Yes. • Evaluate it • Test it • Break it (and file bugs https://github.com/docker/distribution/issues) • Deploy it • Are you on Docker <1.6? - Are you entrenched in v1? • Perhaps, hold off - Run dual stack v1, v2 • Not recommended
  34. 34. Docker Registry 2.0: Deploying • Internal deployments - Use the filesystem driver — it is really fast - Backup with rsync • Scale storage - Use S3 driver • Make sure you are “close” since round trip times can have an effect • Scale Reads - Use round robin DNS • Do not use this for HA - Rsync to followers on read-only filesystem - Add machines to taste • https://docs.docker.com/registry/deploying/
  35. 35. Docker Registry 2.0: Docker Hub • Running the Hub - S3 backend • Having some trouble with round trips to s3 :( - Decent performance with very little caching • A lot of low hanging fruit left to tackle • No longer intertwined with Docker Hub services • Independent Authentication Service • Heightened Availability
  36. 36. Monitoring culture
  37. 37. Docker Hub Adoption 0% 50% 100% Last Three Months V1 (1.5-) V2 (1.6+)
  38. 38. Docker Hub Adoption • Overall usage increasing • A V2 world and growing
  39. 39. V1/V2 Protocol Overall Comparison 0 25 50 75 100 Requests Bandwidth V1 V2 80% Fewer Requests 60% Less Bandwidth
  40. 40. V1/V2 Protocol HTTP Errors Peak Average V1 V2 5
  41. 41. Exceptional Panicking • 1 Panic in Three Months of Production • 4000 protocol level errors per 30 minutes in V1 • 5 protocol level errors per 30 minutes in V2
  42. 42. Docker Registry 2.1
  43. 43. Docker Registry 2.1 • Key Changes - Documentation - Pull-through Caching - Soft-Deletion - Native Basic Auth Support - Stability - Catalog API - Storage Drivers • Release coming by mid-July
  44. 44. Docker Distribution
  45. 45. Docker Distribution: Goals • Goals - Improve the state of image distribution in Docker - Build a solid and secure foundation • Focus - Security - Reliability - Performance • Unlock new distribution models - Integration with trust system (notary!) - Relax reliance on registries - Peer to Peer for large deployments
  46. 46. Docker Distribution: Future • Ingredients - From the start, we have targeted solid packages - Provide Lego to build image distribution systems • Clean up the docker daemon code base - Defined new APIs for working with docker content - Increase feature velocity - Generalize around strong base • Current Manifest format is provisional - Still includes v1 layer JSON - Content-addressability + mediatypes make support new formats trivial - https://github.com/docker/distribution/pull/62 • Feature parity with V1 and maturity - Building collective operational knowledge • Deletes and Garbage Collection - Diverse backend support makes this hard - https://github.com/docker/distribution/issues/461 - https://github.com/docker/distribution/issues/462 • Search - See the goals of Distribution to see why this is interesting • Road Map: https://github.com/docker/distribution/wiki
  47. 47. Thank you Stephen Day Google Group: distribution@dockerproject.org GitHub: https://github.com/docker/distribution IRC on Freenode: #docker-distribution

×