1. Docker Hub
Past, present and future

2. Ken Cochrane
@KenCochrane
bc Wong
@bcwalrus
We are engineers working on Docker Hub

3. Docker Hub
Docker Hub (hub.docker.com) is a cloud registry service for
sharing application and automating workflows.
Major features:
• Public and Private Docker repositories
• Official Repositories
• Collaborators, Organizations and Groups
• Automated builds
• Webhooks
3

4. Docker Hub - A brief history

5. Hub by the Numbers
• 240,000 Users
• 150,000 Repositories
• 1,250,000 Automated Builds
• 500,000,000 Pulls
5

6. Hub Growth (disk space)
6

7. Hub Growth (bandwidth)
7

8. Growth of a distributed application

9. Docker Hub: This is your life

10. Docker Hub: This is your life (cont.)

11. Docker Hub: This is your life (cont.)

12. Docker Hub: This is your life (cont.)

13. Docker Hub: This is your life (cont.)

14. Docker Hub: This is your life (cont.)

15. Docker Hub: This is your life (cont.)

16. Docker Hub: This is your life (cont.)

17. Docker Hub: This is your life (cont.)

18. Docker Hub Timeline
18

19. Docker Hub - More than just a registry

20. Organizations & Groups
• Best suited for businesses
• Collaborate on repos with members of your organization
• Fine-grain permissions
• Add users to groups
• Public and Private repos
20

21. Official Repositories
• 80 Official Repos that you can use to build your Docker applications
• Repos are updated and maintained by Docker
21

22. Private Repositories
• Not searchable or browsable
• Can add collaborators to push or pull
• Only you or a collaborator can access
• 1 free private repo with your docker hub user account
22
Coupon for 3 free months of the micro plan
dockercon15

23. Automated Builds
• Link to your Github or Bitbucket accounts
• Keep your source code in sync with your docker repos
• You write the code, we handle the build, and keep your Docker hub
repos up to date.
• Public and Private repos
23

24. Repository Links
What if your base ubuntu:14.04 image is updated for a security fix?
• Link your automated build repository to another repository
• Allows you to keep your automated build up to date
24

25. Build Triggers
• Trigger an automated build from an external system
• Useful for integrating with CI systems
25

26. Webhooks
• Get notified when a docker repo is updated
• Ability to chain together webhooks to create a build pipeline
• Combine with a continuous integration and deployment solution to
easily move your application from development to server.
26

27. And much more ...
• Search
• Stars
• Comments
• Notifications
27

28. Upcoming Features
• Show image sizes
• GitLab support for automated builds
• Dynamic tagging for automated builds
28

29. Docker Hub 2.0 (beta)

30. Hub 2.0
• New, Easier to use UI
• Re-architected, isomorphic javascript application based on react.js,
react-router and fluxible
• A modern, modular approach using ES2015 enabling faster iterations
30

31. Data stores
Web UI
API
Gateway
Postgresql Redis KafkaRabbitMQ
Elastic
Search
Micro
Services
S3
New Architecture

32. Live demo time: Hub 2.0
hub-beta.docker.com
Left brain: What could go wrong?
Right brain: Some people never learn.

33. Give us feedback please
github.com/docker/hub-beta-feedback
33

34. Docker Hub - Performance

35. Docker Hub Performance - push/pull
35

36. Architecture Upgrade - Registry v2
• Registry v2 protocol is far superior to v1
• Supported by Docker >=1.6
• Content-addressable manifest
• More secure
36
80% fewer requests
60% less bandwitdth

37. Architecture Upgrade - Disentangle
• Clean up inter-service dependencies:
• Hub 2.0 uses a gateway with finer grain services.
• Reduces cascading failures.
37

38. Infrastructure Upgrade
• Aka “throw money at the problem”
• The difficult parts are when (i.e. monitoring), and the execution.
• E.g. database upgrade:
38
External ping latency (worldwide average)DB response time (before vs after)

39. Better Service Isolation
39

40. Automated Builds (v1 current)
Architecture
• A set of long-running build servers w/o auto-scaling
• One build per server at a time (docker in docker)
• Thousand’s of builds per day
40
Challenges
• # of builds <= # of servers, which
means pending builds
• Security concerns
• Build server issue == lost capacity

41. Automated Builds (v2 upcoming)
Speed and efficiency
• Auto-scale on-demand builds
• Per-user queuing
• Better scheduling
Security
• Use VM for isolation between users
• TLS everywhere
• Ephemeral build servers
41

42. For those about to rock ...

43. Scalability & Reliability Work: Myth vs Reality
43
Need a team that can do the first 80% and the last 20%.
Need a culture that understands quality.

44. Why is my sink not draining?
From our post-mortems & bug reports:
• DB capacity
• ORM layer makes slow queries
• Networking flakiness
• RabbitMQ failures (and we didn’t understand its failure models)
• Supervisor failed to restart appserver, but NGINX keeps accepting calls
• Expired intermediate cert
• Docker btrfs driver bug
• ...
Know when to tighten a bolt, and when to get a new sink.
44

45. High Velocity Cycle (invest in tooling)
45
Monitoring
& logging
(what’s slow)
Features & fixes
QA, scale test,
perf regression test
(prove that things work)

46. Future Performance Work
• Continue to eliminate bottlenecks as Hub grows
• Speed up push/pull outside the US
• Scalable pipeline into data warehouse
46

47. Credits: The Hub Team
Ryan Abrams
Chris Biscardi
Ken Cochrane
John Costa
Roberto Hashioka
Daniel Hiltgen
Nathan Hsieh
Nick Kraly
Adrian Kreher
Toli Kuznets
Dustin Lacewell
47
Nick Lang
Sean Li
Amy Lindburg
Marcus Martins
Chinmayee Nirmal
Atul Phadnis
Mario Ponticello
Arunan Rabindran
Josh South
Justen Stepka
bc Wong

48. Credits: The Hub Team

49. Questions?
Remember to check out hub-beta,
and send us feedback:
github.com/docker/hub-beta-feedback