The document summarizes Robert Bosch GmbH's efforts to design and implement a centralized container platform and global image repository for its large, multicluster enterprise environment. It established standardized Docker clusters across its 280+ sites worldwide and deployed multiple Docker Trusted Registries for image distribution. This centralized approach replaced decentralized environments, reduced costs and efforts for users, and ensured security and compliance. Lessons learned included the need for central management of the multi-component system and additional security tools for container environments.

1. DESIGNING A CENTRALIZED
CONTAINER PLATFORM FOR A LARGE,
MULTI-CLUSTER ENTERPRISE
ENVIRONMENT
DockerCon Europe 2018, Barcelona
Mijo Safradin
CaaSP Service Owner
Robert Bosch GmbH
mail: mijo.safradin@de.bosch.com

2. About me
 Linux Engineer ~ 15 years
 System programming on mainframe, PowerPC, x86
 Using Docker since 2013/14
 Linux networking - SDN developer
 Operations engineer since 2015

3. CI/OSB – Mijo Safradin | 2018-12-05
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
2
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outcomes

4. © Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
3
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outcomes
CI/OSB – Mijo Safradin | 2018-12-05

5. Bosch
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
4
Market and figures for 2017*
 Leading in drive and control technology, packaging,
and process technology
Industrial Technology
 One of the world’s largest suppliers of mobility solutionsMobility Solutions
 Leading supplier of power tools and accessories
 Leading supplier of household appliances
Consumer Goods
 One of the leading manufacturers of security and communication technology
 Leading manufacturer of energy-efficient heating products
and hot-water solutions
Energy and Building
Technology
Bosch Group
402,166
associates
78.1 billion euros
in sales
* as of 12.17
CI/OSB – Mijo Safradin | 2018-12-05

6. Market and figures for 2017*
Bosch
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
5
78.1 billion euros
in sales
280
manufacturing sites
Bosch Group
402,166
associates
1 Including other countries
Europe
160
245,040
52%
Asia Pacific1
78
112,817
30%
Americas
42
44,309
18%
Manufacturing sitesAssociatesShare of sales* as of 12.17
CI/OSB – Mijo Safradin | 2018-12-05

7. Bosch is one
of the world’s leading
international providers
of technology
and services
Over the past five
years, Bosch has
invested more than
30 billion euros in
research and
development
~64,5001
R&D associates
1252
engineering locations
worldwide, in a single
network
Bosch
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
6
Technology to enhance quality of life
Our objective:
To develop
innovative, useful, and
exciting products and
solutions to enhance
quality of life – technology
that is
“Invented for life”
1 Preliminary, rounded figures as of 2018 2 R&D locations with >50 associates, as of 12.17
CI/OSB – Mijo Safradin | 2018-12-05

8. Corporate sector information systems & services (CI)
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
7
Facts and Figures
*
Conception, development, implementation and operation of IT
applications within the Bosch Group
IT Partner
 Evaluating and consulting on new IT technologies for business units
 Driving flexibility with scalable solutions
IT Provider
 Offering state-of-the-art IT infrastructure that lives up to business
environment globalization
 Providing attractive global IT services
IT Governance
 Achieving economies of scale with standardization
 Ensuring secure and cost-effective operation
7,750 Employees worldwide
CI Core Locations
 EU: Stuttgart (GER),
 AP: Shanghai (CHN) & Singapore (SGP)
& Bangalore (IND)
 AM: Chicago (USA) & Campinas (BRA)
 Over 260 Locations worldwide
*Plan 2018
CI/OSB – Mijo Safradin | 2018-12-05

9. Corporate sector information systems & services (CI)
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
8
CI Worldwide presence
Americas Europe Asia / Pacific
CI Core Locations
Country with CI presence
Campinas
Chicago
Stuttgart
Shanghai
Singapore
Bangalore
CI/OSB – Mijo Safradin | 2018-12-05

10. © Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
9
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outcomes
CI/OSB – Mijo Safradin | 2018-12-05

11. General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
10
Why a new CI service ?
 Docker engines were found on
hundreds of managed devices
 Investigations showed that
mostly software developers
used Docker engines
CI/OSB – Mijo Safradin | 2018-12-05

12. General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
11
Why a new CI service ?
 At the same time the users of
these engines complained about
 technical challenges
 compliance challenges
 effort for deployment
 effort for operations
 effort for maintenance
Docker Hub
CI/OSB – Mijo Safradin | 2018-12-05

13. General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
12
Why a new CI service ?
 CI recognized the demand for a
central service with the target to
 Replace existing decentralized
environments by clustered systems
 Free up resources at the user
departments
 Reduce the overall effort for
container environment
 Ensure security and compliance of
the managed container environments
CI/OSB – Mijo Safradin | 2018-12-05

14. Who will use the new CI service ?
 Developers
 Enable teams to use CI, CD and
CT methods
 Ensure compatibility of software
with different environments
 Test software in different given
customer configurations
 Engineers
 Scalable, easy-to-configure runtime
for various types of analysis
General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
13 CI/OSB – Mijo Safradin | 2018-12-05

15. Who will use the new CI service ?
 Service provider
 Scalable runtime environment for
applications
 Standardized deployment of
applications to various locations
 Fast deployment of PoC and test
environments
General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
14 CI/OSB – Mijo Safradin | 2018-12-05

16. General information
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
15
What should the new CI service provide ?
 A managed, global image repository
 Managed Docker runtime clusters with
container orchestration functionalities
 Central management and operations for all
deployed systems
 24x7 support via help desk
CI/OSB – Mijo Safradin | 2018-12-05

17. © Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
16
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outcomes
CI/OSB – Mijo Safradin | 2018-12-05

18. Requirements from various stakeholders
 Runtime must be orchestrated e.g. Docker Swarm, Kubernetes, …
 Workload must be executed as close as possible to the data
 API must be supported by current development pipelines
 Deployment in all logical and physical network locations
 Combined use of service with other base services like storage / networking / …
The container platform
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
17 CI/OSB – Mijo Safradin | 2018-12-05

19. Design decision
 Use strongly standardized and well defined
deployment based on cluster blueprint
 Deploy small, customer dedicated clusters
 Deploy clusters at all locations worldwide
 Manage all clusters with already available central
services and tools
 Add optional approved and pre-installed services for
different customer needs
The container platform
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
18 CI/OSB – Mijo Safradin | 2018-12-05

20. Blueprint for a customer container cluster
The container platform
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
19 CI/OSB – Mijo Safradin | 2018-12-05

21. Integration into existing IT factory
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
27
Easy ordering of cluster related functions
Order via Service-Catalog
(new, change, remove, …)
Handover order to automation
Install default services
Connect to networking and storage
Install engine and UCP / Configure base system
Send confirmation and login information to user
Deploy infrastructure in location
Install optional services
CI/OSB – Mijo Safradin | 2018-12-05

22. Integration in the existing IT factory
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
28
Central management of multiple deployments
 Monitoring, statistics and operations for
all clusters based on already existing
service
 Licensing information for reporting to
Docker
 Service-availability calculations for
transparent reporting to customers
CI/OSB – Mijo Safradin | 2018-12-05

23. © Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
29
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outcomes
CI/OSB – Mijo Safradin | 2018-12-05

24. Requirements from various stakeholders
 Repository has to be like the Docker Hub, just on-premises
 Repository has to be accessible from every deployed container
cluster
 Repository has to be located near the container clusters
 Repository has to provide defined images from the Docker Hub
 Repository has to provide compliant and approved base images
The global image repository
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
30 CI/OSB – Mijo Safradin | 2018-12-05

25. Design decision
 Use Docker Trusted Registry as repository
with support from Docker Inc.
 Deploy multiple repositories at different
locations worldwide
 Deploy read-only registries with special
security restrictions in high risk installations
 Make image distribution processes available
between these repositories
The container platform
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
31 CI/OSB – Mijo Safradin | 2018-12-05

26. High risk installations
 Bidirectional replication between registries
 Complete registry
 Namespaces / Organizations
 Repositories
 Staging of well defined images to read-only
registries
 No possibility to push an image
 Defined process to bring images to
special zones
 Unidirectional replication from Docker Hub
 Defined clearing process
The global image repository
Image distribution
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
32
Docker Hub
replication
staging
push/pull
replication
CI/OSB – Mijo Safradin | 2018-12-05

27. Provided base images
 Base images are validated, approved and maintained
regarding
 vulnerability compliance
 open source compliance
 security compliance
 image behaviour compliance
 … more to come
 Configuration of base images meet requirements for
 network configuration
 certificate management and configuration
 user and identity management
 … more to come
The global image repository
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
33 CI/OSB – Mijo Safradin | 2018-12-05

28. How we manage all these
 Identity management
 IT service catalog
 Monitoring and alerting
 Charging
 Automation and orchestration
 …
The global image repository
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
34
LDAP Request access
RestAPI New, Delete, Change
RestAPI Alerting
RestAPI Reporting
RestAPI Configuration
CI/OSB – Mijo Safradin | 2018-12-05

29. © Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
35
Agenda
1. Who we are
2. General information
3. The container platform
4. The global image repository
5. Lessons learned and outlook
CI/OSB – Mijo Safradin | 2018-12-05

30. Current statistics
Lessons learned and outlook
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
36
 ~ 300 starting containers per minute
 Clusters from 6 to 50 nodes
 99.999 % availability on average
 ~ 1000 Repositories
 One new cluster per week
CI/OSB – Mijo Safradin | 2018-12-05

31. Lessons learned
 Actually no central management tool for a
multi-cluster and multi-registry environment
available
 Integration in the existing IT factory is
possible, mostly via the Rest-API
 The registry is a software distribution system
and therefore the most security sensitive
system in the container ecosystem
 Containerized environments needs to be
extended by additional security tools
Lessons learned and outlook
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
37 CI/OSB – Mijo Safradin | 2018-12-05

32. Outlook
 Extension of functionality with various
services
 Tight integration in software defined
networking over all provided services
Lessons learned and outlook
© Robert Bosch GmbH 2018. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
38 CI/OSB – Mijo Safradin | 2018-12-05

33. Q & A

34. Take A Breakout Survey
Access your session and/or workshop surveys for the conference at any time by tapping the Sessions
link on the navigation menu or block on the home screen.
Find the session/workshop you attended and tap on it to view the session details. On this page, you will
find a link to the survey.

35. Come Join Us In San Francisco
April 29-May 2
2019

36. THANK YOU
www.bosch.com