This slide-set was presented at Budapest Microservices Meetup on 21-01-2019. The topic is the cloud native transformation of a traditional bare-metal (later on virtualized) telecommunication product.

1. © 2019 Nokia11
Budapest Microservices Meetup
Gergo Huszty
21-01-2019

2. © 2019 Nokia2
$ whoami
Bio slide
Gergő Huszty
Acting as system architect for cloud
native telco products @Nokia
Geek, electric engineer, father
@gergo_huszty
libesz

3. © 2019 Nokia3 © 2018 Nokia
At home in
North America
where we have been at the heart
of the Bell system for a hundred years
At home in
Europe
where we are rooted
and headquartered
At home in Asia
where we employ more people
than anywhere else
Nokia
At home everywhere
where we operate in 100+ countries
3
100k+ employee
37k R&D
We have the global and local presence to service customers, people and societies through technology.
We are equally at home in Europe, where we are rooted and headquartered; in North America, where we have been at the heart of the Bell system for a century; and in Asia, where we employ more
people than anywhere else.
In fact, we are at home throughout the world, where our Nokia employees work in over 100 different countries.

4. © 2019 Nokia4
Let’s get on the same page
Telco vs IT
• Telco vendor develop the product system/SW and then we deliver to the
customer to operate it
• Multiple product installations (i.e. ~1000x globally)
• Telco products complies to hundreds of standards and specs from ETSI, ANSI,
3GPP, RFC and local laws
• The subject product consists of ~70 virtual machines per product instance (for
3 million users)
• The subject product is developed hundreds at multiple location

5. © 2019 Nokia55
History
Harri Holkeri, the prime minister of Finland, makes the world’s first GSM call 1 July 1991 in Helsinki, Finland.
Some parts of this equipment is still in our Budapest office..
The base of the source code serving this call is still in the product.

6. © 2019 Nokia6
DX200 HW

7. © 2019 Nokia7
What must be part of the product delivery and what isn’t
Proprietary HW form
factor and devices
Proprietary cluster and
HW management
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
Commercial HW
platform
Proprietary
virtualization, cluster
and HW management
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
IT Hardware
Cloud
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
IT Hardware
Cloud
Any Linux
Container
infrastructure
Application coded in
many programming
languages
3rd party platform
services
Traditional Bare metal
Bare metal with
virtualization
IaaS Cloud Cloud native
All starting with an environment with embedded system characteristics (i.e. no native compile toolchain) and with proprietary everything.
All transitions were done with saving the same components, including the Operating System. The cloud native step is the real modernization in all sense.
We have an idea, what legacy means .

8. Complexity?

9. © 2019 Nokia9
VoLTE solution
This is your phone
This is the TAS
Standardized interfaces everywhere. Introducing a new option or flag is hard.

10. This is the task of the product we are talking about. Delivering phone calls’ signaling.

11. © 2019 Nokia11
Quite more than a webserver
Supported TAS interfaces
One reason for microservices transformation: different customer wants different set of features.

12. © 2019 Nokia12
Challenge: Cluster and lifecycle
management

13. © 2019 Nokia13
Remember this? Sorry I lied ;-(
Proprietary HW form
factor and devices
Proprietary cluster and
HW management
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
Commercial HW
platform
Proprietary
virtualization, cluster
and HW management
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
IT Hardware
Cloud
Proprietary OS (32bit,
segmented memory)
Proprietary device
drivers and system
services
Application coded in a
proprietary
programming language
IT Hardware
Cloud
Any Linux
Container
infrastructure
Application coded in
many programming
languages
3rd party platform
services
Traditional Bare metal
Bare metal with
virtualization
IaaS Cloud Cloud native
The current demand is to deliver the container infrastructure to traditional VM based IaaS clouds. So we created one.

14. © 2019 Nokia14
Any NodeAny Node Any Node
O&M NodeO&M NodeInfra Node
Architecture
Nokia TAS container infrastructure
Etcd
SkyDNS
Kubernetes
Networking Kubelet
Helm
Docker
Registry
SwiftTiller
Ansible
Chart repo
LB manager
VNFM Agent
VNFM
App. Images
Application
Heapster
Storage Node
Persistent
volumes
Storage server
Connection between
infra services
Resource provided
to applications.
Not infra or not
SW component
Interacts with all
other services
Infra SW
component
SecretFS
Interacts with
Application
Persistent
volumes
NTP server/client
NTP client
External NTP
server
Note: Services of “Any Node” are present also on Infra and storage.

15. © 2019 Nokia15
Instantiation data handling
Nokia TAS container infrastructure
• All VNF instance has a critical set of immutable data to bootstrap (e.g. VNF name, Target ID)
• This figure shows the data flow
+User supplied
values
Default values
Ren
der
Resource
templates
Ren
der
Guest
data
Container
Infrastructure
Application
Pod
Envvariables
VM
Ansible
Manager
WF Engine
Template
The essential key-values for the application is delivered as helm values, originating from the mandatory management system. The user only interacts with the management system right now.
Later on, with hosted kubernetes environments, the same helm values can be provided directly without changing the application.

16. © 2019 Nokia16
Challenge: Integration and testing

17. © 2019 Nokia17
Before
Open-source vs in-house SW
Open-source
>90%
In-house SW
<10%
Initial steps: port application on top of linux + into containers. Drop supporting components and use 3rd party/open-source alternatives where possible.

18. © 2019 Nokia18
After
Open-source vs in-house SW
Open-source
>90%
In-house SW
<10%
Initial steps: port application on top of linux + into containers. Drop supporting components and use 3rd party/open-source alternatives where possible.

19. © 2019 Nokia19
The big numbers (functional testing)
Integration and testing
• Code
• Multiple git superprojects with 600
submodules total
• 120 container images
• 90 Pod type
• Functional test
• Running on thousands of vCPU
cores
• Thousands of test cases
• Average time to run: 2h
• Other test levels
• Unit test
• Module test
• Performance test
• Robustness
• Security
• E2E verification
• Tooling
• Custom pipelines and schedulers
• Automated container and VM image
build
• Quality gates
• Chaos monkey

20. © 2019 Nokia20
Challenge: Log management

21. © 2019 Nokia21
Logging with containers: What?
Log management from the perspective of a cloud-native service
Logs are the stream of aggregated, time-ordered events collected from the output streams
of all running processes and backing services. […]
-12factor.net
• Classification
• The amount
• The format
• The timestamp
• Log level
• The source
• Footprint
Classification: application log from container, container infrastructure log, audit log for user action, performance metrics, alarms…
Amount: ~big. 600 MB / h
Format: should be common in all containers? Should be discarded is format is wrong? Or should be fixed?
Timestamp: what time is it? In what TZ? Who knows the time? The application or the logging framework?
Log level: from trace to critical
The source: process, functional service, container ID, hosting node ID
Footprint: “all measurement influences the measured object”. Footprint for logging can be huge.

22. © 2019 Nokia22
Centralized logging: the life of a log
Log management from the perspective of a cloud-native service
json-file
syslog
journald
3rd party log driver
collect (pre-process)
forward index & store
query, filter,
visualize
Our stack:
Application follows a log format.
Docker json logs.
Further formatting before kafka.
Elasticsearch for storage and filter.
CLI and WebUI access.

23. © 2019 Nokia23
Before
Logging

24. © 2019 Nokia24
After
Logging
Log viewer and filter is also available with CLI.

25. © 2019 Nokia25
Challenge: Configuration
management

26. © 2019 Nokia26
Configuration management
Telnet
session
Semantic
checker
Co-process
Binary file
User process
Consumer
Netconf+SSH Model
based
validation
Config DB
User process
In-memory
copy
Netconf
adapter
Redis cluster
Redis cluster
Key-Value DB
cluster
Hooks
Keystrokes XML
Before After

27. © 2019 Nokia27
Before
Configuration management

28. © 2019 Nokia28
After
Configuration
Again, not the WebUI is the point. Model based, generated CLI is also available for configuration management.

29. © 2019 Nokia29
Challenge: Service discovery and IPC

30. © 2019 Nokia30
IPC and addressing
• Before
• Proprietary, binary messaging
protocol
• PID type contains node
information
• L2 ethernet frames
• Multi node addressing
• After
• ZeroMQ for intra-container
• Use-case dependent protocol for
inter-container (REST, gRPC,
Thrift, …)
• DNS and etcd for service
discovery
• No 2N and input sync anymore (it
is anyway not a good idea in cloud
Node 1 Node 2
Node 2’
Process A
Process B
Process C
Process C’
The business logic code is not dealing with the low level messaging, so the change in the messaging platform was implemented in the platform code (16 bit ASM, C, PL/M…) in the original
product.

31. © 2019 Nokia31
Challenge: Networking

32. © 2018 Nokia32
Networking requirements
Nokia TAS container infrastructure
• OpenStack, VMware
• Meaning that the solution shall integrate with all L2/L3/L4 security
rules and filters in IaaS clouds
• Real, external IP address in Pod, without NAT!
• Multi interface in Pod
• Support of SCTP

33. © 2018 Nokia33
Networking solution
Nokia TAS container infrastructure
• Invented a new CNI plugin.
• Dynamic IP address ranges
• Static IP address attachments -> no bridging, no packet processing
• Multi-network support
• Plugin delegation
• Pod network requirements are in Podspec annotations
• Static routes (network level) and policy routes (Pod level) are also available
• It is opensource! https://github.com/nokia/danm

34. © 2018 Nokia34
Networking for containers – example configuration
Nokia TAS container infrastructure
Pod on the left has
both external and
internal
connectivity, while
the other has only
internal.
NTAS (VNF)
VM (VNFC)
Pod
oampub0 oam
int-oam
vx_oam
ext-oam
external-oam
VM (VNFC)
Pod
oam
int-oam
vx_oam
xxx Virtual NIC
Data path
IP address assigned
internal-oam
Legend

35. © 2018 Nokia35
Networking for containers – example configuration
Nokia TAS container infrastructure
NTAS (VNF)
VM (VNFC)
Pod
oampub0 oam
int-oam
vx_oam
ext-oam
OpenStack neutron subnets
external-oam
Ipvlan interfaces, created
when Pod starting
VM interfaces
Vxlan interfaces, created in
deployment time
VM (VNFC)
Pod
oam
int-oam
vx_oam
Same external IP address is
assigned to the VM by the
cloud and to the Pod by
NTAS Infra
xxx Virtual NIC
Data path
IP address assigned
internal-oam
Legend

36. © 2018 Nokia36
Internal network management
Nokia TAS container infrastructure
• When the infrastructure is
deployed, network definitions are
populated as CRDs.
• NetWatcher is triggered with CRD
on every Node and creates vxlan
VTEPs for internal networks.
• When any Pod is scheduled, CNI
plugin is invoked.
• It fetches the Pod manifest to get
the annotations
• It reserves IP endpoints (also
stored as CRD).
• With the acquired IP information it
can set the Pod network
namespace
• SvcWatcher watches Pod creation
and updates the Service EndPoints
on demand  updates DNS
NTAS VNF
Infra Node
K8S APIserver
Any Node
Kubelet
Danm
NetWatcher Initial network NS
Pod network NS
Network config
(mapping to host
devices, vxlan IDs, IP
ranges…)
Pod and Service
manifests
SvcWatcher
CNI: Setup Pod
DanmEP
watch
EndPoint
create

37. © 2018 Nokia37
Manifest example for networking
Nokia TAS container infrastructure
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: loadbalancer
spec:
replicas: 1
template:
metadata:
[…]
annotations:
danm.k8s.io/interfaces: |
[
{"network":"oam", "ip":"dynamic"},
{"network":"oampub", "ip":"10.55.28.8/24","routes":{"0.0.0.0/0":"10.55.28.1"}}
]
spec:
containers:
[…]

38. Credits to: Laszlo Janosi, Lajos Hajdu, Gergely Csapo, Attila Esztergalyos

39. © 2019 Nokia39
Copyright and confidentiality
The contents of this document are proprietary and
confidential property of Nokia. This document is
provided subject to confidentiality obligations of the
applicable agreement(s).
This document is intended for use of Nokia’s
customers and collaborators only for the purpose
for which this document is submitted by Nokia. No
part of this document may be reproduced or made
available to the public or to any third party in any
form or means without the prior written permission
of Nokia. This document is to be used by properly
trained professional personnel. Any use of the
contents in this document is limited strictly to the
use(s) specifically created in the applicable
agreement(s) under which the document is
submitted. The user of this document may
voluntarily provide suggestions, comments or other
feedback to Nokia in respect of the contents of this
document ("Feedback").
Such Feedback may be used in Nokia products and
related specifications or other documentation.
Accordingly, if the user of this document gives Nokia
Feedback on the contents of this document, Nokia
may freely use, disclose, reproduce, license,
distribute and otherwise commercialize the
feedback in any Nokia product, technology, service,
specification or other documentation.
Nokia operates a policy of ongoing development.
Nokia reserves the right to make changes and
improvements to any of the products and/or
services described in this document or withdraw this
document at any time without prior notice.
The contents of this document are provided "as is".
Except as required by applicable law, no warranties
of any kind, either express or implied, including, but
not limited to, the implied warranties of
merchantability and fitness for a particular purpose,
are made in relation to the accuracy, reliability or
contents of this document. NOKIA SHALL NOT BE
RESPONSIBLE IN ANY EVENT FOR ERRORS IN THIS
DOCUMENT or for any loss of data or income or any
special, incidental, consequential, indirect or direct
damages howsoever caused, that might arise from
the use of this document or any contents of this
document.
This document and the product(s) it describes
are protected by copyright according to the
applicable laws.
Nokia is a registered trademark of Nokia
Corporation. Other product and company names
mentioned herein may be trademarks or trade
names of their respective owners.