The document discusses the security of microservices within T-Mobile's hybrid cloud architecture, detailing the shift from monolithic systems to microservices with a focus on API access processes. An emphasis is placed on token-based authentication using JWT and OpenID Connect to enhance scalability and performance while ensuring user identity integrity. It also describes the evolution of the T-Mobile app's access mechanism towards a more efficient and secure model through the T-Mobile API Access Process (TAAP).

1. Securing Microservices in Hybrid Cloud
By Komes Subramaniam, Senthil Velusamy
T-Mobile USA

2. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
About us
Komes Subramaniam, Sr. Manager, Digital Services Group
• 18+ years of IT experience
• Responsible for Digital Transformation Architecture, API Products
and Platform Management
• Member of OSS working group @ T-Mobile
• Passion for running and biking
2
Senthil Velusamy, Sr. Member of Technical Staff, Digital Customer Experience
• 18+ years of product and technology experience
• Responsible for strategy, technical and service architecture for Mobile
Apps Ecosystem
• Head of Mobile Center of Excellence, Wi-Fi Alliance board for T-Mobile
• Passion for running and cooking

3. About T-Mobile US
America's Un-carrier: Redefining the way consumers and businesses buy wireless through leading product and service
innovation.
• Based in Bellevue, Washington
• NASDAQ traded public company – TMUS
• Two flagship brands: T-Mobile and MetroPCS
• 21 consecutive quarters with more than one million net adds
• 323 million Americans covered today
• 18 quarters in a row with the fastest download and upload speeds (Ookla and
OpenSignal)
• #1 U.S. wireless carrier in customer care (J.D. Power)
3

4. About T-Mobile US (cont..)
4
T-Mobile Open Source Projects (https://opensource.t-mobile.com)
Digital Transformation
• Monolithic to Microservices
• DevOps You Build It, You Own It
• Telemetry
• CI/CD/CT
• Investment in FTEs
Product & Technology
• World-class product management
organization
• Empowering teams to build and own
customer experience-obsessed products
from design through sustainment
casquatch:
Java abstraction
layer for Cassandra
databases
next-identity:
Highly auditable
blockchain-based,
access
management
solution
keybiner:
Library for encoding
authorized business
functions in an ID
token
t-vault:
Simplified secrets
management
solution
jazz:
A platform for
building serverless
applications

5. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Traditional API /Service Security
• Trust based access to all internal API / Services
• Traditional authentication method uses user session id or cookie
• Stateful : Server keeps track of active authenticated sessions
5
Source image: montgomerynews.com

6. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Securing Microservices
6
Token based Authentication
• Stateless
• Scalable
SPA: Single page Application
• Performant
• Supports Modern SPA
Source image: depositphotos.com

7. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
JWT, ODIC
JWT (JSON Web Token)
• Optionally validated and/or encrypted container
format that is used to securely transfer information
between two parties.
7
{
“type”:”jwt”,
“alg”:”sha256”
}
{
"exp" : “xx600",
"iat" : "1xxx1699266017",
"iss" : "https://xxx.t-mobile.com",
"aud" : “TMOApp",
"nonce" : "NONCE",
"auth_time" : "1481699265",
“AT" : "3285.4326xyabbss521112m4",
"sub" : "U-9645rra1cf7-0xxxf-450c-bdbe-1yyyyy926“
}
PayloadandclaimsHeader
Source: OpenID Connect & OAuth - Demystifying Cloud Identity - Filip Hanik, Sree Tumidi
{
bGmI4ujxjRgc7OKKNATgvXGMADfnFmrwfwxBoTM2g8
8ndi3mGU1i6xo2jr6NQE_..}
Signature
OIDC (Open ID Connect)
• Identity layer on top of OAuth 2.0 flow
• Token issued has access and profile information
• Use the token information to accordingly retrieve resource
data
Source: DOL.WA.gov

8. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile API Access Process (TAAP)
• TAAP developed based on Open ID Connect flow
• Introduced POP (Proof of Possession) for Message Integrity
• ID Tokens user identity and basic information (JWT)
• AT (Access Token) is based on JWT and supports self-validation
PoP Token (Proof of Possession)
• JWT Format
• Claims include:
• Request Payload
• Header Parameters
• Signed by Client’s Private Key
{
"iat": 1xx6yyy776,
"exp": 1aabc435ddee,
"ehts": "authorization; content-type, uri",
"edts":
"109e2ee7xxbbvvefe457bbe3c1065e3
c510744511cnbvg411e956ea836370
d605"
}

9. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
TAAP Call flow
Client Registration
AuthN & AuthZ
Get Token
Get Resources

10. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Use Public cloud for high-volume internet traffic
• All API layers deployed to handle burst traffic
Use Private / On-Premises for sensitive, business-critical
operations
• All events traffic handled on-premises /w legacy system
integration
• Sensitive data filtrations and handling
Ability to scale to the public cloud, you pay for extra computing
power only when needed
Need for Hybrid Cloud
Public
Private
On-Premise

11. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Hybrid Cloud /w TAAP flow
11
IDP: Identity Provider, TAAP: T-Mobile API Access Process
A
P
I
G
A
T
E
W
A
Y
IDPOIDC
AT, ID Token
AT
AT1 µ Service - A
1. Client Application follows TAAP Flow for
obtaining Access Token & ID Token
2. Client sends AT to API Gateway. Gateway
does a cache lookup for ID Token
3. API Gateway sends AT & ID Tokens as part
of µ Service request
4. µ Service-A may require system level access
for µ Service-B. In this case, it follows Client
Credential grant flow to obtain Access Token
(AT1).
5. µ Service-A sends AT1 and ID Token
(Original) to µ Service-B
AT, ID Token
A
P
I
G
A
T
E
W
A
Y
AT1, ID Token
µ Service - B
AT1, ID Token
AT1, Request

12. T-Mobile App : Evolution to TAAP for Secure API Access in
Hybrid Cloud
• T-Mobile App Overview
• Using Opaque Token and its challenges
• Evolution to TAAP based Token and its benefits

13. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App
13
• Android & iOS
• 75M+ installs
• 4.3 rating
V1
Launched in 2015
Opaque Token based
V2
Launched in 2018
TAAP based tokens
T-Mobile app core functions :
1. Self-serve  Retrieve user info
2. Analytics  Submit device info

14. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App: Data in Hybrid Cloud
14
On-Premises:
• Identity provider
• Credentials, Profile, Tokens
Public Cloud:
• AWS
• Device, Plan, Promo etc.
• Data Orchestration for UI
Private Cloud:
• PCF
• Customer Account, Bill,
Lease etc.
(425)-xxx-yyyy
Priya
(425)-xyz-yyyy
Identity Provider Generic data,
Orchestration
(Device, Plan, Promo etc.)
Restricted data
(Profile, Bill, Lease etc.)
1 2 3

15. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V1) : Opaque Token Call-flow
15
OT: Opaque Token, UI: User Interface
Generic data,
Orchestration
(Device, Plan, Promo etc.)
Restricted data
(Profile, Bill, Lease etc.)
Identity Provider Opaque Token:
02.USR.KtDZXYxzleIDLWjOSvVP
Device identifier:
subscriber._0JOBCZpSwWWxyZxXKLCs3t56shu8bd
D77_HfB76KUg=
1. Submit Credentials
2. Opaque Token, Device Identifier
3. Request app data (OT)
10. App data for UI display
4. Validate
(OT)
5. Valid
7. Validate
(OT)
8. Valid
6. Request account info (OT)
9. Account data

16. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V1) : Opaque Token Challenges
16
Opaque Token requires validation with a central system and adds latency to critical path of user experience
• 100 – 200ms latency, up to 6 requests to IDP per login session
Tokens when stolen can be used to replay until its validity
OT: Opaque Token, IDP: Identify Provider

17. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V2) : TAAP Call-flow
17
JWK: JSON Web key, TAAP: T-Mobile API Access Process, POP: Proof of Possession
Generic data,
Orchestration
(Device, Plan, Promo etc.)
Restricted data
(Profile, Bill, Lease etc.)
Identity Provider
1. Submit Credentials
2. User ID token, Device ID Token
3. App data (User ID Token,
POP)
6. App data for UI display
4. Account info (User ID Token)
5. Account data
JWK
• JWK: Retrieve Identity Provider public key
• Each domain receiving the ID token self-
validates

18. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V2) : TAAP Tokens
18
{
"iss": "https://xxxxx.t-mobile.com",
"network": {
id: “35679xxx0011980”
phone: “425xxxyyy”},
"device": {
"cnf": "-----BEGIN PUBLIC KEY-----
nMIIBIjANBgkqhkiG9w0BAQE
FAAOCAQ8AMIIBCgKCAQEA5
oaIEnqExSKXK/J7mvgx...........
n-----END PUBLIC KEY-----"
},
"exp": 1aabc435ddee,
"aud": “TMOApp",
"iat": 1xx6yyy776
}
Device ID Token User ID Token Proof Of Possession (POP)
{
"iat": 1xx6yyy776,
"exp": 1aabc435ddee,
"iss": "https://xxxxx.t-mobile.com",
"aud": "TMOApp",
"auth_time": 1536870785,
"sub": "U-xxxxx-yyyyyy-zzzzzzzz",
"amr": [ "password“ ],
"cnf": "-----BEGIN PUBLIC KEY-----
nMIIBIjANBgkqhkiG9w0BAQE
FAAOCAQ8AMIIBCgKCAQEA5
oaIEnqExSKXK/J7mvgx...........
.........n-----END PUBLIC KEY--
---",
"usn": “abcabcabcabcabc",
"ent": {
"acct": [
{
"r": “XX",
"tst": “YY",
"line_count": 5,
"lines": [
{
"phnum": “1234567890",
"r": “Z"
}
]
{
"iat": 1xx6yyy776,
"exp": 1aabc435ddee,
"ehts": "a11ept;x-tmo-device-os;x-
txx-yym;x-b3-paxxyyyzznid;x-
aao-clixxt-name;x-tmo-model;x-
nno-cvcnt-version;x-dat;x-tmo-
xnbvce-os-
version;authorization;x-tmo-
oem-id;x-b3-spanid;x-tmo-post-
sequence-number;x-b3-
traceid;content-length;content-
type",
"edts":
"109e2ee7xxbbvvefe457bbe3c
1065e3c510744511cnbvg411e
956ea836370d605"
}

19. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V2) : How TAAP is used?
19
POP: Proof of Possession, UIT: User ID Token, DIT: Device ID Token, IDP: Identity Provider
Header ()
Iss: IDP
Sub: Identity
KEY: cPub
Entitlement:…
Signature (iPri)
UserIDToken
Header ()
Iss: Client
Sub: Identity
Headers: …..
Hash: xxxxx
Signature
(cPri)
POP
Self-serve: User ID Token + POP
1. Validate User ID Token signature (JWK, iPub)
2. Retrieve cPub from User ID Token
3. Validate POP signature (cPub)
4. Validate POP hash (message integrity)
5. Use Entitlement info (Role, lines etc)
Header ()
Iss: IDP
Sub: Identity
Network: …
KEY: cPub
Signature (iPri)
DeviceIDToken
Header ()
Iss: Client
Sub: Identity
Headers: …..
Hash: xxxxx
Signature
(cPri)
POP
Analytics: Device ID Token + POP
1. Validate Device ID Token signature (JWK, iPub)
2. Retrieve cPub from Device ID Token
3. Validate POP signature (cPub)
4. Validate POP hash (message integrity)
5. Use Network identifiers (Hardware ID, Phone
Number)

20. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
T-Mobile App (V2) : TAAP Benefits
20
TAAP: T-Mobile API Application Process, IDP: Identity Provider
• Improved Performance: Remove dependency on
centralized system - IDP not in critical path of experience
• Reduced load to downstream (e.g., no downstream call
for unsupported account, plan types)
Sample metrics shown below for T-Mobile app load time:
Opaque Token : ~3.2s User ID Token : ~2.5s
~20% improvement in app load time
• Flexibility to integrate with different IDPs
• Selfheal, no coordinated key rotations
• Improves security: Can’t replay tokens

21. > Stay Connected.
#springone@s1p
@komethagan, @mvsenthil