Keystone supports four different types of tokens, UUID, PKI, PKIZ, and Fernet. Let’s take a deep dive into:
Understanding token formats
Pros and Cons of each format in Production
Performance across multiple data centers
Token revocation workflow for each of the formats
Horizon usage of the different token types
We previously deployed UUID and PKI in Production and are now moving towards the latest format, Fernet. We would like to share our lessons learned with different formats and help you decide on which format is suitable for your cloud.
(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...
Deep Dive into Keystone Tokens and Lessons Learned
1. Deep Dive into Keystone
Tokens and Lessons Learned
Priti Desai & Brad Pokorny
2. Who are we?
Priti Desai
Advisory Software Engineer, IBM
Brad Pokorny
Principal Software Engineer, Symantec
Deep Dive into Keystone Tokens and Lessons Learned
6. UUID
• Simplest and Most Light Weight
• Version 4 UUID
• Configuration in keystone.conf :
[token]
provider = keystone.token.providers.uuid.Provider
Deep Dive into Keystone Tokens and Lessons Learned
7. UUID – Token Generation Workflow
Keystone
KVS
Request Token with:
• User Name
• Password
• Project Name
Identity Resources Assignment Catalog
User Validation
Retrieves User ID
Token
Project Validation
Retrieves Project ID and
Domain ID
Retrieves Roles for this
User on the Project or
Domain
Returns Failure if the
User does not have
any Role
Retrieves Services and
Endpoints for all the
services
Bundles Identity, Resource,
Assignment, and Catalog
information into Token Payload
Creates Token ID :
uuid.uuid4().hex
Store them in SQL/KVS:
• Token ID
• Expiration
• Valid
• User ID
• Extra
Token Generation Workflow
9. UUID – Keystone Token Validation Workflow
Parse Token and Retrieve
Metadata
Validate Token with:
GET v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Retrieves Token
payload from
token backend
KVS/SQL
Read cached token reference and parse:
• User ID
• Project ID
• Audit ID
• Token Expiry
Token
KVS
Valid?
Current Time <
Expiry Time
Token Not Found
Token Not Found
Is
Revoked?
Token Not Found
HTTP/1.1 200 OK
Yes
No
No
No
Yes
Yes
Check if a token matches
any revocation events
Check if a token is
expired, current time is
calculated in UTC
Token Validation Workflow
10. UUID – Keystone Token Revocation Workflow
Revoke by Audit ID
Revoke Token with:
DELETE v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Before revoking
a token, validate
it based on
Token Validation
Workflow
Validate
X-Subject-Token
Audit
ID?
Revoke by Token Expiry
Create Revoke Event with:
Audit ID
Revoke At
Issued Before
Prune Expired Events
Set valid to
False
Yes
No
Filter existing revocation
events based on Revoke At
Sample Revocation Event:
{
"audit_id": "HVvI0d-cTD21yatAfQc4IQ",
"issued_before”: "2015-10-24T21:20:45.000000Z"
},
Token Revocation Workflow
Create Revoke Event with:
User ID
Project ID
Revoke At
Issued Before
Token Expiry
Token
KVS
11. UUID Across Multiple Data Centers
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
MySQL Replication
(Database is always in sync)
LDAP Replication
(Directory Tree is always in sync)
Tokens
KVS
UUID Tokens
Tokens
KVS
UUID Tokens
Keystone
UUID - Multiple Data Centers
Nova
Keystone Middleware
Keystone
Nova
Keystone Middleware
US-EASTUS-WEST
Request
Token
UUID Token
Token Validation
VM Instance
Token Validation
Token Not Found
Token Found Token Not Found
12. Pros and cons
• Pros
– Simplest and Smallest Token Format
– Recommended for Simple OpenStack Deployment
• Cons
– Persistent Token Format
– Token validation can only be done by Identity service
– Not feasible for multiple OpenStack deployments
Deep Dive into Keystone Tokens and Lessons Learned
14. PKI
• Cryptographically Encrypted
Signed Document using
X509 Standards
• CMS
• Converted to custom URL-
Safe format
• Compressed PKI
• Prefixed with “PKIZ”
Deep Dive into Keystone Tokens and Lessons Learned
PKIZ
15. PKI/PKIZ Configuration - Certificates
• Signing Key (signing_key.pem) :
• Generate private key in PEM format
• Signing Certificate (signing_cert.pem) :
• Generate CSR using Signing Key
• Submit CSR to CA
• Receive Certificate from CA
• Certificate Authority Certificate (ca.pem)
Deep Dive into Keystone Tokens and Lessons Learned
16. PKI/PKIZ Configuration
• Configuration in keystone.conf :
[token]
provider = keystone.token.providers.[pki|pkiz].Provider
[signing]
certfile = /etc/keystone/ssl/certs/signing_cert.pem
keyfile = /etc/keystone/ssl/private/signing_key.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
Deep Dive into Keystone Tokens and Lessons Learned
17. PKI/PKIZ – Token Generation Workflow
Validate Identity, Resource, and Assignment
Request Token with:
• User Name
• Password
• Project Name
Token Generation Workflow
Create JSON Token Payload
Sign JSON Payload with Signing Key and Signing Certificate
openssl cms –sign –outform PEM
Convert it to UTF-8
Convert CMS Signed Token in PEM
format to custom URL Safe format:
• “/” replaced with “-”
• Deleted: “n”, “----BEGIN
CMS----”,“----END CMS----
”
Compress using zlib
Convert it to UTF-8
Base64 URL Safe
Append Prefix PKIZ
PKI PKIZ
Store Token into SQL/KVS
20. PKI/PKIZ – Token Validation Workflow
Parse Token and Retrieve
Metadata
Validate Token with:
GET v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Retrieves Token
reference from
token backend
KVS/SQL
Read cached token reference and parse:
• User ID
• Project ID
• Audit ID
• Token Expiry
Token
KVS
Valid?
Current Time
< Expiry Time
Token Not Found
Token Not Found
Is
Revoked?
Token Not Found
HTTP/1.1 200 OK
Yes
No
No
No
Yes
Yes
Check if a token matches any
revocation events
Check if a token is expired,
current time is calculated in
UTC
Token Validation Workflow
Unique ID of
X-Subject-Token
Hash PKI Token
with the pre-
configured hashing
algorithm
21. PKI/PKIZ – Keystone Token Revocation Workflow
Revoke by Audit ID
Revoke Token with:
DELETE v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Before revoking
a token, validate
it based on
Token Validation
Workflow
Validate
X-Subject-Token
Audit
ID?
Revoke by Token Expiry
Create Revoke Event with:
Audit ID
Revoke At
Issued Before
Prune Expired Events
Set valid to
False
Yes
No
Filter existing revocation
events based on Revoke At
Sample Revocation Event:
{
"audit_id": "HVvI0d-cTD21yatAfQc4IQ",
"issued_before”: "2015-10-24T21:20:45.000000Z"
},
Token Revocation Workflow
Create Revoke Event with:
User ID
Project ID
Revoke At
Issued Before
Token Expiry
Token
KVS
22. PKI/PKIZ - Across Multiple Data Centers
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
MySQL Replication
(Database is always in sync)
LDAP Replication
(Directory Tree is always in sync)
Tokens
KVS
PKI/PKIZ
Tokens
Tokens
KVS
PKI/PKIZ
Tokens
Keystone
PKI/PKIZ - Multiple Data Centers
Nova
Keystone Middleware
Keystone
Nova
Keystone Middleware
US-EASTUS-WEST
Request Token
PKI/PKIZ Token
Token Validation
VM Instance
Token Validation
VM Instance
23. Pros and Cons
PKI
• Pros
– Token validation without a request to
Keystone
• Cons
– Larger than standard HTTP Header Size
– Complex configuration
– base64 –d <pki_token
– Not truly feasible for multiple OpenStack
Deployments
PKIZ
• Pros
– Token validation without a request
to Keystone
• Cons
– Still Larger than standard HTTP
Header Size
– Similar to PKI
Deep Dive into Keystone Tokens and Lessons Learned
25. Fernet
• Cryptographic Authentication Method – Fernet
• Symmetric Key Encryption
• Fernet Keys stored in /etc/keystone/fernet-keys/
– Encrypted with Primary Fernet Key
– Decrypted with a list of Fernet Keys
Deep Dive into Keystone Tokens and Lessons Learned
26. • Configuration in keystone.conf :
[token]
provider = keystone.token.providers.fernet.Provider
[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/
max_active_keys = <number of keys> # default is 3
Deep Dive into Keystone Tokens and Lessons Learned
Fernet Configuration
28. Fernet Keys
• Fernet Key File Name - Integers starting from 0
• ls /etc/keystone/fernet-keys => 0 1 2 3 4
• Type 1: Primary Key
– Encrypt and Decrypt
– Key file named with the highest index
• Type 2: Secondary Key
– Only Decrypt
– Lowest Index < Secondary Key File Name < Highest Index
• Type 3: Staged Key
– Decrypt and Next In Line to become Primary Key
– Key file named with lowest index (of 0)
Deep Dive into Keystone Tokens and Lessons Learned
30. Fernet – Token Generation Workflow
Token Generation Workflow
HMACFernet Token Version Current Timestamp IV Cipher Text
Token Payload:
Version
User ID
Methods
Project ID
Expiry Time
Audit ID
Padding
Encrypted using Encrypting Key
Signed using Signing Key
31. Sample Fernet Token in SQL Backend
Deep Dive into Keystone Tokens and Lessons Learned
gAAAAABWLUzy0dxSNo2--K-
3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNA-
SwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8
f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k
32. Fernet – Keystone Token Validation Workflow
Determine the Version from the Token Payload
Validate Token with:
GET v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Re-inflate token
with “=” and return
token with correct
padding
Version: Fixed Versioning by Keystone:
• Unscoped Payload : 0
• Domain Scoped Payload : 1
• Project Scoped Payload : 2
Restore
Padding
Is
Revoked?
Token Not Found
HTTP/1.1 200 OK
No
No
Yes
Yes
Check if a token matches
any revocation events
Check if a token is
expired, current time is
calculated in UTC
Token Validation Workflow
Decrypt using Fernet Keys to retrieve Token Payload
Disassemble payload to
determine validation fields
For Project Scoped Token:
• User ID Project ID
• Methods Token
Expiry
• Audit ID
Current Time <
Expiry Time
Token Not Found
No
33. Fernet – Keystone Token Revocation Workflow
Revoke by Audit ID
Revoke Token with:
DELETE v3/auth/tokens
• X-Subject-Token
• X-Auth-Token
Before revoking
a token, validate
it based on
Token Validation
Workflow
Validate
X-Subject-Token
Audit
ID?
Revoke by Token Expiry
Create Revoke Event with:
Audit ID
Revoke At
Issued Before
Prune Expired Events
Set valid to
False
Yes
No
Filter existing revocation
events based on Revoke At
Sample Revocation Event:
{
"audit_id": "HVvI0d-cTD21yatAfQc4IQ",
"issued_before”: "2015-10-24T21:20:45.000000Z"
},
Token Revocation Workflow
Create Revoke Event with:
User ID
Project ID
Revoke At
Issued Before
Token Expiry
Token
KVS
34. Fernet - Across Multiple Data Centers
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
Users
Groups
Domains
Projects
Roles
Catalog
Assignments
MySQL Replication
(Database is always in sync)
LDAP Replication
(Directory Tree is always in sync)
Keystone
Fernet - Multiple Data Centers
Nova
Keystone Middleware
Keystone
Nova
Keystone Middleware
US-EASTUS-WEST
Request
Token
Fernet Token
Token Validation
VM Instance
Token Validation
VM Instance
Validate Fernet Token Validate Fernet Token
35. Pros and cons
• Pros
– No persistence
– Reasonable Token Size
– Multiple Data Center
• Cons
– Token validation impacted by the number of revocation events
Deep Dive into Keystone Tokens and Lessons Learned
38. What token format should we configure in our
OpenStack Deployment?
Fernet for Multiple OpenStack Deployments
with minimal Revocation Events
Deep Dive into Keystone Tokens and Lessons Learned
40. How horizon uses tokens
Deep Dive into Keystone Tokens and Lessons Learned
• Tokens for each logged in user
• Unscoped token and project scoped token
• Token reuse
• Reduced transaction load on Keystone
• Stored in the session
• Configurable token storage methods
• Local memory cache
• Cookie backend
• Memcache
• Database
• Cached Database
41. Cookie backend
Deep Dive into Keystone Tokens and Lessons Learned
• Currently the devstack default
• Token stored in browser cookie
• Secure cookies in production, use https
• CSRF_COOKIE_SECURE = True
• SESSION_COOKIE_SECURE = True
• http://docs.openstack.org/developer/horizon/topics/settings.html
• http://docs.openstack.org/security-guide/dashboard/cookies.html
• Highly scalable
• The dreaded boot back to login
42. Cookie backend
Deep Dive into Keystone Tokens and Lessons Learned
• The dreaded boot back to login
• Now sign in…
43. Cookie backend
Deep Dive into Keystone Tokens and Lessons Learned
• And you see…
• Cookie overflow!
44. memcache backend
Deep Dive into Keystone Tokens and Lessons Learned
• Allows storage of larger token sizes
• Tokens stored on server side
• Requires memcached
• Can be used with backing DB
• http://docs.openstack.org/developer/horizon/topics/deployment.html
45. Token hashing
Deep Dive into Keystone Tokens and Lessons Learned
• Hashed in Django OpenStack Auth (DOA)
• Keeps stored token data small
• Currently not working for PKI tokens
• New config in Liberty to disable
• OPENSTACK_TOKEN_HASH_ENABLED
• PKI - Will increase memcache storage requirement
46. Multiregion and tokens
Deep Dive into Keystone Tokens and Lessons Learned
• Service regions vs. Authentication regions
• Service regions in Keystone catalog
• Auth regions specified in AVAILABLE_REGIONS
• UUID, PKI, and PKIZ Tokens don’t work across auth regions
• Token replication is infeasible
• But Fernet tokens work between Authentication regions!
Service Region Authentication Region
47. Horizon and Fernet
Deep Dive into Keystone Tokens and Lessons Learned
• Yes, Fernet tokens work with Horizon
• Liberty and beyond – No patches necessary
• Kilo – Needs a patch for DOA
• https://review.openstack.org/#/c/169994/
49. V3 domains
Deep Dive into Keystone Tokens and Lessons Learned
• Requires changes in Django OpenStack Auth and Horizon
• Planned for Mitaka
• Info on usage (a bit out of date):
• http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3
• Domains patches:
• https://review.openstack.org/#/c/148082/
• https://review.openstack.org/#/c/141153/
• https://review.openstack.org/#/c/196328/
50. Will fernet tokens solve all our problems?
Deep Dive into Keystone Tokens and Lessons Learned
• Smaller token size
• No persistence for tokens
• Seamless authentication across regions
• Performance issues with token revocation
54. What is an Openstack Token?
Deep Dive into Keystone Tokens and Lessons Learned
Key to OpenStack Cloud
55. How can I generate a token?
Deep Dive into Keystone Tokens and Lessons Learned
Keystone
curl -s POST https://keystone.com/v3/auth/tokens
"auth": {
"identity": {
"methods": [
"password"
],
"password": {
"user": {
”domain": {
”name": ”MyDomain.com”
},
”name": ”PritiDesai",
"password": "secretsecret”
}
56. How can I generate a token?
Deep Dive into Keystone Tokens and Lessons Learned
Keystone
curl -s POST https://keystone.com/v3/auth/tokens
"auth": {
"identity": {
"methods": [
”token"
],
”token": {
”id": ”e8079ab…”
}
}
59. Identity Token Path
User OpenStack Service
Step 1: Obtain unscoped token with credentials
POST v3/auth/tokens
Keystone
Step 2: Discover projects you have access to
GET v3/users/<user_id>/projects
Step 3: Obtain project scoped token either with your
credentials or unscoped token from step 1.
Step 4: Invoke the target service by sending
requests to endpoints in token from step 3
Step 5: Validate roles and access metadata in token
with Keystone service or Keystone Middleware
Step 6: Serve API
request
Step 7: Return response
60. Token Creation
0
10
20
30
40
50
60
70
80
90
UUID PKI PKIZ Fernet
Time Per Request
Time Per
Request
Deep Dive into Keystone Tokens and Lessons Learned
0
5
10
15
20
UUID PKI PKIZ Fernet
Requests Per Sec
Requests Per
Sec
61. Token Validation
0
2
4
6
8
10
12
14
UUID PKI PKIZ Fernet
Time Per Request
Time Per
Request
Deep Dive into Keystone Tokens and Lessons Learned
0
20
40
60
80
100
UUID PKI PKIZ Fernet
Requests Per Sec
Requests Per
Sec
Identity:
Checks if User exist in User Domain
Check if User is enabled
Retrieves User ID
Matches Password
Resource:
Checks if Domain or Project exist
Check if Domain or Project is enabled
Retrieves Project ID and Domain ID
Catalog:
Retrieves Services associated with User’s Project
Retrieves the list of endpoints for all the services
Identity:
Checks if User exist in User Domain
Check if User is enabled
Retrieves User ID
Matches Password
Resource:
Checks if Domain or Project exist
Check if Domain or Project is enabled
Retrieves Project ID and Domain ID
Catalog:
Retrieves Services associated with User’s Project
Retrieves the list of endpoints for all the services
Identity:
Checks if User exist in User Domain
Check if User is enabled
Retrieves User ID
Matches Password
Resource:
Checks if Domain or Project exist
Check if Domain or Project is enabled
Retrieves Project ID and Domain ID
Catalog:
Retrieves Services associated with User’s Project
Retrieves the list of endpoints for all the services
I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
When logging in, Horizon gets a keystone token using the user’s credentials. This is an important aspect of security in Horizon, as Horizon doesn’t require a service credential, minimizing the impact of an attacker compromising Horizon itself.
First get an unscoped token, which is used to get scoped tokens to projects when the user switches projects.
The storage method for tokens is configurable and can have a large impact depending on the complexity of your cloud. The configurable methods are:
We are using the memcache backend.
I’ll talk more just about the cookie backend and Memcache backend.
The cookie backend has some very strong advantages, and it’s currently the devstack default.
In this case, tokens are stored in a browser cookie on the client side. If using the cookie backend in production, it’s important to configure https connections to Horizon and also configure security for the tokens. Otherwise, someone could recover a token while sniffing the network.
The cookie backend is highly scalable, as token storage is all done on the client side.
However, the cookie backend can’t be used if you have many endpoints in the keystone catalog.
Cookie sizes for most browsers are about 4KB. When the token takes up a lot of that space, you’ll log into Horizon, sign in…
..And see this!
Using the memcache backend resolves cookie overflow issues. We currently use it at Symantec.
Token Hashing has been used in the past to reduce the impact of large token sizes.
We currently use token hashing with Kilo Horizon, which works fine with Hashing, but you could have issues if using the master branch.
For UUID, PKI, and PKIZ, Tokens won’t work across auth regions.
This will be a benefit of Fernet tokens, as they will allow authentication between multiple keystone instances.
Fernet tokens work with Horizon
This is somewhat out of scope for the different token types.
I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
This is just a code example for the Liberty code that disables token hashing.