apidays London 2023 - APIs for Smarter Platforms and Business Processes September 13 & 14, 2023 API Metrics matters in APIOps Ludovic Pourrat, API Architect & API Catalyst, Lombard Odier Group ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

1. 14/09/2023
Ludovic Pourrat
Apidays London 2023
API Metrics Matter in APIOps

2. 1 · Ludovic Pourrat
Public
API Architect & API Catalyst | Platform Architect
Lombard Odier Group
Auxiliary Ski Instructor
Ecole Suisse de Ski, Verbier

3. 2 · Ludovic Pourrat
Public
Lombard Odier Group
Private Bank in Switzerland since 1796
Main businesses
• Private Clients
• Asset Management
• Technology for Banking
Technology
• Financial Software Solution Developer
• BPO activity «Bank as a service»

4. 3 · Ludovic Pourrat
Public
Banking Platform
4 functional development streams
• Market, Front, Tax & Operations, Finance
Modular Service oriented solution
• ≈ 800 application components
GX - Large Modernization Initiative started in 2020
• Functional (e.g. SMF, Order Management)
• Technical
• OpenShift, GitOps, APIOps - Kong, Kafka, Vault, Consul, …
• Angular micro-frontends
• Migration Program - SpringBoot and Quarkus
• Service Now, APM & OTEL, OIDC
• Cloud - development environments

5. API Metrics & Discovered Landscape

6. 5 · Ludovic Pourrat
Public
API Metrics Overview
A wide range of available metrics
Performance Metrics
Latency Time
Throughput
Cache Hit Rate
Resource Utilization
Reliability and Availability Metrics
Uptime
Error Code Breakdown & Error Rate
SLA Compliance

7. 6 · Ludovic Pourrat
Public
API Metrics Overview
A wide range of available metrics
Performance Metrics
Latency Time
Throughput
Cache Hit Rate
Resource Utilization
Reliability and Availability Metrics
Uptime
Error Code Breakdown & Error Rate
SLA Compliance
Security Metrics
OWASP Top 10 API Conformance
Authentication & Authorization
Rate Limiting
Developer & Consumer Experience Metrics
Engagement Rate
Adoption Rate
Retention & Churn Rate
Documentation Quality and Coverage Rate
Validation Error Rate

8. 7 · Ludovic Pourrat
Public
API Metrics Overview
A wide range of available metrics
Standard Conformance Metrics
OpenAPI / GraphQL / AsyncAPI / gRPC Conformance
Custom API Standard Conformance
Change Management Metrics
API Version Adoption & Retention Rate
Deprecation Effectiveness
Deprecation Usage
API Version Views & Sunset Compliance
SDK Version Usage
Performance Metrics
Latency Time
Throughput
Cache Hit Rate
Resource Utilization
Reliability and Availability Metrics
Uptime
Error Code Breakdown & Error Rate
SLA Compliance
Security Metrics
OWASP Top 10 API Conformance
Authentication & Authorization
Rate Limiting
Developer & Consumer Experience Metrics
Engagement Rate
Adoption Rate
Retention & Churn Rate
Documentation Quality and Coverage Rate
Validation Error Rate

9. 8 · Ludovic Pourrat
Public
API Metrics Overview
A wide range of available metrics
AI Readiness & Usability Metrics
AI & bot consumption readiness
AI context prompt awareness
Data Governance Metrics
Data Classification & Tagging
Data Privacy
Data Quality
Standard Conformance Metrics
OpenAPI / GraphQL / AsyncAPI / gRPC Conformance
Custom API Standard Conformance
Change Management Metrics
API Version Adoption & Retention Rate
Deprecation Effectiveness
Deprecation Usage
API Version Views & Sunset Compliance
SDK Version Usage
Performance Metrics
Latency Time
Throughput
Cache Hit Rate
Resource Utilization
Reliability and Availability Metrics
Uptime
Error Code Breakdown & Error Rate
SLA Compliance
Security Metrics
OWASP Top 10 API Conformance
Authentication & Authorization
Rate Limiting
Developer & Consumer Experience Metrics
Engagement Rate
Adoption Rate
Retention & Churn Rate
Documentation Quality and Coverage Rate
Validation Error Rate

10. 9 · Ludovic Pourrat
Public
API Metrics Landscape
Identified so far and not exhaustive
Reliability and Availability Metrics
Standard Conformance Metrics
Data Governance Metrics
Performance Metrics
Change Management Metrics
Security Metrics
AI Readiness & Usability Metrics
Developer Experience Metrics
API Gateways
API Gateways
API Portals
OTEL
OTEL

11. API Metrics Applied to APIOps

12. 11 · Ludovic Pourrat
Public
API Metrics applied to APIOps
Design
Develop
Secure
Deploy
Publish
Monitor
Analyze
Developer Experience Metrics
API documentation via descriptions and attached Markdown files
API conformance with the OpenAPI / GraphQL / AsyncAPI standard
API conformance with your own API standard
API Standard Conformance Metrics
API samples explained with a business context
Built-in and custom resources constraints definition and documentation
API backward compatibility checks & API breaking changes
API Security Score
OWASP API Security Top 10 2023 Assessment
Data Governance Metrics
Data classification
Data ownership
Performance Metrics
Resource allocations & limitations
Refine Metrics Collections
Review metrics accuracy
Engage with new metric sensors
Focus & Prioritization
Review the topmost areas of improvement
Data Governance Metrics
Data Privacy
Nothing Yet ?
We discuss that later

13. API Specification Overlay

14. 13 · Ludovic Pourrat
Public
Current State
Solution Driver
But we already got something we
invested upon and that is well
understood. Problem Statement
It looks quite complex to compute all
those metrics and to feed them back into
your APIOps cycles.
Market Readiness
There is not yet a single solution or
provider to gather all the API metrics you
may need.

15. 14 · Ludovic Pourrat
Public
Overlaying API Specification with API Metrics
API Overlay
API Specification
API Extensions
The API overlay definition provide a common definition of how to apply modifications on the API
specification. The API overlay is a method to patch the API specification to apply the API metrics.
3. API Overlay
Allows extending the existing API definitions with custom metadata.
The extensions are the placeholder for the API metrics.
2. API Extensions
We start from what matter the most, so the API specification.
1. API Specification
https://github.com/OAI/Overlay-Specification
But we selected an easier approach by using the overlayed result instead.

16. 15 · Ludovic Pourrat
Public
API Specification Overlayed
openapi: 3.1.0
info:
[…]
termsOfService: www.lombardodier.com/terms-and-conditions.html
contact:
name: Lombard Odier
url: 'https://www.lombardodier.com/home.html'
email: contact@lombardodier.com
x-api-conformance-score: 80
paths:
‘/my-resource’:
x-api-operation-latency: 20ms
[…]
asyncapi: 2.6.0
info:
[…]
termsOfService: www.lombardodier.com/terms-and-conditions.html
contact:
name: Lombard Odier
url: 'https://www.lombardodier.com/home.html'
email: contact@lombardodier.com
x-api-conformance-score: 40
channels:
[…]
graphql: October 2021
info:
[…]
termsOfService: www.lombardodier.com/terms-and-conditions.html
contact:
name: Lombard Odier
url: 'https://www.lombardodier.com/home.html'
email: contact@lombardodier.com
x-api-conformance-score: 60
spec: |
type Query {
[…]
}
openapi: 3.1.0
info:
x-api-conformance-score: 80
paths:
‘/my-resource’:
x-api-operation-latency: 20ms
Overlay with Extensions
1. OpenAPI
2. AsyncAPI
3. GraphQL

17. API Overlay Computation

18. 17 · Ludovic Pourrat
Public
API Specification & API Mock | A Kubernetes Native Automation
The API specification and the API mock definition are handled as
Kubernetes custom resource definitions.
Those can be also created at deployment for upper environments.
API Artefacts as CRDs

19. 18 · Ludovic Pourrat
Public
API Specification & API Mock | A Kubernetes Native Automation
The API specification is automatically deployed into our API
developer portal and an API mock is instantiated within our API
sandbox.
Kubernetes Controllers

20. 19 · Ludovic Pourrat
Public
API Overlay Computation | A Kubernetes Extension
The API overlay is ‘yet another’ Kubernetes custom resource
definition.
API Artefacts as CRDs
We added a Kubernetes controller to compute the API metrics from
the various sources and to generate the API overlay..
Kubernetes Controllers

21. 20 · Ludovic Pourrat
Public
API Overlay Computation | A Kubernetes Extension
Then the overlay could extend the API specification definition and
the API mock definition.
Kubernetes Controllers

22. 21 · Ludovic Pourrat
Public
Finally, the API Developer Portal View
Once collected the API overlay is published in the API developer portal.
The API card can be flipped to display synthetics about the API metrics.
API Card

23. 22 · Ludovic Pourrat
Public
API Overlay Overview
From the source
The API artefacts are collected
from the source in Git.
API Metrics Collection
The API metrics are collected
from the different sources.
Enhance API Developer Portal
The API metrics are visible directly from
the API specification.
API Overlay
The API overlay describe
the collected API metrics.
API Sandbox
The API mocks can be updated
according to the API metrics.
API Artefacts
The API specification and the API
mock definition are handled as CRDs.

24. 23 · Ludovic Pourrat
Public
Recap about the process
1
5
4
3
2
All the organization can leverage and benefits
from the API metrics and ease decision making
The API overlay can be fed into you
APIOps cycles along your iterations
The API overlay and the API extensions are a simple
vehicle to carry-on API metrics
The API metrics are computed from
their sources
The API specifications are the source of truth
and ease any API related analysis
API Metrics matter in APIOps
As an outcome your APIOps cycles are
enriched with valuable API metrics to foster
better developer and consumer
experiences.
API Metrics in the API Portal
We think that is the sweet spot to display
back the collected API metrics.

25. API Metrics | Maturity Matrix

26. 25 · Ludovic Pourrat
Public
Implement fundamental metrics to establish a
baseline.
Focus on performance metrics
Level 1 - Basic
Expand your metrics with cover
usage, utilization, security and start
documenting the APIs
Level 2 - Intermediate
Add the data governance metrics and
gather the reliability and availability
metrics
Level 4 - Expert
Dive deeper by adding conformance,
consumer and developer experience
metrics
Level 3 - Advanced
Continuously monitor and optimize
the metrics. Explore AI
consumption and integration
metrics
Level 5- Optimized
API Metrics | Maturity Matrix
The five levels of maturity

27. 26 · Ludovic Pourrat
Public
Q&A
Wish you some h[API]ness at Apidays !
Better skiers have more fun.