This document discusses the need for reconstructing API specifications from runtime traffic without source code access. It provides an overview of existing open source solutions and their limitations. It then introduces APIClarity, an open source tool that monitors API traffic using a service mesh to automatically generate OpenAPI specifications, allows users to review and approve specifications, and detects differences between specifications and runtime behavior. The document demonstrates APIClarity using a sample application and discusses plans for additional features.

1. © 2021 Cisco and/or its affiliates. All rights reserved.
Open Source API Observability Tool
Zohar Kaufman, Alexei Kravtsov
Sep 2021

2. © 2021 Cisco and/or its affiliates. All rights reserved.
Agenda
Why do we need API
Specs reconstruction ?
Possible open source
packages survey
Introducing APIClarity
(design, architecture)
It’s Demo Time Roadmap & closing
comments
Q&A

3. © 2021 Cisco and/or its affiliates. All rights reserved.
Cloud services and microservices architecture gaining popularity
Typically using OpenAPI Specification:
Standard, language-agnostic
interface
Allows both humans and
computers to discover and
understand the capabilities of
a service
No access to source code or
other documentation is needed
Interface files for describing, producing, consuming, and visualizing RESTful web services
Allows parallel development of client and server applications using generated code and mocks
OpenAPI Specs

4. © 2021 Cisco and/or its affiliates. All rights reserved.
API Specs Challenges
Not all applications have their OpenAPI specification available (legacy,
external)
Designed API vs Runtime API
Deprecated APIs
(Zombie APIs)
Undocumented
APIs (Shadow
APIs)
Breaking
changes
Differences
over time
Obtaining OpenAPI Spec without code instrumentation or modifying existing
workloads

5. © 2021 Cisco and/or its affiliates. All rights reserved.
API Specs Challenges - Gartner
Every connected mobile, modern
web or cloud-hosted application uses
and exposes APIs:
Easy to expose
but difficult to
defend
Used to access
data and to call
application functionality
Large and growing attack surface, leading
to a growing number of publicized API
attacks and breaches

6. © 2021 Cisco and/or its affiliates. All rights reserved.
Existing Open Source Solutions
Similar products:
Akita, Imvision, Salt
Open sources:
Optic, Avantation, har2openapi, Response2Schema,
InducOapi, SwagDefGen
Community tool (free but not open source):
API Shark, SwaggerHub
List of Tools:
Open API Tools, API Specification Toolbox

7. © 2021 Cisco and/or its affiliates. All rights reserved.
Open API Tools https://openapi.tools/
Tool Types
• Auto Generators: Tools that will take your code and turn it into an OpenAPI Specification document
• Converters: Various tools to convert to and from OpenAPI and other API description formats
• Data Validators: Check to see if API requests and responses are lining up with the API description
• Description Validators: Check your API description to see if it is valid OpenAPI
• Documentation: Render API Description as HTML (or maybe a PDF) so slightly less technical people can figure out how to work with the API
• DSL: Writing YAML by hand is no fun, and maybe you don't want a GUI, so use a Domain Specific Language to write OpenAPI in your language of choice
• GUI Editors: Visual editors help you design APIs without needing to memorize the entire OpenAPI specification
• Learning: Whether you're trying to get documentation for a third-party API based on traffic or are trying to switch to design-first at an organization with no
OpenAPI at all, learning can help you move your API spec forward and keep it up to date
• Miscellaneous: Anything else that does stuff with OpenAPI but hasn't quite got enough to warrant its own category
• Mock Servers: Fake servers that take description document as input, then route incoming HTTP requests to example responses or dynamically generates
examples
• Parsers: Loads and read OpenAPI descriptions, so you can work with them programmatically
• SDK Generators: Generate code to give to consumers, to help them avoid interacting at a HTTP level
• Security: By poking around your OpenAPI description, some tools can look out for attack vectors you might not have noticed
• Server Implementations: Easily create and implement resources and routes for your APIs
• Testing: Quickly execute API requests and validate responses on the fly through command line or GUI interfaces
• Text Editors: Text editors give you visual feedback whilst you write OpenAPI, so you can see what docs might look like

8. © 2021 Cisco and/or its affiliates. All rights reserved.
API Specification Toolbox http://api.specificationtoolbox.com/services/

9. © 2021 Cisco and/or its affiliates. All rights reserved.
Optic
Open Source tool
Helps developers to document, review and approve API
changes prior to deploying them
Language agnostic, works with any REST API
Observes development traffic and learns your API’s
behavior
Detects API changes by diffing traffic against the current
specification
Adds an accurate API changelog to every Pull Request
Mechanism to manually review and update the specification
Not designed to monitor multi service environments

10. © 2021 Cisco and/or its affiliates. All rights reserved.
SwaggerHub
No need for a setup – generate API
traffic from Web UI
Records API traffic and converts it to
OAS using SwaggerHub
Not an open source
No integration with runtime environments

11. © 2021 Cisco and/or its affiliates. All rights reserved.
API Shark
Live monitoring of multi-service environments
OAS spec learning from runtime traffic
Automatic parameter detection
Seamless
No Spec Diff detection
Not an open source
No review option of the learned spec

12. © 2021 Cisco and/or its affiliates. All rights reserved.
Live monitoring of multi-service environments
Spec Diff detection
Not free
Not an open source
Imvision API
Security
OAS spec learning from runtime traffic
Mechanism to manually review and update the
specification

13. © 2021 Cisco and/or its affiliates. All rights reserved.
No code changes are needed to any of your App
Construct the OpenAPI specification by observing the
API traffic
UI dashboard to audit and monitor the API findings
Alert the user on any difference between the
approved API specification and the one that is
observed in runtime, detects shadow & zombie APIs
Introducing
APIClarity
Capture all API traffic in an existing environment
using a service-mesh framework
Allow the User to upload OpenAPI spec, review,
modify and approve generated OpenAPI specs

14. © 2021 Cisco and/or its affiliates. All rights reserved.
APIClarity: High-Level Architecture
Users
Istio
Client App
Envoy Proxy
Fronted App
Envoy Proxy
Backend App
Envoy Proxy
POST/login
GET/clients
Mirror API
traffic from
Proxies
OpenAPI Spec Engine
UI
• API events
• Learned Specs
• API Diff
API spec
user review
Frontend
Pod
Client
Pod
Backend
Pod
K8s
Cluster

15. © 2021 Cisco and/or its affiliates. All rights reserved.
Spec Reconstruction Features
Parameter detection
(Path, query, header, cookie)
Object references
Security definitions
File transfer

16. © 2021 Cisco and/or its affiliates. All rights reserved.
Demo Environment
K8s cluster Istio service
mesh
APIClarity
installed
Sock-shop
demo app
Setup

17. © 2021 Cisco and/or its affiliates. All rights reserved.
Demo Flow
1 2 3 4 5 6 7
APIClarity
deployment overview
(clone, build, deploy)
Visibility of runtime
API events
Event filtering, hit
count graphs
OAS learning –
generating traffic
using the sock-shop
demo app
Review of
automatically
generated OAS
learned from the
traffic
Approve the
reviewed spec and
see in Swagger UI
User provided spec
Monitor and observe
whether the API events
correspond to set specs
API event diffs, Shadow
APIs, Zombie APIs

18. © 2021 Cisco and/or its affiliates. All rights reserved.
It’s Demo
Time
© 2021 Cisco and/or its affiliates. All rights reserved.

19. © 2021 Cisco and/or its affiliates. All rights reserved.
API development and security aspects of APIClarity
Analyze spec for security
issues and best practices
Run API fuzzing tests
using the spec
Generate Client and
Server code
Generate Client and
Server mocks
Documentation
Knowing the
API Spec
is the first
step to
identifying
your API
risks Mock
generation
Documentation
Fuzzing
tests
OAS issue
analysis
Security Development
Code
generation

20. © 2021 Cisco and/or its affiliates. All rights reserved.
Roadmap
OAS 3.0,
GraphQL, gRPC
Integration with
additional
environments
(Browsers,
Postman, API
gateways, etc.)
Support spec
schema review
in UI
System
settings
User
management

21. © 2021 Cisco and/or its affiliates. All rights reserved.
Questions?
© 2021 Cisco and/or its affiliates. All rights reserved.