Cloud Foundry API for Fun and Ops

EXPLORINGTHE CLOUD FOUNDRY API
FOR FUN AND OPS
Josh Kruck 
professional doer of things
github.com/krujos
@krujos

WHO AM I?
• Today
• pivotal customer success
• cf cli plugin author
• service broker author
• lots of little utilities for getting shit done
• Past
• a lot of storage and application management
• Why
• use software to make work more fun and be happier

https://lynxbat.ﬁles.wordpress.com/2012/06/api.jpg

done. good game. thanks everyone!
http://www.wired.com/2013/11/pivotal-one/

SOME IDEAS
• apps (dashboard, provisioner, automation, crypt keeper)
• cli plugins
• scripts (why aren’t you writing a cli plugin?)
• A one time thing? (jq will change your life)

SCRIPTS ARE AN ANTI PATTERN
CLI plugins FTW!

What are the things that got API on
them?

Service Brokers(more of a spec really)
Lots of other people talk about service brokers and how to author them, i won’t be covering them.

UAA(User Account and Authentication Server)
This is where we go to login. If you’re writing CLI plugins and scripts, it’s a non issue, but for any kind of app we’ll have to deal with it.

Firehose(the tattletale)
the emitter of most of the things

Collector(older tattletale)
emits a lot of the readable things

Cloud Controller(the brains)
The CC is where we got to listen and act.

but wait!
Router(eventually)
A feature that’s coming down the pipe is “router services”. While I don’t have anything to say on this today it is something to be aware of. It will allow us to do some inline
things at the router that we previously had to insert between our LB’s and routers.

WHEN/WHY DO WE INTERACT WITH
UAA?
• When we’re writing an application and DON’T want to inherit the
credentials of the user
• Dashboards, system automation and on boarding
System automation examples:

Clean up orphaned services

Check for rouge users

Reports (new URL’s for nessus, apps pushed in the last day, crash summaries… whatevs).

WHEN/WHY DO WE INTERACT WITH
UAA?
• When we’re writing an (web) application and we DO want to inherit
the credentials of the user
• cli plugins and scripts can usually get this for free
System automation examples:

Onboarding, cleanup, CI stuﬀ (slackbots)

UN HUMANS
• For apps that do not act on behalf of a human we use
client_credentials
• See the OAuth 2.0 spec for more information
• https://tools.ietf.org/html/rfc6749#section-1.3.4
• http://tools.ietf.org/html/rfc6749#section-4.4
If you google, you will ﬁnd people who have translated this into somewhat more complicated english than the spec provides.

GO
uri, _ := url.Parse(uaaURI.String() + "/oauth/token?
grant_type=client_credentials")
creds := &UaaClientCredentials{
uaaURI: uri,
clientID: clientID,
clientSecret: clientSecret,
}
client := http.Client{tls.Config{}}
req, _ := http.NewRequest("POST", creds.uaaURI.String(), nil)
req.SetBasicAuth(creds.clientID, creds.clientSecret)
resp, _ := client.Do(req)

auth = requests.auth.HTTPBasicAuth(client_id, client_secret)
r = requests.post(
url=uaa_uri,
headers={'accept': 'application/json'},
params={'grant_type': ‘client_credentials'},
auth=client_auth)
PYTHON
both of those examples assume certiﬁcate validation, see the docs for skipping it.

➜ curl “https://user:pass@uaa.mycf.xip.io/oauth/token" -d
grant_type=client_credentials -X POST | jq “.”
{
"access_token": “eyJhbG…C6Fvww”,
"token_type": "bearer",
"expires_in": 43199,
"scope": "doppler.firehose",
"jti": "2d56398d-ba65-4dbd-9f43-4d1a302709a1"
}
STOP WITHYOURTRICKERY AND GIVE
ME A SHELL MAN

THE IMPORTANTTHINGS
1. UAA exposes an API protected by HTTP Basic Auth that gives us a token 
http(s)://uaa.example.com/oauth/token
2. We use a client_id & client_secret for auth
3. We pass it the grant type as a parameter 
grant_type=client_credentials
4. It gives us back a token and time to live
note that you can indeed send your credentials in clear text. that’s a bad idea.

you can, and should prevent http requests from getting to uaa as your traﬃc can be sniﬀed.

UAA IS FOR PEOPLETOO?
• CLI Plugins you’ve already assumed the identity of the user
• Scripts should use & assume the identity of the logged in user via cf
curl
• password_grant otherwise (but srsly, why do it the hard way?)
• Web apps use the regular OAuth ﬂow
For applications that act on behalf of humans we have a few options.

For scripts cf curl is a good option as it means you can avoid (hopefully) managing usernames and passwords in the scripts.

require 'bundler'
require 'sinatra'
require 'omniauth'
require 'omniauth-uaa-oauth2'
use Rack::Session::Cookie, :secret => ‘its a secret'
use OmniAuth::Builder do
provider :cloudfoundry, 'app', 'appclientsecret',
{:auth_server_url => “https://login.run.cf.io", :token_server_url => “https://
uaa.run.cf.io"}
end
class App < Sinatra::Base
get '/' do
<ul><li><a href='/auth/cloudfoundry'>Sign in with Cloud Foundry</a></li></ul>
end
get '/auth/cloudfoundry/callback' do
content_type 'application/json'
request.env['omniauth.auth'].to_hash.to_json rescue "No Data"
end
get '/auth/failure' do
content_type 'text/plain'
request.env['omniauth.auth'].to_hash.inspect rescue "No Data"
end
end
run App.new
This one doesn’t do a redirect like I normally do, use has to click the auth link.

WHAT ISTHE FIREHOSE?
• The ﬁrehose is where we go to listen
• Combined stream of logs from all apps, plus metrics data from CF
components
• Exposed as a web socket endpoint
• Accessed via doppler.firehose scope
The scope means it’s unlikely your user accounts have access to it.

Login w/ admin

Or create user in uaac.

WHATS INTHE FIREHOSE
• A lot, but not everything
• https://github.com/cloudfoundry/loggregator/wiki/Firehose-Metric-Catalog
• https://docs.google.com/spreadsheets/d/176yIaJChXEmvm-
CjybmwopdRGQfDGrSzo3J_Mx8mMnk/edit#gid=472741134
• heartbeats are being removed (rejoice!)
• https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/message/
EPTBPAJLBV4LAG72GOSK3KYO7MKZPL7B/
I find the most interesting things in the firehose to be http related. I can tell the status of every request in and out of the system, as well as it’s response time. This helps
me paint a picture of “normal”. It also helps me figure out “lights are on but no one’s home” situations. CF health check just make a connection on a socket, they don’t
check http response codes. So the firehose http metrics give me a good view of what my users are seeing.

HOWTO DISCOVER STUFF?
• noaa ﬁrehose sample app and grep
• ➜ export DOPPLER_ADDR=wss://doppler.10.244.0.34.xip.io:443
• ➜ CF_ACCESS_TOKEN=`cf oauth-token | grep bearer` ./firehose_sample
• https://github.com/cloudfoundry/noaa/blob/master/ﬁrehose_sample/
main.go
*note the app ignores cert validation errors.

WHAT’S INTERESTING
• POST’s to the Cloud Controller
origin:"router__0" eventType:HttpStartStop timestamp:1439755833436264212
deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22"
httpStartStop:<startTimestamp:1439755833419943436 stopTimestamp:
1439755833436264212 requestId:<low:8452822747789629966 high:
15632988695117975649 > peerType:Client method:POST uri:"api.10.244.0.34.xip.io/
v2/organizations" remoteAddress:"10.0.2.15:39470" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1298
parentRequestId:<low:5137755037457773227 high:6003671325381183557 >
instanceId:"10.244.0.134:9022" >

WHAT’S INTERESTING
• POST’s to the Cloud Controller
origin:"router__0" eventType:HttpStartStop timestamp:1439755833436264212 …
peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/organizations"
remoteAddress:"10.0.2.15:39470" userAgent:"go-cli 6.11.2-2a26d55 / darwin"
statusCode:201 … >
Someone just created an org successfully.

WHATS INTERESTING
➜ egrep "method:PUT|method:POST" /tmp/firehose.out | egrep -v "uaa|login"
origin:"router__0" eventType:HttpStartStop timestamp:1439756656563113538 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22"
httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 requestId:<low:5639804705379164088 high:13655764414813434448 >
peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
contentLength:1485 parentRequestId:<low:9677111003368022686 high:16762836456174757743 > instanceId:"10.244.0.134:9022" >
peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
contentLength:1485 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1503 parentRequestId:<low:16449840403178608098 high:
13013670532211215483 > instanceId:"10.244.0.134:9022" >
origin:"legacy" eventType:LogMessage deployment:"cf-warden" job:"api_z1" index:"0" ip:"10.244.0.134" logMessage:<message:"Updated app with guid
d2cd4770-64e0-44fe-a997-f2e7e6dc5c79 ({"route"=>"15879527-b813-4165-9b04-8dc3f5af2a31"})" message_type:OUT timestamp:143975
6656694498836 app_id:"d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" source_type:"API" source_instance:"0" >
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1503 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 contentLength:270 parentRequestId:<low:13565402856673872720 high:11582024932942486143 > instanceId:"10.244.0.134:9022"
>
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 contentLength:270 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 contentLength:4073 parentRequestId:<low:16375959389728685162 high:5351609031812398929 > instanceId:"10.244.0.134:9022"
>
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 contentLength:4073 >
lets distill this and just look at app interactions

WHATS INTERESTING
timestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538
peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-
b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-
b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits"
remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits"
remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79"
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79"
lets distill further and just grab some of the ﬁelds we’re interested in. We can see some requests being passed through the system by looking at the http traﬃc. Kind of a
poor mans after the fact zipkin.

WHATS INTERESTING
peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli
6.11.2-2a26d55 / darwin" statusCode:201 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-
b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-
b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 >
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits"
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits"
peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79"
peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79"
So now we can see that we have a client talking to a server and a server talking to a server

WHATS INTERESTING
timestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795
stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/
v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin"
statusCode:201 instanceId:"10.244.0.134:9022" >
stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167"
lets distill further, to just client operations.

WHATS INTERESTING
instanceId:"10.244.0.134:9022" >
Lets make sure we’re tracing one user’s experience through the system.

note the port changed? Why? I suspect the CLI had to open a new connection..

WHATS INTERESTING
instanceId:"10.244.0.134:9022" >
lets distill further, to just client operations, we can trace one user’s experience through the system.

How long did their operation take? (6.5 seconds), ﬁrehose timestamps are nano seconds.

WHATS INTERESTING
instanceId:"10.244.0.134:9022" >
What did they do what was the output?

First one crates the app.

second one uploads the bits

third one does a state change to start apps.

WHY ISTHAT INTERESTING?
• If I can learn what normal interaction patterns are, I can identify
abnormal ones and investigate
• Are apps failing to start after a push?
• For all teams?
to me anyway.

this lets us identify and view actions across the entirety of the user base.

Could you do this with platform logs? Probably. The beauty of the ﬁrehose is it gives you all of it, I omitted the logs from my example for readability and dramatic eﬀect.

WHAT ISTHE COLLECTOR
• The collector is the “legacy” aggregator of CF metrics
• No log messages
• Grabbed all the /varz and /healthz data
• The collector still has a lot of relevant runtime information in it
• available_stagers, healthy, system.cpu.wait
Most of the collector usage is for dashboards.

WHAT ISTHE COLLECTOR FOR
• It’s also how we export metrics in open source via historian plugins
• https://github.com/cloudfoundry/collector/tree/master/lib/collector/
historian
• Vendors (well Pivotal at least) have their own spin
• Pivotal Ops Metrics (JMX)
The historian plugins change the way data is presented. They are usually monitoring solution speciﬁc.

COLLECTOR DOWNSIDE
• Vendors don’t install all the plugins in their distribution
• Consumption can be irritating, lots of guesswork
• You may end up pretending to be a historian plugin, or consuming
ones output.

CLOUD CONTROLLER
• The CC API is what we talk to the most
• api.run.pivotal.io
• The CC API is where we go to act and report

THE CC API IS UNSURPRISING
• Which is to say, the api has all of the things you would expect
• /v2/apps
• /v3/apps (experimental)
• /v2/organizations
• /v2/events
• /v2/spaces
or kinda boring

API DETAILS
• the v3 api is totes HATEOAS compliant!!!
• the v2 api has some links conventions

THE APITEAM COMMITTEDTO
MAINTAINING BACKWARDS COMPATIBILITY
• v3 api’s will have v2 compatibility layers
• I was there, they said it at cf summit, it’s like they wrote it in their
own blood

API DETAILS
• The v2 api is very, very, very verbose
• The v3 api is more targeted

API DETAILS
• v3 api is fun to talk about, but as of today much of it doesn’t exist
outside of documentation.
• we will see some examples later.

Pitfalls and gotchas
http://imgur.com/gallery/LaJ9Kmo

VERSIONS
does thing you read about exist in your
version

EXPERIMENTAL REALLY MEANS
EXPERIMENTAL.
Theres a good chance it won’t be there (or be the same) in your
vendors next release
(I’m looking at you async service broker api)

SOMETIMES IT’S NOT PRETTY
➜ cf curl /v2/apps/<a guid> | grep buildpack
"buildpack": null,
"detected_buildpack": "PHP",

TOKENS MUST BE ACQUIRED
ANDTHEY EXPIRE
https://github.com/cloudfoundry/cf-uaa-lib (Ruby)
https://github.com/cloudfoundry/omniauth-uaa-oauth2 (more ruby)
https://github.com/krujos/uaaclientcredentials (Go)

CLIENTSVS USERS
https://docs.google.com/document/d/1YU01vrGmIhSHCKOFIRY7EhU8qCbUHJsw3ZXgdnUIfeI/edit
TODO: move that somewhere permanent, and more public.
What’s a client?
What’s a user?

SCOPES,AUTHORITY ANDTHE
GRANTING OF POWER

SO HOW DO I BUILD ATHING?
• it starts with cf curl and jq
• or the ﬁrehose and grep

• lets see what the ﬁrehose has to say about started apps?
export DOPPLER_ADDR=wss://doppler.10.244.0.34.xip.io:443

CF_ACCESS_TOKEN=`cf oauth-token | grep bearer` ./ﬁrehose_sample | grep -i started

cf stop hello-node

cf start hello-node

• how about orphaned user provided services?
cf cups my-svc -p '{"foo":"bar"}'

cf curl /v2/user_provided_service_instances

cf curl `cf curl /v2/user_provided_service_instances | jq -r '.resources[0].entity.service_bindings_url'` | jq ‘.total_results'

Think about what we would have had to do to report on that with the CLI?

• lets look at something more complex.
• how many ai’s are you running?
• How does the usage-report plugin work?
• https://github.com/krujos/usagereport-plugin
demo the plugin 
sometimes its about mashing things up

Why is how many ai’s your running a hard question to answer?

Why is how many ai’s your running an important question to answer?

As someone who’s paying for IT charge back, am I paying too much?

How much ram am I consuming, are my quotas right?

make note of the lack of auth, plugin assumes current user, works for everyone based on their cf access.

 
magic starts here: https://github.com/krujos/usagereport-plugin/blob/master/usagereport.go#L57

https://github.com/krujos/usagereport-plugin/blob/master/apihelper/apihelper.go

• What’s fun? what should we try to ﬁgure out?

Cloud Foundry API for Fun and Ops

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Cloud Foundry API for Fun and Ops

Similar to Cloud Foundry API for Fun and Ops (20)

Recently uploaded

Recently uploaded (20)

Cloud Foundry API for Fun and Ops