A Great API is Hard to Find

       Dan Diephouse
         MuleSoft
         @dandiep
About Me
About MuleSoft
• Connect anything to
  everything!
• Publish APIs
• Mediate services
• Integrate applications
• Load data
• Over 100K dev
  community
• 3200+ production
  deployments
The Impact of APIs
API Proliferation
                                                      9000




                                               4676



                                     2647

                              1628
                       1116
                601
       352
105

2005   2006     2007   2008   2009   2010     2011    2012

                                            Source: Programmable Web
API Billionaires Club 2011




                                     Source: Programmable Web
All contents Copyright ©
                             6
2011, MuleSoft Inc.
The traditional 3-tier architecture

                               Client

                HTML


          Presentation Tier
                              App Server
            Middle Tier



                              Database
              Data Tier


                                           7
…is being decomposed

Presentation Tier           Presentation Tier    Client


   JSON / XML                   JSON / XML


                Middle Tier                     Server




                                                 Data
 database           Data Tier


                                                          8
…is being decomposed

Presentation Tier    Presentation Tier   3rd party Apps    Client

   JSON / XML            JSON / XML       JSON / XML



                     Middle Tier                          Server




                                                           Data
 database              Data Tier


                                                              9
…is being decomposed

Presentation Tier                       Presentation Tier               3rd party Apps    Client

   JSON / XML                               JSON / XML                   JSON / XML



                                        Middle Tier                                      Server



                                          API      API      API
                                API                               API
                            SaaS, Infrastructure Services,                                Data
 database
                    API
                                  Social Media APIs        API             Data Tier
                          API                            API
                                  API              API
                                             API

                                                                                            10
Platform Shift
Traditional Application Environments


         Application

       Web/App Server
          Database
      Operating System
Platform Shift
New Application Environments


        Application      Application

      Web/App Server        PaaS
         Database
                            IaaS
      Operating System
Technology Shift
Traditional Application Environments

                        Application
    Application
                                 UI




                                            Security
     Database
                           Business Logic
    Web Server
  Operating System              Data
Technology Shift
Newer Application Environments

                        Application
    Application




                                              Security
                            UI        API
     Database
                           Business Logic
    Web Server
  Operating System          Data      Integration
Technology Shift
Application Decomposition
Application


                      Security
    UI        API

   Business Logic

    Data      Integration
What APIs are you using?
•   CRM – Salesforce, MS Dynamics, SAP
•   Data services – Xeround, Mongo, RDS
•   eCommerce – PayPal, QuickBooks, Xero, Freshbooks
•   Email – Amazon SES, SendGrid
•   Messaging – PubNub, Cloud AMQP
•   Notifications – Urban Airship, Twilio
•   Security – Katasoft
•   Social – Facebook, Twitter, LinkedIn
•   Storage – S3, DropBox
Changing business models
Build an eco-system of
integrations which
provide more value to                 CRM


your customers            Mobile              ERPs


                                     Your
Plethora of business
models – fremium, pay    eCommerce
                                      API   Marketing

for use, tiers, etc
                                      HRM
GREAT APIS
A GREAT API IS … USER FRIENDLY
What does the user want?


  How do they want it?
Sidebar: REST is awesome!
5 interaction patterns to consider

choose the right one for the job
#1: CRUD + Actions
Create    POST /widgets
 Read     GET /widgets
          GET /widgets?name=Foo
          GET /widgets/123
Update    PUT /widgets/123
 Delete   DELETE /widgets/123
   …
Execute   POST /widgets/123/execute
#2: Batch
 “Web architects must understand that resources
are just consistent mappings from an identifier to
some set of views on server-side state. If one view
doesn’t suit your needs, then feel free to create a
different resource that provides a better view (for
 any definition of “better”). These views need not
 have anything to do with how the information is
stored on the server, or even what kind of state it
       ultimately reflects. It just needs to be
understandable (and actionable) by the recipient.”
                      - Fielding
#2: Batch
  Bulk Load      POST /jobs
                 * , widget1 -, ,widget2-, … +

                 200 OK
                 Location /jobs/123

Get Job Status   GET /jobs/123

                 [ status1, status2, status3, etc ]
#3: Streaming


             Long poll
Client                      API
             Async events
#4:
• Instant notification for the web!
• Example:
  – Client creates an invoice
  – Freshbooks calls HTTP webhook to synchronize
    invoice to Salesforce
#5: Async
1. Send message

  POST /messages
  { … }

 201 Received
 Location /messages/123

2. Check Status

  GET /messages/123
A GREAT API IS … CORRECT*




* Except when it shouldn’t be
Partial responses
  Dates & Timezones
                            Hypertext
     Stateful
            Details matter
Error 500       Content-Types

                      GET       Pagination
   Data modeling
Data Types
OrganizationServiceStub.AttributeCollection updateCollection =
  new OrganizationServiceStub.AttributeCollection();

OrganizationServiceStub.KeyValuePairOfstringanyType telephone =
 new OrganizationServiceStub.KeyValuePairOfstringanyType();
telephone.setKey("telephone1");
telephone.setValue("425-555-1212");

updateCollection.addKeyValuePairOfstringanyType(telephone);
Dates
{
    createdAt : 124059811
    …
}
Dates
{
    createdAt : “2008-03-01T13:00:00Z”
    …
}
GET
GET /api/contacts/delete

200 OK
GET
DELETE /api/contacts/123

200 OK
Hypertext
GET /api/contacts

200 OK
[
  {
  “id” : “123”
  }
]
Hypertext
GET /api/contacts

200 OK
[
 {
   “href” : “/api/contacts/123”
   “pictureHref” : “/api/contacts/123/johndoe.jpg”
 }
]
A GREAT API IS … SECURE

• A GREAT API IS…SECURE
Do you think you’re special?
“Special” Companies           Normal Companies
• Microsoft (WS-              • Salesforce (OAuth 2 or Basic
  Security/Policy + Live ID     Auth*)
  variant)                    • Twitter (OAuth 1)
• QuickBooks (SAML/OAuth      • Facebook (OAuth 2)
  variation)
• AWS (Custom encryption)
Basic Auth + SSL
• Easy
• Accessible
• Not great for public APIs…
OAuth!
• 1.0: out of band tokens
• 2.0:
  – 2 legged authentication
  – No more encryption of tokens
  – Short lived tokens with expiration & refresh
  – Grant types
WS-Security
A GREAT API IS … DOCUMENTED
• TODO: screenshots
  – Amazon
• Magento
• Apiary
A GREAT API IS … VERSIONED
POST /api/v1/foo
POST /api/1.0/foo
POST /api/2012-01-01/foo
POST /api/foo?v=2012-01-01
POST /api/foo
Version: 1.0
POST /api/foo
Content-Type: application/vnd.foo+json;v=1.0
Things to consider
•   Include versioning from the start
•   How long should you maintain versions?
•   How often will you make changes?
•   Will you have minor versions?
•   Date based?
Which approach
Header                         URL
• Potentially more “correct”   • Easier to hack in the
  HATEOS approach                browser & with curl
                               • Provides clarity when there
                                 are structural changes
                                  – e.g. it’s clear that resource
                                    /foo went away in version 2
A GREAT API … FAILS GRACEFULLY
A great error has
1. A machine understandable HTTP status code
2. An end user message
3. If relevant, details for the developer to
   escalate the issue (tracking #)
POST /foo
{ … bad data … }

200 OK
{
  “message” : “Invalid request”
}
POST /foo
{ … bad data … }

400 Bad Request
Content-Length: 0
Good
POST /foo
{ … bad data … }

400 Bad Request
{
  “message” : “The field foo123
is not allowed on the request.”
}
Good
POST /contacts
{ “name” : “Dan Diephouse” }

409 Contact Exists
{
  “message” : “A contact with
that name already exists.”
}
Good
POST /contacts
{ “name” : “Dan Diephouse” }

500 Error
{
  “message” : “We were not able to process
you’re request due to an unexpected error.
Please contact support for help in resolving
this request (Request ID 19022334).”
  “requestId” : 19022334
  “time” : “2012-03-01T13:00:00Z”
}
A Great API
•   User friendly
•   “Correct”
•   Secure
•   Documented
•   Versioned
•   Fails Gracefully
Questions?

         @dandiep
dan.diephouse@mulesoft.com

A great api is hard to find

  • 1.
    A Great APIis Hard to Find Dan Diephouse MuleSoft @dandiep
  • 2.
  • 3.
    About MuleSoft • Connectanything to everything! • Publish APIs • Mediate services • Integrate applications • Load data • Over 100K dev community • 3200+ production deployments
  • 4.
  • 5.
    API Proliferation 9000 4676 2647 1628 1116 601 352 105 2005 2006 2007 2008 2009 2010 2011 2012 Source: Programmable Web
  • 6.
    API Billionaires Club2011 Source: Programmable Web All contents Copyright © 6 2011, MuleSoft Inc.
  • 7.
    The traditional 3-tierarchitecture Client HTML Presentation Tier App Server Middle Tier Database Data Tier 7
  • 8.
    …is being decomposed PresentationTier Presentation Tier Client JSON / XML JSON / XML Middle Tier Server Data database Data Tier 8
  • 9.
    …is being decomposed PresentationTier Presentation Tier 3rd party Apps Client JSON / XML JSON / XML JSON / XML Middle Tier Server Data database Data Tier 9
  • 10.
    …is being decomposed PresentationTier Presentation Tier 3rd party Apps Client JSON / XML JSON / XML JSON / XML Middle Tier Server API API API API API SaaS, Infrastructure Services, Data database API Social Media APIs API Data Tier API API API API API 10
  • 11.
    Platform Shift Traditional ApplicationEnvironments Application Web/App Server Database Operating System
  • 12.
    Platform Shift New ApplicationEnvironments Application Application Web/App Server PaaS Database IaaS Operating System
  • 13.
    Technology Shift Traditional ApplicationEnvironments Application Application UI Security Database Business Logic Web Server Operating System Data
  • 14.
    Technology Shift Newer ApplicationEnvironments Application Application Security UI API Database Business Logic Web Server Operating System Data Integration
  • 15.
    Technology Shift Application Decomposition Application Security UI API Business Logic Data Integration
  • 16.
    What APIs areyou using? • CRM – Salesforce, MS Dynamics, SAP • Data services – Xeround, Mongo, RDS • eCommerce – PayPal, QuickBooks, Xero, Freshbooks • Email – Amazon SES, SendGrid • Messaging – PubNub, Cloud AMQP • Notifications – Urban Airship, Twilio • Security – Katasoft • Social – Facebook, Twitter, LinkedIn • Storage – S3, DropBox
  • 17.
    Changing business models Buildan eco-system of integrations which provide more value to CRM your customers Mobile ERPs Your Plethora of business models – fremium, pay eCommerce API Marketing for use, tiers, etc HRM
  • 18.
  • 19.
    A GREAT APIIS … USER FRIENDLY
  • 20.
    What does theuser want? How do they want it?
  • 21.
  • 22.
    5 interaction patternsto consider choose the right one for the job
  • 23.
    #1: CRUD +Actions Create POST /widgets Read GET /widgets GET /widgets?name=Foo GET /widgets/123 Update PUT /widgets/123 Delete DELETE /widgets/123 … Execute POST /widgets/123/execute
  • 24.
    #2: Batch “Webarchitects must understand that resources are just consistent mappings from an identifier to some set of views on server-side state. If one view doesn’t suit your needs, then feel free to create a different resource that provides a better view (for any definition of “better”). These views need not have anything to do with how the information is stored on the server, or even what kind of state it ultimately reflects. It just needs to be understandable (and actionable) by the recipient.” - Fielding
  • 25.
    #2: Batch Bulk Load POST /jobs * , widget1 -, ,widget2-, … + 200 OK Location /jobs/123 Get Job Status GET /jobs/123 [ status1, status2, status3, etc ]
  • 26.
    #3: Streaming Long poll Client API Async events
  • 27.
    #4: • Instant notificationfor the web! • Example: – Client creates an invoice – Freshbooks calls HTTP webhook to synchronize invoice to Salesforce
  • 29.
    #5: Async 1. Sendmessage POST /messages { … } 201 Received Location /messages/123 2. Check Status GET /messages/123
  • 30.
    A GREAT APIIS … CORRECT* * Except when it shouldn’t be
  • 31.
    Partial responses Dates & Timezones Hypertext Stateful Details matter Error 500 Content-Types GET Pagination Data modeling
  • 32.
    Data Types OrganizationServiceStub.AttributeCollection updateCollection= new OrganizationServiceStub.AttributeCollection(); OrganizationServiceStub.KeyValuePairOfstringanyType telephone = new OrganizationServiceStub.KeyValuePairOfstringanyType(); telephone.setKey("telephone1"); telephone.setValue("425-555-1212"); updateCollection.addKeyValuePairOfstringanyType(telephone);
  • 33.
    Dates { createdAt : 124059811 … }
  • 34.
    Dates { createdAt : “2008-03-01T13:00:00Z” … }
  • 35.
  • 36.
  • 37.
    Hypertext GET /api/contacts 200 OK [ { “id” : “123” } ]
  • 38.
    Hypertext GET /api/contacts 200 OK [ { “href” : “/api/contacts/123” “pictureHref” : “/api/contacts/123/johndoe.jpg” } ]
  • 39.
    A GREAT APIIS … SECURE • A GREAT API IS…SECURE
  • 40.
    Do you thinkyou’re special?
  • 41.
    “Special” Companies Normal Companies • Microsoft (WS- • Salesforce (OAuth 2 or Basic Security/Policy + Live ID Auth*) variant) • Twitter (OAuth 1) • QuickBooks (SAML/OAuth • Facebook (OAuth 2) variation) • AWS (Custom encryption)
  • 42.
    Basic Auth +SSL • Easy • Accessible • Not great for public APIs…
  • 43.
    OAuth! • 1.0: outof band tokens • 2.0: – 2 legged authentication – No more encryption of tokens – Short lived tokens with expiration & refresh – Grant types
  • 44.
  • 45.
    A GREAT APIIS … DOCUMENTED
  • 46.
  • 47.
  • 48.
  • 50.
    A GREAT APIIS … VERSIONED
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
    Things to consider • Include versioning from the start • How long should you maintain versions? • How often will you make changes? • Will you have minor versions? • Date based?
  • 58.
    Which approach Header URL • Potentially more “correct” • Easier to hack in the HATEOS approach browser & with curl • Provides clarity when there are structural changes – e.g. it’s clear that resource /foo went away in version 2
  • 59.
    A GREAT API… FAILS GRACEFULLY
  • 60.
    A great errorhas 1. A machine understandable HTTP status code 2. An end user message 3. If relevant, details for the developer to escalate the issue (tracking #)
  • 61.
    POST /foo { …bad data … } 200 OK { “message” : “Invalid request” }
  • 62.
    POST /foo { …bad data … } 400 Bad Request Content-Length: 0
  • 63.
    Good POST /foo { …bad data … } 400 Bad Request { “message” : “The field foo123 is not allowed on the request.” }
  • 64.
    Good POST /contacts { “name”: “Dan Diephouse” } 409 Contact Exists { “message” : “A contact with that name already exists.” }
  • 65.
    Good POST /contacts { “name”: “Dan Diephouse” } 500 Error { “message” : “We were not able to process you’re request due to an unexpected error. Please contact support for help in resolving this request (Request ID 19022334).” “requestId” : 19022334 “time” : “2012-03-01T13:00:00Z” }
  • 66.
    A Great API • User friendly • “Correct” • Secure • Documented • Versioned • Fails Gracefully
  • 67.
    Questions? @dandiep dan.diephouse@mulesoft.com

Editor's Notes

  • #3 Thanks for inviting me.Background in integration I live in San FranciscoI love wineI play the accordionI love building products and products need APIs.Currently focused on the problem of connecting everything together – how do we make it as simple as possible? How do we break down cloud silos?
  • #5 Let’s talk about the impact of APIs and I don’t mean all the schwag we see around this conference. Look around, all these companies are focusing on APIs. It’s not just developers anymore, it’s everyone.How many of you integrate with more than one API?
  • #6 Center of gravity shifting to the cloudRise of the composite appLess visibility and controlRapid innovation means middleware must adapt and support change
  • #8 We’ll start around the early 2000s, where the traditional application looked something .HTML rendered by serverData tier maintained by db admins, run on premise
  • #9 Presentation tier moves to client side – AJAX (Backbone, Ext, Sencha, Jquery, etc) and Mobile (iPhone, Android)Expectation now from consumers that you have a mobile appForces a shift – everything now has a REST API
  • #10 People DEMAND an API and your business wants to monetize it Look at all the companies here around this…
  • #11 Correspondingly, you want to use everyone else’s APIsYour application is no longer a database, it’s a composite application based on a number of different infrastructure services, social media APIs, etc
  • #12 Simultaneously there is a platform shift, we’re moving from the 3 tier server to PaaS and IaaSHow many here deploy their applications on Heroku?
  • #13 Building it themselves
  • #14 Data is still important, but integration to external services is becoming just as important. Is all your data in the database? Or do you have it spread out over different services/datastore? There is no one DB to rule them all for example
  • #15 Building it themselves
  • #16 Building it themselves
  • #20 The worst APIs are the ones that were designed without any thought. The ones that just expose whatever at random. They don’t worry about the how and why, but as any user knows, that’s the most important part.
  • #22 This talk presupposes that REST is the way to go, so I figure I better say a few words on that. I really go to REST+JSON by default. You have to make a strong argument with me to go another way. I’m sure you know by now – REST stands for Representation State transfer. The simple way to explain it is that you have a set of nouns (widgets) and fixed set of verbs. REST has so many good things going for it-it is the language of the webit is scalableit is easy to get started – you just hit the URL in the browser or with Curl. No complicated libraries. You can also take it in pieces. it is universalit is evolvable – we can use URIs and data formats to build up a set of nouns The only thing about REST that is not great is that people easily screw it up. I’m sure people will say the same thing about me – because I’m advocating a kind of REST-lite approach. but as long as you’re concious of the downsides of leaving other stuff behind, you can do it.I think the market is bearing this out as well in terms of what we’re seeing from vendors and the services they produce.
  • #24 Choose your nouns, not your verbs.For things that don’t fit, then put the action in the URI
  • #25 A great API supports batch if your users need it. Don’t make people hit your API a million times to import a million rows.While there is also HTTP pipelineing, it has a couple challengeserrors returned in same order, which can add latencynot well supported by client libraries
  • #27 I call this streaming because this is what Salesforce calls it, but by it I simply mean pub sub over HTTP-It gives up some of the advantages of REST (statefulness), but it gives you real time updates and reduces the # of requests the client needs to make-Difficult to implement for consumers as it’s uncommon-
  • #28 “so simple you’ll think it’s stupid”
  • #31 I can’t make you to write beautiful code, but I can try! So I will highlight a few problems if you don’t implement stuff correctly
  • #33 MS Dynamics CRMno thought to data structuresTried to do a schemaless design in a schema – XML schema just deal well with that. Ended up with Arrayofstringtype and so on-would be better off with a schemaless JSON based design
  • #34 Timezones!
  • #35 Timezones!
  • #38 Make your application hypertext. Then users can browse to different resources
  • #42 AWS I will give a little bit of slack because pre-date everything – and at least it’s consistent and easy.
  • #44 OAuth allows you to build an authentication mechanism which allows applications to act, not just usersIt is much more secure for 3rd party applications because you can track turn off access for invididual applications.
  • #45 If anything use XML encryption. I completely get that there are valid use cases that WS-Security meets that other specs don’t, but crikey, it’s impossible for people to use, so don’t expect your API to catch on with it.