Presented at 'All Things API' in Denver, CO by Travis McChesney, Director of Engineering at Cloud Elements.
How do you build and use user defined callback URLs (known as Webhooks) to notify your users of events that occurred on your system? Or use those URLs to get remote notification from API connected systems you use?
Using Webhooks is becoming more common as APIs become essential to all programming models. We will cover four common usage models: API capture, TCP Tunneling, Dynamic DNS and Remote Development.
2. Confidential & ProprietaryConfidential & Proprietary
Applications are InherentlyDecoupled
APIs glue applications together
Most APIs can be used synchronously
3. Confidential & ProprietaryConfidential & Proprietary
Application should provide EventDrivenAPIs
Very useful for some integration use cases
Improves UX of application APIs
4. Confidential & ProprietaryConfidential & Proprietary
UseCasesforEventDrivenAPIs
Integrations that need to consume large volumes of data
Can’t keep a synchronous connection open for 15 minutes
Long running, intensive operations such as complicated searches
Services that cannot provide a definitive response synchronously
5. Confidential & ProprietaryConfidential & Proprietary
Asynchronous APIOptions
Polling
Uses unnecessary compute cycles
Frequency is application specific
Concurrent polling is complicated and introduces instability
6. Confidential & ProprietaryConfidential & Proprietary
LettheAPIProviderNotifyYourApplication
Register a callback API with Event Handler logic
Registered Event Handler is invoked when the data is ready for consumption
This is basically a Webhook
Trigger action for Callback API, and respond immediately to your users
7. Confidential & ProprietaryConfidential & Proprietary
Example WebhookWorkflow
Your app send an email using a Cloud Email Service
Create an email message and send it via the Cloud Email Service
Respond to your user that the message has been queued
Persist the Email Message ID
Your app wants to know when the email is processed, sent and opened
Register the Event Handler or Callback API for the Webhook
8. Confidential & ProprietaryConfidential & Proprietary
Example WebhookWorkflow
The Cloud Email Service Provider invokes the Callback API for each event
Your app receives the event and message ID via the Callback API request
Your app handles each event and notifies the user of each event
9. Confidential & ProprietaryConfidential & Proprietary
WebhookPitfalls
Watch for circular references
Updating Field 1 invokes a callback with a specific event
Your app processes the event and invokes the API to update Field 2
The update of Field 2 invokes a callback, which in turn updates Field 1 again
Security concerns
Public API endpoint is now exposed
Temp tokens or consumable message IDs
10. Confidential & ProprietaryConfidential & Proprietary
Using WebhooksinDevelopment
A Webhook needs a public callback URL to invoke
Most developers use localhost
So, how can I test a webhook while developing?
11. Confidential & ProprietaryConfidential & Proprietary
Testing WebhooksinDevelopment
API Capture
TCP Tunneling
OpenSSH Tunneling
Dynamic DNS
Remote Server
12. Confidential & ProprietaryConfidential & Proprietary
API Capture forWebhooksinDevelopment
API Capture is one mechanism to test webhooks in development
Leverage services such a Runscope’s requestb.in
Register this URL as the application callback URL for the Webhook
Great for inspecting the posted callback data
When the callback is invoked, use this URL in a browser
Create a webhook callback URL at requestb.in
Inspect the posted data
Not as easy for debugging when something’s not quite right
13. Confidential & ProprietaryConfidential & Proprietary
TCP Tunneling forWebhooksinDevelopment
Services such as localtunnel.me, localtunnel.com, ProxyLocal or ForwardHQ
> npm install -g localtunnel
Need to reset tunnel when internet connection changes
> lt —port <port>
localtunnel.me example
Need to install client on localhost
14. Confidential & ProprietaryConfidential & Proprietary
OpenSSH Tunneling forWebhooksinDevelopment
Open and keep open socket from localhost to remote server
Configure remote server to pass requests down the connection
Setup OpenSSH reverse proxy to a public server
Need to reset tunnel when internet connection changes
Needs GatewayPorts and AllowTcpForwarding enabled on remote server
ssh -N -g -R :8080:localhost:8080 user@remote_server &
Needs remote server with root access or system administrator access
15. Confidential & ProprietaryConfidential & Proprietary
Dynamic DNS forWebhooksinDevelopment
Use DynDNS or No-IP based DNS to route to localhost
Needs router or firewall configuration
Get a free DNS from one of the providers
Configure DNS to forward to external IP of home office
Configure home office router to route requests to private IP
Works only from a single location
16. Confidential & ProprietaryConfidential & Proprietary
Remote Server forWebhooksinDevelopment
Use a public remote server for development
Deploy application to publicly available host
Need a public host, although easier now with public cloud providers
Connect your IDE’s debugger to public host
Debugging over the internet is typically slow
17. Confidential & ProprietaryConfidential & Proprietary
TCP Tunneling DemowithSendGridElement
Get localtunnel.me based application callback URL
Register localtunnel.me application callback URL with SendGrid
Send email using Cloud Elements’ Messaging Hub
Upon email reception, handle event callback
18. Cloud Elements 2014 - Confidential & Proprietary
CLOUD ELEMENTS
One-to-Many
One API connects to Many of the leading services in a category
(e.g. CRM, Documents, Marketing)
Multi-Tenant Connectors
Each Element automatically authenticates to an unlimited number
of customer accounts
Embed Seamlessly in Your App
Keeps your app all about your brand as our Elements work behind
the scenes through RESTful APIs
Monthly, Pay As You Go
Charged only when your customers use your application
18
Less work, less cost, and more connections for your application.
Set the stage for use cases for asynchronous APIs.
As APIs get more and more prevalent and start doing more and more tasks, an asynchronous approach is going to be necessary
Bulk uploads and downloads from CRM systems. Don’t want to do that synchronously.
Long search operations
Using compute cycles if there is no data to be pulled.
Cases where the frequency of polling is application specific. Some cases every 5 seconds, other times every hour. Could depend on the data.
Example: Buying Rockies tickets. Rather than refreshing the page, you could get notified.
Multiple threads trying to get the same data.
Providing an event handler for your callback API. Ex: SFDC. Notified when an Opportunity object probability goes above 60%.
There are things to consider with web hooks as well.
Some service providers only allow one callback URL, and your app has to handle the event based on the payload received.