This document provides an overview of how to build complex calls to APIs without writing code using FME. It discusses features of the HTTPCaller like cookies, concurrency, rate limiting, building SQL-like queries, uploading/downloading files, pagination, and creating custom web tools with custom transformers. Examples are given around traffic cameras, photo processing, and retrieving all jobs from an API. Tips discussed include using other tools like Postman to test workflows and starting simple before adding complexity.

1. How to Build Complex Calls to
APIs (Without Writing Any Code)

2. Just getting started with REST
APIs?
How to Connect to Any REST API (Without Writing
Any Code)

3. Meet the Presenters
Sanae Mendoza
FME Server Support
Specialist
Sienna Emery
FME Server Support
Specialist

4. Agenda
What we’re covering today.
● Intro to FME
● Did you know the HTTPCaller
could do that?
● Building HTTP “SQL” Queries
● Turning cURL into HTTP Calls
● Building Custom Web Tools
● Pagination in the HTTPCaller

5. Our Mission
To help you maximize the value of your data.

6. FME®
Integration Platform
Connect. Transform. Automate.
FME Desktop
Build & Run Data Workflows
Get a free trial of FME Desktop and FME Server at safe.com
FME Server
Automate Data Workflows
FME Cloud
Hosted FME Server
FME Mobile
Deliver the Power of
FME to Mobile

7. Connect Your Data Sources Transform Your Data Automate Your Workflows
Integrate and convert information
across 400+ applications.
450+ powerful transformers
safe.com/transformers | hub.safe.com
Build event based workflows in a visual
interface

8. Did you know the HTTPCaller could do that?
New and Advanced Features.

9. Cookies
A small chunk of data given a user’s
web browser by the server. The
“cookie” is exchanged back and forth,
so that the server can track that user’s
requests.

10. All About Cookies
Cookies are Stateful.
Both the client and server keep a record of the ongoing “session” with a
unique cookie.
Typical Uses for Cookies
Authentication. Keeping you logged in.
User Personalization. Dark mode or light mode.
Tracking. Following you around the internet.

11. Maintaining a Session inside a Workspace
The HTTPCaller
Response Headers (Feature Information)

12. Concurrency
Multiple requests can be sent and
processed at the same time
(make your workflows faster!)

13. Concurrency / Looping Options
Control how many HTTP requests you send at one time
Sending one request at a time might slow down your workspace.
By increasing the number of concurrent requests allowed, you can process
multiple requests at the same time. This can provide significant
performance benefits.
(Just make sure the HTTP server allows it!)

14. HTTPCaller Racing
50 requests 1 request

15. Rate Limiting
The number of HTTP requests
allowed to be made within a given
time period.

16. Working within server rate limits
Some servers limit how many requests can be sent in a time frame. Exceeding those
limits can result in an error or temporary block.
Control the maximum number of requests that can be made within a time frame to
make sure your workspaces always run smoothly.
Rate Limiting
What is the API rate limit here? 1 per year?

17. Building HTTP “SQL” Queries
Dynamically build HTTP queries in your workspaces

18. API “Databases”
Socrata Open Data Portal
Online platform for government, non-profit, and NGO datasets.
Socrata Open Data API (SODA)
A tabular “database” that can be accessed programmically.
Webinar: Open Data… Automated! Using FME with Socrata
Don’t forget the Socrata Reader and Writer

19. Combining HTTP parameters with Structured Query Language (SQL)
SoQL (Socrata Query Language) allows for complex queries, functions, or analysis.
(See also: GraphQL, SOQL (Salesforce Query Language), PostgREST….)
SoQL queries are sent with REST calls just like parameters. Append a query to an API
endpoint to pull specific data from an API accessible dataset.
API Query Languages
e.g. ROUND(123.4545, 2) = 123.4500

20. Traffic Camera Demo
The Goal
I want to pull traffic
cameras along my
routes, I need to
build a SoQL clause
to use the Socrata
API.
The Obstacles
I’m not familiar
with API query
languages and the
traffic camera
dataset is very
large.
The Solution
I can use FME to
dynamically build
SoQL clauses so that
I can pull just the
data I need each
time.
The Result
Socrata is queried
for each route.
I can leverage SoQL
queries to make my
workflows more
efficient.

21. Turning cURL into HTTP Calls
What to do when the API documentation is written in cURL

22. API Documentation
Ever get confused by API documentation
like this?
Some API documentation may be written
in cURL or a programming language
Question of the Week: Web Services,
cURL, and the HTTPCaller

23. cURL
● Client URL
● A command line tool that is most
likely already installed on your
machine
● Works for HTTP calls, FTP, and much
more!
● FME uses it in the HTTPCaller
Want to learn more?

24. Using Postman
Using Postman (a free
tool!) cURL can be
automatically transformed
into HTTP requests

25. Understanding cURL
https://idratherbewriting.com/learnapidoc/docapis_understand_curl.html
cURL Description
-X HTTP Method
-H Header
-F Form content. This causes
curl to POST data using
the Content-Type
multipart/form-data
Need more?

26. Demo Intro
Colorize through deep
learning API

27. Uploading Attachments
There are many ways to upload files using the HTTPCaller. Read the API documentation
to determine which to use!

28. Downloading Attachments
Files can be downloaded directly in the HTTPCaller! This sends the response body to a
file instead of an attribute.

29. Photo Processing Demo
The Goal
I have a series of
photos I’d like to
send to an API for
processing.
The Obstacles
I’m not a
programmer and
I’m unsure how to
use cURL.
The Solution
I can use FME and
Postman to break
down this cURL into
an HTTP Request I
can easily use in the
HTTPCaller
The Result
I can now
automatically
process photos
without writing any
code or use a
command line tool!

30. Building Custom Web Tools
Introducing Custom Transformers

31. A sequence of standard transformers becomes… a custom transformer!

32. 5 reasons to Custom Transformers
1. Organization. Separate workspace content into tabs on the canvas.
2. Flexibility. Use existing custom transformers as a prototype, or tweak your own.
3. Efficiency. Reuse sequences of transformers across workspaces.
4. The FMX file. Share it with others or publish it to FME Hub.
5. Custom. Develop personalized FME tools that fit your workflows.

33. Embedded transformers are stored with the workspace.
Linked transformers’ definitions are stored as an external file
(.fmx). If the definition changes, all workspaces will be updated to
reflect those changes.
Where do I find them?
The Transformer Gallery in FME Workbench or The FME Hub, a
library of custom transformers and resources.
How do I make one?
Select a group of transformers, right click and select “Create
Custom Transformer....” Give it a name, description, and set up
user parameters.

34. Building Custom Web Tools in FME
The Goal
I need a tool that
sends text
messages via API so
that I can quickly
add it to my
organization's
workspaces.
The Obstacles
There is not a
transformer that
fits my needs and it
is not convenient to
configure the API
request each time.
The Solution
I can “wrap up” a
sequence of
transformers, with
an API request, into a
convenient package.
The Result
I can reuse and
share a custom
transformer that
accepts, sends, and
processes my API
requests.

35. Pagination in the HTTPCaller
Getting all the data you need!

36. Pagination
If an API is returning many results, the results are often broken down into pages so
the responses from the API are kept short. This is similar to a web page displaying
results on different pages to not overwhelm users.

37. Pagination Variables
Just like this page on FME Hub, HTTP calls often let you select the number of results per
page and how many pages you’d like to return.

38. Using pagination in FME
I’d like to retrieve FME Server job statistics, but the FME Server REST API will only let me
retrieve 1000 jobs at a time. How would I retrieve all jobs from a certain date in FME?

39. Example attributes for pagination
Attribute Definition
limit (integer) limits how many items are returned by this call.
Could set it to 10 to only retrieve 10 jobs at a time.
offset (integer) Offset of this results page

40. To loop or not to loop
Typically, getting all the results
from every page in a REST call
requires a loop in a custom
transformer.
However, you don’t necessarily
need a loop if you know how
many pages you need in advance.

41. Okay, I need to loop
● In FME looping functionality can be found in custom transformers
● Loops can be added by right clicking on the canvas and selecting Insert
Transformer Input, then Insert Transformer Loop

42. Looping Considerations
● Without the right logic, it is very easy
to create an infinite loop!
● Avoid blocking transformers if
possible. Blocking transformers can
only be used in linked transformers
which are much more difficult to
troubleshoot
● Set the HTTPCaller concurrency to 1

43. Pagination demo
The Goal
I’d like to pull data
from all FME Server
jobs around a
certain date
The Obstacles
I need to make
multiple calls to
the FME Server API
to get all results
The Solution
I can use a loop in a
custom transformer
to get all the results
The Result
Using FME, I can
get all the results I
need in an easy
way

44. Tips & Tricks
● The documentation should have all
you need to know about working
with REST APIs
● External tools like Postman can be
an easy way to test the workflow
● Start small and build complexity
● Developer tools in a browser can
be used to help understand the
calls being made
● Fiddler can be used to track
requests and response

45. Which REST API service are
you using? How can we help
you get connected?

46. Poll:
Which REST API training content are you most interested
in?
● Creating your own Web Services
● Debugging HTTP errors
● Web Mapping and Applications
● Parsing JSON
● Other (chat out anything missed!)

47. Getting Started
Download FME 2021.0 Download the
workspaces from this
webinar
Try connecting to a
REST API!
Free Trial | Upgrade

48. Webinars
How to Connect to Any API (Without Writing Any Code)
A Step-By-Step Guide to Building Codeless Web Apps
Automating the Enterprise with Event-Driven Workflows
Useful Links More Resources...
Getting Started with APIs
FME Server REST API Training
Creating Web Connections
FME Hub
FME Community

49. Q&A

50. Thank you!
Try out your REST API workflow today!
Connect with us for more FME