WHITE PAPER




Simplifying RESTful Search




Abstract
  This white paper talks about how search over REST can
  be simplified. We are not aiming at developing
  standards for RESTful search, but will be discussing
  how this problem can be approached.




 Impetus Technologies Inc.
 www.impetus.com
 January 2012
Simplifying RESTful Search



                                                                             Table of Contents
Introduction ........................................................................................................... 2
Search Requirements ............................................................................................. 3
HTTP Method Selection ......................................................................................... 4
URL Representation ............................................................................................... 4
   Query Criteria vs. Embedded Criteria ................................................................ 4
   Modeling Filter Criteria ...................................................................................... 5
   Feed Item Query Language (FIQL) ..................................................................... 5
   Resource Query Language (RQL) ....................................................................... 5
Case study: Apache CXF advance search features ................................................. 6
Summary ................................................................................................................ 7




                                                                                          Introduction
The REST architectural pattern is based around two basic principles:
      1. Resources as URLs: A resource is something like an entity or a noun in
         modeling language. Anything on the web is identified as a resource and
         each unique resource is identified by a unique URL.
      2. Operations as HTTP methods: REST leverages existing HTTP methods,
         particularly GET, PUT, POST, and DELETE which map to a resource’s
         read, create, modify and removal operations, respectively.
Any action performed by a client over HTTP, contains a URL and an HTTP
method. The URL represents the resource and the HTTP method represents the
action which needs to be performed over the resource.

Being a broad architectural style, REST always has different interpretations. The
ambiguity is exacerbated by the fact that there are not nearly enough HTTP
methods to support common operations. One of the most common examples is
the lack of a ‘search’ method. While search is one of the most extensively used
features across different applications, there have been no standards for
implementing this feature. Due to this, people tend to design search in different
ways. Given that REST aims to unify service architecture, any ambiguity must be
seen as weakening the argument for REST.




                                                                                                                             2
Simplifying RESTful Search



                                               Search Requirements
Search is the most used feature across different web applications, and supports
almost similar features around different applications. Listed below are some of
the common constituents of search features:


•   Search based on one or more criteria at a time
        -   Search red colored cars of type hatchback
                     color=red && type=hatchback
•   Relational and conditional operator support
        -   Search red or black car with mileage greater than 10
                     Colour=red|black&& mileage > 10
•   Wild card search
        -   Search car manufactured from company name starting with M
                     company=M*
•   Pagination
        -   List all cars but fetch 100 results at a time
                     upperLimit=200 &&lowerLimit=101
•   Range searches
        -   Get me all the cars launched between 2000 and 2010
                     launch year between (2000, 2010)


When we support search with such features, the search interface design itself
becomes complex. And when implemented in a REST framework, meeting all
these requirements (whilestill conforming to REST!) is challenging.
Coming back to the basic REST principles, there are two questions that need to
be answered:


1. Which HTTP method should be used for ‘search’?
2. How can an effective resource URL be created for search?
    a. Query parameters versus Embedded URLs
    b. Modeling filter criteria




                                                                                         3
Simplifying RESTful Search



                                           HTTP Method Selection
Effectively, REST categorizes operations by their nature and associates well-
defined semantics with these categories. The idempotent operations are GET,
PUT, and DELETE (GET for read-only, PUT for update, DELETE for remove),
while the POST method is used for non-idempotent procedures like create.
By definition itself, search is a read only operation, which is used to request for
a collection of resources, filtered and based on some criteria. Therefore, the
GET HTTP method for the search feature is an obvious choice. However, with
GET, there is a constraint with respect to URL size if complex criteria are added
in the URL.




                                                 URL Representation
Query Criteria vs. Embedded Criteria
It is important to discuss this using an example. Say a user wishes to search for
four-doored sedan cars of blue color? What will the resource URL for this
request look like? Indicated below are two URLs that are syntactically different
but semantically the same:

    •   /cars/?color=blue&type=sedan&doors=4
    •   /cars/color:blue/type:sedan/doors:4

Both of the above URLs conform to the RESTful way of representing a resource
query, but are represented differently. While the first one uses the URL query
criteria to add filtering details, the latter follows an embedded URL approach.
The embedded URL approach is more readable and can take advantage of the
native caching mechanisms that exist on the web server for HTTP traffic.
However, this approach limits the user to provide parameters in a specific order.
Wrong parameter positions will cause an error or unwanted behavior. The two
instructions stated below look the same, but may not give the correct results

    •   /cars/color:red/type:sedan
    •   /cars/type:sedan/color:red

Also, since there is no standardization for embedding criteria, people can device
their own ways of representation.

It is imperative therefore, to consider the query criteria approach over the
embedded URL approach, though the representation is a bit complex and lacks
readability.



                                                                                          4
Simplifying RESTful Search


                 Modeling Filter Criteria
                 A search-results page is fundamentally RESTful even though its URL identifies a
                 query. The URL can incorporate SQL-like elements. While SQL is meant to filter
                 data fetched from the relational data, the new modeling language can filter data
                 from the hierarchical set of resources. This language can help in devising a
                 mechanism to communicate complex search requirements over URLs. In this
                 section, two such styles are discussed in detail.


                 Feed Item Query Language (FIQL)
                 The Feed Item Query Language (FIQL, pronounced ‘fickle’) is a simple but
                 flexible, URI-friendly syntax for expressing filters across the entries in a
                 syndicated feed. These filter expressions can be mapped at any RESTful service
                 and help in modeling complex filters. Provided below are some samples of such
                 web URLs against their respective SQLs.

                SQL                                          REST Search URLs

select * from actors where                  /actors?_s=firstname==PENELOPE;lastname
firstname=’PENELOPE’ and                    ==GUINESS
lastname=’GUINESS’

select * from actors where lastname         /actors?_s=lastname==PEN*
like ‘PEN%’

select * from films where filmid=1          /films?_s=filmid==1;rentalduration!=0
and rentalduration<> 0

select * from films where filmid>=          /films?_s=filmid=ge=995
995

select * from films where release           /film?_s=releasedate=le=2005-05-
date < ‘27/05/2005’                         27T00:00:00.000%2B00:00



                 Resource Query Language (RQL)
                 Resource Query Language (RQL) defines a syntactically simple query language
                 for querying and retrieving resources. RQL is designed to be URI-friendly,
                 particularly as a query component of a URI, and highly extensible. RQL is a
                 superset of HTML’s URL encoding of form values, and a superset of Feed Item
                 Query Language (FIQL). RQL basically consists of a set of nestable named
                 operators which each have a set of arguments and operate on a collection of
                 resources.




                                                                                                        5
Simplifying RESTful Search



                Case study: Apache CXF advance search features
                To support advance search capabilities, Apache CXF introduced FIQL support
                with its JAX-RS implementation since the 2.3.0 release. With this feature, users
                can now express complex search expressions using URI. Provided below is a
                detailed note on how this feature can be used.

                To work with FIQL queries, a SearchContext needs be injected into an
                application code and used to retrieve a SearchCondition representing the
                current FIQL query. This SearchCondition can be used in a number of ways
                for finding the matching data.

@Path("books")
public class Books {
private Map<Long, Book> books;
@Context
private SearchContext context;

@GET
public List<Book>getBook() {

   SearchCondition<Book>sc = searchContext.getCondition(Book.class);
   //SearchConditionismet method can also be used to build a list of
matching beans
   // iterate over all the values in the books map and return a collection
of          matching beans
   List<Book> found = sc.findAll(books.values());
   return found;
}
}


                The SearchCondition can also be used to get to all the search
                requirements (originally expressed in FIQL) and do some manual comparisons
                against the local data. SearchCondition for instance, provides a utility to
                the SQL(String tableName, String... columnNames) method which internally
                introspects all the search expressions constituting a current query and converts
                them into an SQL expression:

// find all conditions with names starting from 'ami'
// and levels greater than 10 :
// ?_s="name==ami*;level=gt=10"
SearchCondition<Book>sc = searchContext.getCondition(Book.class);
assertEquals("SELECT * FROM table
             WHERE
             name LIKE 'ami%'
             AND
             level > '10'",
             sq.toSQL("table"));



                                                                                                        6
Simplifying RESTful Search



                                                                                                                               Summary
                                       Data querying is a critical component of most applications. With the advance of
                                       rich, client-driven Ajax applications and document oriented databases, new
                                       querying techniques are needed; these techniques must be simple but
                                       extensible, designed to work within URIs and query for collections of resources.
                                       The NoSQL movement is opening the way for a more modular approach to
                                       databases, and separating out modeling, validation, and querying concerns from
                                       storage concerns. What is needed however, are new querying approaches to
                                       match more modern architectural design.




   About Impetus
   Impetus Technologies offers Product Engineering and Technology R&D services for software product development.
   With ongoing investments in research and application of emerging technology areas, innovative business models, and
   an agile approach, we partner with our client base comprising large scale ISVs and technology innovators to deliver
   cutting-edge software products. Our expertise spans the domains of Big Data, SaaS, Cloud Computing, Mobility
   Solutions, Test Engineering, Performance Engineering, and Social Media among others.

   Impetus Technologies, Inc.
   5300 Stevens Creek Boulevard, Suite 450, San Jose, CA 95129, USA
   Tel: 408.252.7111 | Email: inquiry@impetus.com
   Regional Development Centers - INDIA: • New Delhi • Bangalore • Indore • Hyderabad
   To know more visit: www.impetus.com



Disclaimers
The information contained in this document is the proprietary and exclusive property of Impetus Technologies Inc. except as otherwise indicated. No part of
this document, in whole or in part, may be reproduced, stored, transmitted, or used for design purposes without the prior written permission of Impetus
Technologies Inc.
                                                                                                                                                              7

Simplifying RESTful Search- Impetus Webinar

  • 1.
    WHITE PAPER Simplifying RESTfulSearch Abstract This white paper talks about how search over REST can be simplified. We are not aiming at developing standards for RESTful search, but will be discussing how this problem can be approached. Impetus Technologies Inc. www.impetus.com January 2012
  • 2.
    Simplifying RESTful Search Table of Contents Introduction ........................................................................................................... 2 Search Requirements ............................................................................................. 3 HTTP Method Selection ......................................................................................... 4 URL Representation ............................................................................................... 4 Query Criteria vs. Embedded Criteria ................................................................ 4 Modeling Filter Criteria ...................................................................................... 5 Feed Item Query Language (FIQL) ..................................................................... 5 Resource Query Language (RQL) ....................................................................... 5 Case study: Apache CXF advance search features ................................................. 6 Summary ................................................................................................................ 7 Introduction The REST architectural pattern is based around two basic principles: 1. Resources as URLs: A resource is something like an entity or a noun in modeling language. Anything on the web is identified as a resource and each unique resource is identified by a unique URL. 2. Operations as HTTP methods: REST leverages existing HTTP methods, particularly GET, PUT, POST, and DELETE which map to a resource’s read, create, modify and removal operations, respectively. Any action performed by a client over HTTP, contains a URL and an HTTP method. The URL represents the resource and the HTTP method represents the action which needs to be performed over the resource. Being a broad architectural style, REST always has different interpretations. The ambiguity is exacerbated by the fact that there are not nearly enough HTTP methods to support common operations. One of the most common examples is the lack of a ‘search’ method. While search is one of the most extensively used features across different applications, there have been no standards for implementing this feature. Due to this, people tend to design search in different ways. Given that REST aims to unify service architecture, any ambiguity must be seen as weakening the argument for REST. 2
  • 3.
    Simplifying RESTful Search Search Requirements Search is the most used feature across different web applications, and supports almost similar features around different applications. Listed below are some of the common constituents of search features: • Search based on one or more criteria at a time - Search red colored cars of type hatchback color=red && type=hatchback • Relational and conditional operator support - Search red or black car with mileage greater than 10 Colour=red|black&& mileage > 10 • Wild card search - Search car manufactured from company name starting with M company=M* • Pagination - List all cars but fetch 100 results at a time upperLimit=200 &&lowerLimit=101 • Range searches - Get me all the cars launched between 2000 and 2010 launch year between (2000, 2010) When we support search with such features, the search interface design itself becomes complex. And when implemented in a REST framework, meeting all these requirements (whilestill conforming to REST!) is challenging. Coming back to the basic REST principles, there are two questions that need to be answered: 1. Which HTTP method should be used for ‘search’? 2. How can an effective resource URL be created for search? a. Query parameters versus Embedded URLs b. Modeling filter criteria 3
  • 4.
    Simplifying RESTful Search HTTP Method Selection Effectively, REST categorizes operations by their nature and associates well- defined semantics with these categories. The idempotent operations are GET, PUT, and DELETE (GET for read-only, PUT for update, DELETE for remove), while the POST method is used for non-idempotent procedures like create. By definition itself, search is a read only operation, which is used to request for a collection of resources, filtered and based on some criteria. Therefore, the GET HTTP method for the search feature is an obvious choice. However, with GET, there is a constraint with respect to URL size if complex criteria are added in the URL. URL Representation Query Criteria vs. Embedded Criteria It is important to discuss this using an example. Say a user wishes to search for four-doored sedan cars of blue color? What will the resource URL for this request look like? Indicated below are two URLs that are syntactically different but semantically the same: • /cars/?color=blue&type=sedan&doors=4 • /cars/color:blue/type:sedan/doors:4 Both of the above URLs conform to the RESTful way of representing a resource query, but are represented differently. While the first one uses the URL query criteria to add filtering details, the latter follows an embedded URL approach. The embedded URL approach is more readable and can take advantage of the native caching mechanisms that exist on the web server for HTTP traffic. However, this approach limits the user to provide parameters in a specific order. Wrong parameter positions will cause an error or unwanted behavior. The two instructions stated below look the same, but may not give the correct results • /cars/color:red/type:sedan • /cars/type:sedan/color:red Also, since there is no standardization for embedding criteria, people can device their own ways of representation. It is imperative therefore, to consider the query criteria approach over the embedded URL approach, though the representation is a bit complex and lacks readability. 4
  • 5.
    Simplifying RESTful Search Modeling Filter Criteria A search-results page is fundamentally RESTful even though its URL identifies a query. The URL can incorporate SQL-like elements. While SQL is meant to filter data fetched from the relational data, the new modeling language can filter data from the hierarchical set of resources. This language can help in devising a mechanism to communicate complex search requirements over URLs. In this section, two such styles are discussed in detail. Feed Item Query Language (FIQL) The Feed Item Query Language (FIQL, pronounced ‘fickle’) is a simple but flexible, URI-friendly syntax for expressing filters across the entries in a syndicated feed. These filter expressions can be mapped at any RESTful service and help in modeling complex filters. Provided below are some samples of such web URLs against their respective SQLs. SQL REST Search URLs select * from actors where /actors?_s=firstname==PENELOPE;lastname firstname=’PENELOPE’ and ==GUINESS lastname=’GUINESS’ select * from actors where lastname /actors?_s=lastname==PEN* like ‘PEN%’ select * from films where filmid=1 /films?_s=filmid==1;rentalduration!=0 and rentalduration<> 0 select * from films where filmid>= /films?_s=filmid=ge=995 995 select * from films where release /film?_s=releasedate=le=2005-05- date < ‘27/05/2005’ 27T00:00:00.000%2B00:00 Resource Query Language (RQL) Resource Query Language (RQL) defines a syntactically simple query language for querying and retrieving resources. RQL is designed to be URI-friendly, particularly as a query component of a URI, and highly extensible. RQL is a superset of HTML’s URL encoding of form values, and a superset of Feed Item Query Language (FIQL). RQL basically consists of a set of nestable named operators which each have a set of arguments and operate on a collection of resources. 5
  • 6.
    Simplifying RESTful Search Case study: Apache CXF advance search features To support advance search capabilities, Apache CXF introduced FIQL support with its JAX-RS implementation since the 2.3.0 release. With this feature, users can now express complex search expressions using URI. Provided below is a detailed note on how this feature can be used. To work with FIQL queries, a SearchContext needs be injected into an application code and used to retrieve a SearchCondition representing the current FIQL query. This SearchCondition can be used in a number of ways for finding the matching data. @Path("books") public class Books { private Map<Long, Book> books; @Context private SearchContext context; @GET public List<Book>getBook() { SearchCondition<Book>sc = searchContext.getCondition(Book.class); //SearchConditionismet method can also be used to build a list of matching beans // iterate over all the values in the books map and return a collection of matching beans List<Book> found = sc.findAll(books.values()); return found; } } The SearchCondition can also be used to get to all the search requirements (originally expressed in FIQL) and do some manual comparisons against the local data. SearchCondition for instance, provides a utility to the SQL(String tableName, String... columnNames) method which internally introspects all the search expressions constituting a current query and converts them into an SQL expression: // find all conditions with names starting from 'ami' // and levels greater than 10 : // ?_s="name==ami*;level=gt=10" SearchCondition<Book>sc = searchContext.getCondition(Book.class); assertEquals("SELECT * FROM table WHERE name LIKE 'ami%' AND level > '10'", sq.toSQL("table")); 6
  • 7.
    Simplifying RESTful Search Summary Data querying is a critical component of most applications. With the advance of rich, client-driven Ajax applications and document oriented databases, new querying techniques are needed; these techniques must be simple but extensible, designed to work within URIs and query for collections of resources. The NoSQL movement is opening the way for a more modular approach to databases, and separating out modeling, validation, and querying concerns from storage concerns. What is needed however, are new querying approaches to match more modern architectural design. About Impetus Impetus Technologies offers Product Engineering and Technology R&D services for software product development. With ongoing investments in research and application of emerging technology areas, innovative business models, and an agile approach, we partner with our client base comprising large scale ISVs and technology innovators to deliver cutting-edge software products. Our expertise spans the domains of Big Data, SaaS, Cloud Computing, Mobility Solutions, Test Engineering, Performance Engineering, and Social Media among others. Impetus Technologies, Inc. 5300 Stevens Creek Boulevard, Suite 450, San Jose, CA 95129, USA Tel: 408.252.7111 | Email: inquiry@impetus.com Regional Development Centers - INDIA: • New Delhi • Bangalore • Indore • Hyderabad To know more visit: www.impetus.com Disclaimers The information contained in this document is the proprietary and exclusive property of Impetus Technologies Inc. except as otherwise indicated. No part of this document, in whole or in part, may be reproduced, stored, transmitted, or used for design purposes without the prior written permission of Impetus Technologies Inc. 7