Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Making thingsWork Together   QCon NY June 20, 2012                           1	  
Subbu Allamaraju@sallamargithub.com/s3u                   2	  
How to build aninteroperableProgrammable Web?                    3	  
Agreements             4	  
Discovery     Linking, Identity  Schemas, Media Types         FormatsInterfaces (Uniform, RPC)           HTTP            T...
Servers point   of view             6	  
Clients point of view   7	  
Several hundred APIs         Different styles   Written by different teams            Over yearsBuilt under different cons...
Use case:Find things from AFor each thing, find details from BFor each thing, find more details from CMerge results       ...
Use case:Find productsFind dominant categories of productsLook up category infoMerge categories with products             ...
Use case:Get stuff from AIf A is down, try from BAnnotate stuff with other stuff from CIgnore some things from the stuffJo...
[Really important client]Client         Why dont you give me an API optimized         for my use cases?                   ...
[Really important client]Client         Why dont you give me an API optimized         for my use cases?                   ...
Discovery       Linking, Identity    Schemas, Media Types           Formats> Interfaces (Uniform, RPC)             HTTP   ...
Discovery    Linking, Identity> Schemas, Media Types        Formats          HTTP           TCP                         15...
>      Discovery>   Linking, Identity        Formats         HTTP          TCP                        16	  
Formats HTTP  TCP          17	  
LatencySmartClient   Failure Recovery Code     Orchestration                 IO             Formats               HTTP    ...
19	  
20	  
https://github.com/ql-io/ql.io         Platform Engineering                                 21	  
A domain specific language for   HTTP client programming                                 22	  
23	  
24	  
http://ql.io/console                       25	  
Test console               Find script       Compile HTTP interface                     Request state                     ...
1. Interop via HTTP Support the protocol – butdont require style adherence	                                      27	  
create	  table	  mytable1	  	  	  on	  select	  	  	  	  get	  from	  http://...;	  	  create	  table	  mytable2	  	  	  o...
2. Interop using common         formats   Agnostic of domain  specific type systems	                                29	  
/*	  A	  table	  with	  a	  body	  template	  */	  create	  table	  bing.soap.search	  	  	  on	  select	  post	  to	  htt...
<soapenv:Envelope	  xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">	  	  	  <soapenv:Header/>	  	  	  <soapenv:...
-­‐-­‐	  Form	  encoded	  body	  create	  table	  mytable	  	  	  on	  insert	  post	  to	  http://...	  	  	  using	  bod...
create	  table	  {a	  name}	  on	  {select|insert|update|delete}	  	  	  	  {get	  from|post	  to|	  	  	  	  	  put	  to|...
HTTP	  method	  URI	  template	  Headers	  Body	              Media	  type	                                  34	  
-­‐-­‐	  A	  table	  with	  a	  monkey	  patch	  create	  table	  bitly.shorten	  	  	  on	  insert	  get	  from	  "..."	 ...
3. SQL inspired constructs     CRUD operations,      filtering, joins	                               36	  
-­‐-­‐	  Get	  the	  complete	  response	  as	  JSON	  select	  *	  from	  google.geocode	  where	  	  	  address	  =	  Br...
/*	  Join	  datasets	  without	  enforcing	  identity	  	  	  	  	  across	  systems	  */	  	  select	  e.ItemID	  as	  id...
4. OrchestrationImplicit sequential, scatter,  parallel, forks and joins	                                    39	  
-­‐-­‐	  Sequential	  minis	  =	  select	  *	  from	  finditems	  where	  	  	  	  	  	  keywords	  =	  mini	  cooper	  li...
prodid	  =	  select	  ProductID[0].Value	  from	  eBay.FindProducts	  	  	  	  	  	  where	  QueryKeywords	  =	  macbook	 ...
-­‐-­‐	  An	  API	  that	  takes	  one	  ID	  at	  a	  time	  create	  table	  eBay.ProductDetails	  	  	  on	  select	  g...
43	  
5. Failure ModesTimeouts, back-off and      recovery	                           44	  
-­‐-­‐	  Timeout	  on	  idle	  sockets	  	  -­‐-­‐	  Retry	  once	  for	  idempotent	  statements	  select	  id,	  img	  f...
-­‐-­‐	  Use	  a	  fallback	  	  select	  id	  as	  id,	  item_url	  as	  img	  	  	  	  from	  recommendations	  	  	  ti...
6. InterfaceConsumer-specified	                           47	  
var	  Engine	  =	  require(ql.io-­‐engine),	  	  	  	  	  fs	  =	  require(fs);	  var	  engine	  =	  new	  Engine({	  	  	...
Push orchestration to the data center	  •  Reduce HTTP requests•  Reduce bandwidth use                                    ...
/*	  Shows	  daily	  deals	  -­‐	  use	  siteId=0	  for	  US	  and	  siteId=3	  for	  UK.	  */	  resp	  =	  select	  *	  f...
Latency         Failure Recovery          Orchestration          IO workloads             FormatsServer         HTTP      ...
code   https://github.com/ql-io/ql.iodocs/demos   http://ql.ioblog   http://ql-io.github.com                              ...
Upcoming SlideShare
Loading in …5
×

Making Things Work Together

2,768 views

Published on

Published in: Technology
  • Be the first to comment

Making Things Work Together

  1. 1. Making thingsWork Together QCon NY June 20, 2012 1  
  2. 2. Subbu Allamaraju@sallamargithub.com/s3u 2  
  3. 3. How to build aninteroperableProgrammable Web? 3  
  4. 4. Agreements 4  
  5. 5. Discovery Linking, Identity Schemas, Media Types FormatsInterfaces (Uniform, RPC) HTTP TCP 5  
  6. 6. Servers point of view 6  
  7. 7. Clients point of view 7  
  8. 8. Several hundred APIs Different styles Written by different teams Over yearsBuilt under different constraints Built with different goals 8  
  9. 9. Use case:Find things from AFor each thing, find details from BFor each thing, find more details from CMerge results 9  
  10. 10. Use case:Find productsFind dominant categories of productsLook up category infoMerge categories with products 10  
  11. 11. Use case:Get stuff from AIf A is down, try from BAnnotate stuff with other stuff from CIgnore some things from the stuffJoin all 11  
  12. 12. [Really important client]Client Why dont you give me an API optimized for my use cases? Producers   Server [Really important producer] Thanks. Get a number and stand in the line! 12  
  13. 13. [Really important client]Client Why dont you give me an API optimized for my use cases? Producers   Who gets to decide the right thing? Server [Really important producer] Thanks. Get a number and stand in the line! 13  
  14. 14. Discovery Linking, Identity Schemas, Media Types Formats> Interfaces (Uniform, RPC) HTTP TCP 14  
  15. 15. Discovery Linking, Identity> Schemas, Media Types Formats HTTP TCP 15  
  16. 16. > Discovery> Linking, Identity Formats HTTP TCP 16  
  17. 17. Formats HTTP TCP 17  
  18. 18. LatencySmartClient Failure Recovery Code Orchestration IO Formats HTTP TCP 18  
  19. 19. 19  
  20. 20. 20  
  21. 21. https://github.com/ql-io/ql.io Platform Engineering 21  
  22. 22. A domain specific language for HTTP client programming 22  
  23. 23. 23  
  24. 24. 24  
  25. 25. http://ql.io/console 25  
  26. 26. Test console Find script Compile HTTP interface Request state HTTP client Evented orchestration TransformationsA runtime designed for IO workloads Runs on node.js 26  
  27. 27. 1. Interop via HTTP Support the protocol – butdont require style adherence   27  
  28. 28. create  table  mytable1      on  select        get  from  http://...;    create  table  mytable2      on  insert      get  from  http://...&do=create          create  table  mytable3      on  delete      post  to  http://...      using  headers  X-­‐Do  =  Delete     28  
  29. 29. 2. Interop using common formats Agnostic of domain specific type systems   29  
  30. 30. /*  A  table  with  a  body  template  */  create  table  bing.soap.search      on  select  post  to  http://api.bing.net/soap.asmx      using  defaults  appid  =  xxx      using  bodyTemplate  bing.xml.mu        type  application/xml      resultset  soapenv:Envelope.soapenv:Body.SearchResponse.parameters.Web.Results.WebResult;   30  
  31. 31. <soapenv:Envelope  xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">      <soapenv:Header/>      <soapenv:Body>          <SearchRequest  xmlns="http://schemas.microsoft.com/LiveSearch/2008/03/Search">              <parameters>              {{#params}}              <Query>{{q}}</Query>              <AppId>{{appid}}</AppId>              {{/params}}              <Sources>                  <SourceType>Web</SourceType>              </Sources>              </parameters>          </SearchRequest>      </soapenv:Body>  </soapenv:Envelope>   31  
  32. 32. -­‐-­‐  Form  encoded  body  create  table  mytable      on  insert  post  to  http://...      using  bodyTemplate  somefile.txt          type  application/x-­‐www-­‐form-­‐urlencoded    -­‐-­‐  Or  JSON  create  table  mytable      on  update  put  to  http://...      using  bodyTemplate  file.json  type  application/json   32  
  33. 33. create  table  {a  name}  on  {select|insert|update|delete}        {get  from|post  to|          put  to|delete|patch}          {a  URI  template}  using  headers  {name}={value}*  using  bodyTemplate={a  file}                type={media  type}   33  
  34. 34. HTTP  method  URI  template  Headers  Body   Media  type   34  
  35. 35. -­‐-­‐  A  table  with  a  monkey  patch  create  table  bitly.shorten      on  insert  get  from  "..."          using  patch  bitly.js      on  select  get  from  "..."          using  patch  bitly.js    /*  bitly.js  A  Javascript  monkey  patch  */  exports[patch  status]  =  function(options)  {          return  args.body  ?                args.body.status_code  :  200;    };   35  
  36. 36. 3. SQL inspired constructs CRUD operations, filtering, joins   36  
  37. 37. -­‐-­‐  Get  the  complete  response  as  JSON  select  *  from  google.geocode  where      address  =  Brooklyn,  NY    -­‐-­‐  Project  fields  select  ProductID[0].Value,        DetailsURL,  StockPhotoURL      from  eBay.FindProducts        where  QueryKeywords=mini  cooper   37  
  38. 38. /*  Join  datasets  without  enforcing  identity          across  systems  */    select  e.ItemID  as  id,  e.Title  as  title,        e.ViewItemURLForNaturalSearch  as  url,        g.geometry.location  as  latlng      from  details    as  e,  google.geocode  as  g      where  e.itemId  in  (select  itemId  from        finditems  where  keywords  =  "iPad")      and  g.address  =  e.Location       38  
  39. 39. 4. OrchestrationImplicit sequential, scatter, parallel, forks and joins   39  
  40. 40. -­‐-­‐  Sequential  minis  =  select  *  from  finditems  where            keywords  =  mini  cooper  limit  10;  return  select  PictureURL  from  details  where            itemId  =  "{minis.itemId}";    -­‐-­‐  Or  parallel  keyword  =  "ql.io";  web  =  select  *  from  bing.search  where  q  =  "{keyword}";  tweets  =  select  id  as  id,  from_user_name  as  user_name,          text  as  text    from  twitter.search  where  q  =  "ql.io";      return  {      "keyword":  "{keyword}",      "web":  "{web}",      "tweets":  "{tweets}"  }   40  
  41. 41. prodid  =  select  ProductID[0].Value  from  eBay.FindProducts            where  QueryKeywords  =  macbook  pro;    -­‐-­‐  Scatters  "n"  requests  details  =  select  *  from  eBay.ProductDetails  where          ProductID  in  ({prodid})  and            ProductType  =  Reference;    -­‐-­‐  Scatters  "n"  requests  reviews  =  select  *  from  eBay.ProductReviews  where          ProductID  in  ({prodid})  and            ProductType  =  Reference;    -­‐-­‐  Gather,  merge,  and  join  return  select  d.ProductID[0].Value  as  id,            d.Title  as  title,  d.ReviewCount  as  reviewCount,          r.ReviewDetails.AverageRating  as  rating      from  details  as  d,  reviews  as  r          where  d.ProductID[0].Value  =  r.ProductID.Value   41  
  42. 42. -­‐-­‐  An  API  that  takes  one  ID  at  a  time  create  table  eBay.ProductDetails      on  select  get  from  "...&...={^ProductID}"      ...        resultset  Product;      -­‐-­‐  A  batch  API  –  takes  upto  20  IDs  create  table  details      on  select  get  from  "...&ItemID={20|itemId}      ...      resultset  Item;       42  
  43. 43. 43  
  44. 44. 5. Failure ModesTimeouts, back-off and recovery   44  
  45. 45. -­‐-­‐  Timeout  on  idle  sockets    -­‐-­‐  Retry  once  for  idempotent  statements  select  id,  img  from  recommendations      timeout  100    -­‐-­‐  On  three  successive  failures,  backoff    -­‐-­‐  for  100  –  2000  msec  select  id,  img  from  recommendations      timeout  100  minDelay  100  maxDelay  2000     45  
  46. 46. -­‐-­‐  Use  a  fallback    select  id  as  id,  item_url  as  img        from  recommendations      timeout  100        ||      select  ID  as  id,  deal_url  as  img          from  deals;   46  
  47. 47. 6. InterfaceConsumer-specified   47  
  48. 48. var  Engine  =  require(ql.io-­‐engine),          fs  =  require(fs);  var  engine  =  new  Engine({          tables  :  __dirname  +  /../tables,          config:  __dirname  +  /../config/dev.json  });  var  script  =  fs.readFileSync(__dirname  +  /some.ql,  UTF-­‐8);    //  Exec  the  script  engine.execute(script,  function(emitter)  {          emitter.on(end,  function(err,  result)  {                  result.body.forEach(function(row)  {                          console.log(row);                  });          });  });     48  
  49. 49. Push orchestration to the data center  •  Reduce HTTP requests•  Reduce bandwidth use 49  
  50. 50. /*  Shows  daily  deals  -­‐  use  siteId=0  for  US  and  siteId=3  for  UK.  */  resp  =  select  *  from  dailydeals  where  siteId="{siteId}";  deals  =  "{resp.$..Item}";  itemDetails  =  select  ItemID  as  itemId,  Title  as  title,        GalleryURL  as  pic,  Seller.UserID  as  sellerUserId,            Seller.FeedbackScore  as  rating,        Seller.PositiveFeedbackPercent  as  feedback,      HitCount  as  hits  from  details        where  itemId  in  (deals.ItemID);    return  itemDetails  via  route  /deals/{siteId}  using  method  get;   50  
  51. 51. Latency Failure Recovery Orchestration IO workloads FormatsServer HTTP TCP 51  
  52. 52. code https://github.com/ql-io/ql.iodocs/demos http://ql.ioblog http://ql-io.github.com 52  

×