Wealthfront's Query Engine
Upcoming SlideShare
Loading in...5
×
 

Wealthfront's Query Engine

on

  • 540 views

Presentation of Wealthfront's Service Framework and Standardized RPC given at Square in March 2012. The majority of the slides comes from an internal presentation I gave in February 2011.

Presentation of Wealthfront's Service Framework and Standardized RPC given at Square in March 2012. The majority of the slides comes from an internal presentation I gave in February 2011.

Statistics

Views

Total Views
540
Views on SlideShare
540
Embed Views
0

Actions

Likes
2
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Wealthfront's Query Engine Wealthfront's Query Engine Presentation Transcript

    • Wealthfront’s  Query  Engine  Service  Framework  and  Standardized  RPC   julien@wealthfront.com   March  8th,  2012  
    • Roadmap  •  Query  Engine  •  Queries  •  Services  •  Remote  Query  InvocaKon  
    • The  Query  Engine  is  •  a  plaMorm  to  build  distributed  services  quickly  •  designed  with  testability  in  mind  •  powering  all  of  Wealthfront’s  backend  services  •  running  on  the  JVM  
    • The  Query  Engine  is  not  •  a  web  framework  •  a  RESTful  web  service   –  a  query  is  a  funcKon,  not  a  resource  •  Java-­‐specific  
    • Queries  •  First-­‐class  ciKzens   –  Queries  can  be  passed  around  and  later  invoked  •  Serializable   –  Queries  can  be  persisted  •  Closed  with  their  dependencies  •  Composable  units  of  work   –  Queries  can  invoke  other  queries   –  Queries  can  produce  other  queries  •  Entry  points  into  our  backend  services   –  Services  can  invoke  queries  on  other  services   –  Queries  can  also  be  invoked  from  the  command  line  
    • Queries  (concretely)  •  Queries  are  classes  •  Constructors  define  input  parameters  •  Instances  are  invokable  using  a  driver  •  Dependencies  can  be  requested  at  run-­‐Kme  •  InvocaKons  produce  a  result  •  Queries  are  easy  to  test  
    • UNIX  Processes   stdin   Environment   Process  stdout   Return  code   stderr  
    • Queries  Arguments   Dependencies   Query   Result   ExcepKon  
    • Invoking  a  Query   Query  Class  Arguments   Query  Instance   Dependencies   Driver   Result  
    • Invoking  a  Query  Query  Instance   Dependencies   Run   Scoping   Monitoring   Retrying   TransacKng   InjecKng   Result  
    • Services  •  A  collecKon  of  Queries   –  Usually  with  a  similar  purpose,  e.g.  all  the  queries   related  to  customer  management  •  Able  to  saKsfy  the  dependencies  required  by   its  Queries   –  E.g.  access  to  the  customer  database,  connecKon   to  the  NASDAQ  NLS  feed,  …  •  Queries  can  be  installed  in  different  services  
    • Remote  Query  InvocaKon  •  Queries  are  remotely  invoked  by  doing  an   HTTP  POST  request  •  Arguments  are  encoded  in  the  HTTP  request  •  Results  are  returned  in  the  HTTP  response  
    • Request  SerializaKon  •  Most  of  our  services  rely  on  the  so-­‐called   “qp0p1”  serializaKon   –  The  simple  name  of  the  invoked  query  is  passed   with  the  q  parameter   –  The  nth  argument  is  passed  with  the  pnth   parameter   –  Arguments  are  serialized  and  de-­‐serialized  to  and   from  strings  using  converters  
    • Hello(@OpKonal(“World”)  String):  String   Instan&a&on   Serializa&on   new  Hello(“Bob”)   q=Hello&p0=Bob   new  Hello(null)   q=Hello  
    • [julien@glados  ~]$  curl  um0:8085  -­‐-­‐data  q=Hello&p0=Bob’  -­‐v  >  POST  /  HTTP/1.1>  Content-­‐Length:  14  >  Content-­‐Type:  applicaKon/x-­‐www-­‐form-­‐urlencoded  >    >  q=Hello&p0=Bob  <  HTTP/1.1  200  OK  <  X-­‐KC-­‐TraceToken:  962dcf36-­‐9b25-­‐4731-­‐a022-­‐e054b925637c  <  Server:  kawala  <  Content-­‐Length:  11  Hello,  Bob!  
    • Request  De-­‐serializaKon  •  Services’  request  interpreters  de-­‐serialize  the   HTTP  POST  requests  and  create  instances  of   the  query  class  using  the  specified  arguments  •  Then,  they  invoke  the  query  instance  using  a   driver  
    • Request  De-­‐serializaKon  HTTP  POST  request   Request  Interpreter   Query  Class   Arguments   Query  Instance   Driver  
    • Smart  Clients  •  Queries  can  be  remotely  invoked  from  Java   code  using  smart  clients  •  Smart  clients  analyze  the  bytecode  to  recover   the  arguments  passed  to  the  query   constructor  and  generate  the  HTTP  POST   request  •  Smart  clients  also  de-­‐serialize  the  result  from   the  HTTP  response  (see  “Result  SerializaKon”)  
    • Smart  Clients   Query  Class  Arguments   Query  Instance   Smart  Client   HTTP  POST  request  
    • The  Life  of  a  Remote  Request   Client   Server   Query  Class  Arguments   Query  Instance   Smart  Client   HTTP  POST  request   Request  Interpreter   Query  Class   Arguments   Query  Instance   Driver  
    • Constructor  Requirements  •  In  order  to  be  analyzable,  query  constructors   must  follow  strict  requirements   –  Basically,  they  should  only  assign  their  arguments   to  fields  hwps://github.com/wealthfront/kawala/wiki/InstanKators  
    • Sugar  for  the  Constructor  •  @OpKonal(“2011-­‐02-­‐25”)  LocalDate  •  @OpKonal(“false”)  boolean  •  OpKon<LocalDate>  •  @PosiKve  int  age  
    • Early  ValidaKon  •  Arguments  are  validated  when  instanKaKng  a   serialized  query  thanks  to  the  converters  [julien@glados  ~]$  ikq  um0  GetUser  foo  HTTP  Error  400:  For  input  string:  "foo”  X-­‐KC-­‐TraceToken:  962dcf36-­‐9b25-­‐4731-­‐a022-­‐e054b925637c  Server:  kawala  ConnecKon:  close  
    • Result  SerializaKon  •  Results  are  usually  serialized  as   –  JSON   –  Protobuf  •  The  service  has  the  responsibility  to  serialize   the  result  of  a  query  with  the  proper   serializaKon  method  •  Smart  clients  rely  on  the  same  logic  to  select   the  proper  de-­‐serializaKon  method  
    • Result  SerializaKon  Return  Type   Addi&onal  Constraints   Serializa&on  ConverKble  type*   JSON  value**  T   T  annotated  with  @EnKty   JSON  object  List<T>   T  annotated  with  @EnKty   JSON  array  Set<T>   T  annotated  with  @EnKty   JSON  array  T  extends  Message   Protobuf  List<T  extends  Message>   Protobuf  array  *  As  specified  in  KachingMarshallers  **  JSON  strings  are  returned  without  surrounding  quotes  
    • The  Life  of  a  Remote  InvocaKon   Client   Server   Query  Class   Request  Interpreter  Arguments   Query  Class   Query  Instance   Arguments   Query  Instance   Smart  Client   Driver   Result   Result   Serializer  
    • Query  InvocaKon  Summary  •  Local  InvocaKon   QueryExecutor  executor  =  …   User  user  =  executor.submit(new  GetUser(Id.<User>  of(10));  •  Remote  InvocaKon   SmartClient<UM>  um  =  …   User  user  =  um.invoke(new  GetUser(Id.<User>  of(10));