Wealthfront’s	  Query	  Engine	  Service	  Framework	  and	  Standardized	  RPC	              julien@wealthfront.com	     ...
Roadmap	  •    Query	  Engine	  •    Queries	  •    Services	  •    Remote	  Query	  InvocaKon	  
The	  Query	  Engine	  is	  •    a	  plaMorm	  to	  build	  distributed	  services	  quickly	  •    designed	  with	  test...
The	  Query	  Engine	  is	  not	  •  a	  web	  framework	  •  a	  RESTful	  web	  service	      –  a	  query	  is	  a	  fu...
Queries	  •  First-­‐class	  ciKzens	       –  Queries	  can	  be	  passed	  around	  and	  later	  invoked	  •  Serializa...
Queries	  (concretely)	  •    Queries	  are	  classes	  •    Constructors	  define	  input	  parameters	  •    Instances	  ...
UNIX	  Processes	               stdin	                   Environment	                           Process	  stdout	         ...
Queries	  Arguments	                    Dependencies	                     Query	    Result	                       ExcepKon...
Invoking	  a	  Query	                     Query	  Class	  Arguments	                   Query	  Instance	     Dependencies	...
Invoking	  a	  Query	  Query	  Instance	                                         Dependencies	                            ...
Services	  •  A	  collecKon	  of	  Queries	      –  Usually	  with	  a	  similar	  purpose,	  e.g.	  all	  the	  queries	 ...
Remote	  Query	  InvocaKon	  •  Queries	  are	  remotely	  invoked	  by	  doing	  an	     HTTP	  POST	  request	  •  Argum...
Request	  SerializaKon	  •  Most	  of	  our	  services	  rely	  on	  the	  so-­‐called	     “qp0p1”	  serializaKon	      –...
Hello(@OpKonal(“World”)	  String):	  String	            Instan&a&on	                Serializa&on	            new	  Hello(“...
[julien@glados	  ~]$	  curl	  um0:8085	  -­‐-­‐data	  q=Hello&p0=Bob’	  -­‐v	  >	  POST	  /	  HTTP/1.1>	  Content-­‐Length...
Request	  De-­‐serializaKon	  •  Services’	  request	  interpreters	  de-­‐serialize	  the	     HTTP	  POST	  requests	  a...
Request	  De-­‐serializaKon	  HTTP	  POST	  request	     Request	  Interpreter	                                           ...
Smart	  Clients	  •  Queries	  can	  be	  remotely	  invoked	  from	  Java	     code	  using	  smart	  clients	  •  Smart	...
Smart	  Clients	                    Query	  Class	  Arguments	                  Query	  Instance	                         ...
The	  Life	  of	  a	  Remote	  Request	                       Client	                                                     ...
Constructor	  Requirements	  •  In	  order	  to	  be	  analyzable,	  query	  constructors	     must	  follow	  strict	  re...
Sugar	  for	  the	  Constructor	  •    @OpKonal(“2011-­‐02-­‐25”)	  LocalDate	  •    @OpKonal(“false”)	  boolean	  •    Op...
Early	  ValidaKon	  •  Arguments	  are	  validated	  when	  instanKaKng	  a	     serialized	  query	  thanks	  to	  the	  ...
Result	  SerializaKon	  •  Results	  are	  usually	  serialized	  as	      –  JSON	      –  Protobuf	  •  The	  service	  ...
Result	  SerializaKon	  Return	  Type	                         Addi&onal	  Constraints	                Serializa&on	  Conv...
The	  Life	  of	  a	  Remote	  InvocaKon	                        Client	                                                  ...
Query	  InvocaKon	  Summary	  •  Local	  InvocaKon	     QueryExecutor	  executor	  =	  …	     User	  user	  =	  executor.s...
Upcoming SlideShare
Loading in …5
×

Wealthfront's Query Engine

916 views

Published on

Presentation of Wealthfront's Service Framework and Standardized RPC given at Square in March 2012. The majority of the slides comes from an internal presentation I gave in February 2011.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
916
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Wealthfront's Query Engine

  1. 1. Wealthfront’s  Query  Engine  Service  Framework  and  Standardized  RPC   julien@wealthfront.com   March  8th,  2012  
  2. 2. Roadmap  •  Query  Engine  •  Queries  •  Services  •  Remote  Query  InvocaKon  
  3. 3. The  Query  Engine  is  •  a  plaMorm  to  build  distributed  services  quickly  •  designed  with  testability  in  mind  •  powering  all  of  Wealthfront’s  backend  services  •  running  on  the  JVM  
  4. 4. The  Query  Engine  is  not  •  a  web  framework  •  a  RESTful  web  service   –  a  query  is  a  funcKon,  not  a  resource  •  Java-­‐specific  
  5. 5. Queries  •  First-­‐class  ciKzens   –  Queries  can  be  passed  around  and  later  invoked  •  Serializable   –  Queries  can  be  persisted  •  Closed  with  their  dependencies  •  Composable  units  of  work   –  Queries  can  invoke  other  queries   –  Queries  can  produce  other  queries  •  Entry  points  into  our  backend  services   –  Services  can  invoke  queries  on  other  services   –  Queries  can  also  be  invoked  from  the  command  line  
  6. 6. Queries  (concretely)  •  Queries  are  classes  •  Constructors  define  input  parameters  •  Instances  are  invokable  using  a  driver  •  Dependencies  can  be  requested  at  run-­‐Kme  •  InvocaKons  produce  a  result  •  Queries  are  easy  to  test  
  7. 7. UNIX  Processes   stdin   Environment   Process  stdout   Return  code   stderr  
  8. 8. Queries  Arguments   Dependencies   Query   Result   ExcepKon  
  9. 9. Invoking  a  Query   Query  Class  Arguments   Query  Instance   Dependencies   Driver   Result  
  10. 10. Invoking  a  Query  Query  Instance   Dependencies   Run   Scoping   Monitoring   Retrying   TransacKng   InjecKng   Result  
  11. 11. Services  •  A  collecKon  of  Queries   –  Usually  with  a  similar  purpose,  e.g.  all  the  queries   related  to  customer  management  •  Able  to  saKsfy  the  dependencies  required  by   its  Queries   –  E.g.  access  to  the  customer  database,  connecKon   to  the  NASDAQ  NLS  feed,  …  •  Queries  can  be  installed  in  different  services  
  12. 12. Remote  Query  InvocaKon  •  Queries  are  remotely  invoked  by  doing  an   HTTP  POST  request  •  Arguments  are  encoded  in  the  HTTP  request  •  Results  are  returned  in  the  HTTP  response  
  13. 13. Request  SerializaKon  •  Most  of  our  services  rely  on  the  so-­‐called   “qp0p1”  serializaKon   –  The  simple  name  of  the  invoked  query  is  passed   with  the  q  parameter   –  The  nth  argument  is  passed  with  the  pnth   parameter   –  Arguments  are  serialized  and  de-­‐serialized  to  and   from  strings  using  converters  
  14. 14. Hello(@OpKonal(“World”)  String):  String   Instan&a&on   Serializa&on   new  Hello(“Bob”)   q=Hello&p0=Bob   new  Hello(null)   q=Hello  
  15. 15. [julien@glados  ~]$  curl  um0:8085  -­‐-­‐data  q=Hello&p0=Bob’  -­‐v  >  POST  /  HTTP/1.1>  Content-­‐Length:  14  >  Content-­‐Type:  applicaKon/x-­‐www-­‐form-­‐urlencoded  >    >  q=Hello&p0=Bob  <  HTTP/1.1  200  OK  <  X-­‐KC-­‐TraceToken:  962dcf36-­‐9b25-­‐4731-­‐a022-­‐e054b925637c  <  Server:  kawala  <  Content-­‐Length:  11  Hello,  Bob!  
  16. 16. Request  De-­‐serializaKon  •  Services’  request  interpreters  de-­‐serialize  the   HTTP  POST  requests  and  create  instances  of   the  query  class  using  the  specified  arguments  •  Then,  they  invoke  the  query  instance  using  a   driver  
  17. 17. Request  De-­‐serializaKon  HTTP  POST  request   Request  Interpreter   Query  Class   Arguments   Query  Instance   Driver  
  18. 18. Smart  Clients  •  Queries  can  be  remotely  invoked  from  Java   code  using  smart  clients  •  Smart  clients  analyze  the  bytecode  to  recover   the  arguments  passed  to  the  query   constructor  and  generate  the  HTTP  POST   request  •  Smart  clients  also  de-­‐serialize  the  result  from   the  HTTP  response  (see  “Result  SerializaKon”)  
  19. 19. Smart  Clients   Query  Class  Arguments   Query  Instance   Smart  Client   HTTP  POST  request  
  20. 20. The  Life  of  a  Remote  Request   Client   Server   Query  Class  Arguments   Query  Instance   Smart  Client   HTTP  POST  request   Request  Interpreter   Query  Class   Arguments   Query  Instance   Driver  
  21. 21. Constructor  Requirements  •  In  order  to  be  analyzable,  query  constructors   must  follow  strict  requirements   –  Basically,  they  should  only  assign  their  arguments   to  fields  hwps://github.com/wealthfront/kawala/wiki/InstanKators  
  22. 22. Sugar  for  the  Constructor  •  @OpKonal(“2011-­‐02-­‐25”)  LocalDate  •  @OpKonal(“false”)  boolean  •  OpKon<LocalDate>  •  @PosiKve  int  age  
  23. 23. Early  ValidaKon  •  Arguments  are  validated  when  instanKaKng  a   serialized  query  thanks  to  the  converters  [julien@glados  ~]$  ikq  um0  GetUser  foo  HTTP  Error  400:  For  input  string:  "foo”  X-­‐KC-­‐TraceToken:  962dcf36-­‐9b25-­‐4731-­‐a022-­‐e054b925637c  Server:  kawala  ConnecKon:  close  
  24. 24. Result  SerializaKon  •  Results  are  usually  serialized  as   –  JSON   –  Protobuf  •  The  service  has  the  responsibility  to  serialize   the  result  of  a  query  with  the  proper   serializaKon  method  •  Smart  clients  rely  on  the  same  logic  to  select   the  proper  de-­‐serializaKon  method  
  25. 25. Result  SerializaKon  Return  Type   Addi&onal  Constraints   Serializa&on  ConverKble  type*   JSON  value**  T   T  annotated  with  @EnKty   JSON  object  List<T>   T  annotated  with  @EnKty   JSON  array  Set<T>   T  annotated  with  @EnKty   JSON  array  T  extends  Message   Protobuf  List<T  extends  Message>   Protobuf  array  *  As  specified  in  KachingMarshallers  **  JSON  strings  are  returned  without  surrounding  quotes  
  26. 26. The  Life  of  a  Remote  InvocaKon   Client   Server   Query  Class   Request  Interpreter  Arguments   Query  Class   Query  Instance   Arguments   Query  Instance   Smart  Client   Driver   Result   Result   Serializer  
  27. 27. Query  InvocaKon  Summary  •  Local  InvocaKon   QueryExecutor  executor  =  …   User  user  =  executor.submit(new  GetUser(Id.<User>  of(10));  •  Remote  InvocaKon   SmartClient<UM>  um  =  …   User  user  =  um.invoke(new  GetUser(Id.<User>  of(10));  

×