Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to implement a truly modular ecommerce platform on the example of Spryker at code.talks commerce special 2017


Published on

The focus of my presentation are patterns, principles and especially practical solutions how to implement an ecommerce platform that consists of hundreds decoupled and coherent modules. This talk is suitable for senior developers who design large scale systems and are interested in architectural insights and inspirations.
Keywords: SOLID, Packaging Principles, Design Patterns, Macro- and Microservices, Semantic Versioning, Composer

Published in: Software
  • Be the first to comment

How to implement a truly modular ecommerce platform on the example of Spryker at code.talks commerce special 2017

  1. 1. How  to  implement  a  truly  modular   ecommerce  pla2orm   code.talks  commerce  special  2017  
  2. 2. 2017  Spryker  Systems  GmbH   Mo>va>on  
  3. 3. 2017  Spryker  Systems  GmbH   About  me   Fabian  Wesner   CTO  @  Spryker  Systems     TwiGer:  @FabianWesner     hGp://  
  4. 4. 2017  Spryker  Systems  GmbH   What  is  Spryker?   SPRYKER  is  a  Commerce  Opera>ng  System   for  customer  or  machine  ini>ated  transac>ons.  
  5. 5. 2017  Spryker  Systems  GmbH   Modularity   One  of  Spryker’s  key  characteris>cs:     Modularity       This  talk  is  about  learnings,  tradeoffs,  pa5erns,  tools  and   principles  to  build  a  truly  modular  ecommerce  pla9orm.  
  6. 6. 2017  Spryker  Systems  GmbH   Monolith  or  Modules  or   Microservices?  
  7. 7. 2017  Spryker  Systems  GmbH   Monolith  vs  Modules  vs  Microservices  
  8. 8. 2017  Spryker  Systems  GmbH   Microservices  are  compelling   •  Independent  teams,  hos>ng  and  deployments   •  Reduced  complexity  for  single  services   •  Free  choice  of  technologies….   •  Big  challenge.  New  tools  to  discover.   •  Great  for  conference  talks  
  9. 9. 2017  Spryker  Systems  GmbH   But:  Distributed  compu>ng  is  hard   •  Asynchronous  calls   •  Network  is  not  reliable  and  has  latency   •  Debuging  via  network   •  Performance  issues   •  API  versioning   •  Authen>fica>on   •  Service  discovery   •  Distributed  logging   •  Eventual  consistency,  no  simple  transac>ons   •  Distributed  sets  of  data,  no  straight-­‐forward  joins   •  No  protec>on  by  foreign  key  constraints   •  …  
  10. 10. 2017  Spryker  Systems  GmbH   Good  use  case  for  Microservices   Microservices  can  work  very  well  when  there  are  very  clean  boundaries  and  communica>on  direc>ons.     Example:  Pla,orm  for  Online  Games     Free  choice  of  technology  per  game.   Independent  deployment,  hos>ng,  data-­‐ storage,  etc.     Basic  func>onality  is  implemented  in  a  shared   service  (e.g.  Customer,  Payment,  …)     In  this  scenario  Microservice  is  natural  choice,   even  for  companies  that  are  not  the  scale  of   Google  or  Amazon.  
  11. 11. 2017  Spryker  Systems  GmbH   Bad  use  case  for  Microservices   Microservices  become  pain  in  the  ass  when  the  system  has  high  amount  of  internal  dependencies   which  usually  becomes  visible  in  the  ER  diagram  of  the  database  schema.   Example:  Online-­‐Shop   The  diagram  shows  a  typical   database  schema  of  a  online  shop.       All  rela>onships  are  meaningful  and   need  to  be  protected  by  the   database.     It’s  not  even  big!  
  12. 12. 2017  Spryker  Systems  GmbH   Modularity  
  13. 13. 2017  Spryker  Systems  GmbH   Prinicples  of  modularity   Strong  encapsula/on   Hide  implementa>on  details  inside  components,  leading  to  low  coupling  between  different  parts.   Teams  can  work  in  isola>on  on  decoupled  parts  of  the  system.     Well-­‐defined  interfaces   You  can't  hide  everything  (or  else  your  system  won't  do  anything  meaningful),  so  well-­‐defined  and   stable  APIs  between  components  are  a  must.  A  component  can  be  replaced  by  any  implementa>on   that  conforms  to  the  interface  specifica>on.     Explicit  dependencies   Having  a  modular  system  means  dis>nct  components  must  work  together.  You'd  beGer  have  a  good   way  of  expressing  (and  verifying)  their  rela>onships.   Quoted  from  hGps://­‐vs-­‐microservices  
  14. 14. 2017  Spryker  Systems  GmbH   Bad  news!         PHP  does  not  have  built-­‐in  support  for  modules!     So  we  need  to  solve  it  with  conven>ons,  paGerns  and  the  help  of  composer.  
  15. 15. 2017  Spryker  Systems  GmbH   Programming  modules  
  16. 16. 2017  Spryker  Systems  GmbH   What  is  a  bundle  (~module)?   Spryker’s  defini/on  of  a  bundle     •  “A  bundle  is  a  func>onal  unit!”   •  It  defines  its  internal  API   •  It  defines  its  data  structures   •  It  defines  its  dependencies   •  It  contains  all  required  layers:   •  Presenta>on   •  Communica>on   •  Business   •  Persistence   •  It  contains  all  unit  tests     Bundles  are  small  (~  micro),   coherent  and  loose  coupled.     There  will  be  a  high  numer   of  bundles!  (>  100)     We  need  a  consistent  way   how  to  communicate   between  bundles.   è  
  17. 17. 2017  Spryker  Systems  GmbH   Facades  as  internal  API  for  bundles   Client   Facade! Cart! Facade  pa5ern  is  structural  pa5ern  by  the  GOF.  
  18. 18. 2017  Spryker  Systems  GmbH   Facade  interface  
  19. 19. 2017  Spryker  Systems  GmbH   Bundle  to  Bundle  Communica>on   Facade! Facade! Cart! Calculation! Facade! Discount! calculateDiscounts()!recalculate()!add()!
  20. 20. 2017  Spryker  Systems  GmbH   Facade  interface  challenge   Internal  class  CartOpera>on  programs  against  interface  of  Calcula>onFacade   Where  are  the  bundle  boundaries?  
  21. 21. 2017  Spryker  Systems  GmbH   Facade  interface  challenge   Internal  class  CartOpera>on  programs  against  interface  of  Calcula>onFacade   Where  are  the  bundle  boundaries?  
  22. 22. 2017  Spryker  Systems  GmbH   Facade  interface  challenge   Approach  1:  cart-­‐opera>on  class  and  interface  are  in  same  bundle.   Problem:  The  facade  needs  to  implement  an  interface  from  another  bundle.  
  23. 23. 2017  Spryker  Systems  GmbH   Facade  interface  challenge   Approach  2:  Facade  and  interface  are  in  the  same  bundle   Problem:  The  Opera>on  class  has  a  hard-­‐coded  dependency  to  a  specific   bundle.  
  24. 24. 2017  Spryker  Systems  GmbH   Solu>on  with  bridge  paGern   Solu/on:  Add  a  bridge-­‐paGern  and  a  second  interface  to  solve  the  problem!   The  CartToCalcula>on  contains  only  the  required  methods  (ISP).  
  25. 25. 2017  Spryker  Systems  GmbH   Solu>on  with  bridge  paGern   The  bridge  allows  us  to   use  any  class  that  fits  to   the  interface.     Coupling  in  run-­‐>me  only.   Not  in  development-­‐>me.  
  26. 26. 2017  Spryker  Systems  GmbH   Solu>on  with  bridge  paGern   Facade  of  Calcula>on-­‐bunde  is  located  and  becomes  wrapped  into  the   CartToCalcula>on-­‐bridge.  
  27. 27. 2017  Spryker  Systems  GmbH   Shared  data  structure   Challenge:  Data  needs  to  be  exchanged  between  bundles.   Approach:  Usage  of  arrays!?   public function recalculate(array $quote) { return $this->getFactory()->createStackExecutor()->recalculate($quote); } Not  expressive.   No  clear  structure.  
  28. 28. 2017  Spryker  Systems  GmbH   Shared  data  structure   Challenge:  Data  needs  to  be  exchanged  between  bundles.   Approach:  Data  transfer  objects  (DTOs)  
  29. 29. 2017  Spryker  Systems  GmbH   Shared  data  structure   Challenge:  Data  needs  to  be  exchanged  between  bundles.   Approach:  Data  transfer  objects  (DTOs)  
  30. 30. 2017  Spryker  Systems  GmbH   Shared  data  structure   Challenge:  Where  to  put  these  DTOs?   Approaches:     •  Have  the  same  DTO  in  both  bundle   •  Have  the  DTO  in  the  “main”  bundle  (Which  one  is  it?)  
  31. 31. 2017  Spryker  Systems  GmbH   Shared  data  structure   Solu/on:  Define  the  DTO  in  XML  in  all  related  bundles,  merge  them  and   generate  the  object.   è  
  32. 32. 2017  Spryker  Systems  GmbH   Summary  Communica>on   Bundles  talk  to  each  other  via     Facades  (~  internal  APIs)       Bridges  and  interfaces  connect     the  bundles  in  run-­‐>me  but  keep     them  decoupled  in  development-­‐>me.     Data  is  transported  via  Data  transfer  objects  
  33. 33. 2017  Spryker  Systems  GmbH   Modular  database   In  a  perfect  world  every  bundle  /  service  has  it’s  own  database  and  data  is   only  transmiGed  via  messages.   But…  
  34. 34. 2017  Spryker  Systems  GmbH   Modular  database   In  an  e-­‐commerce  applica>on  things     are  connected.     Real-­‐world  use  cases   •  Show  all  customers  with  their  number  of  sales-­‐orders   •  Discounts  are  valid  only  for  products  with  a  high  stock   •  Discounts  are  valid  only  for  specific  customers   •  Products  have  specific  prices  per  customers  (Typical  B2B  feature)   •  …   What  we  need  is  normalized  database  with  protected  foreign-­‐key  rela>ons!  
  35. 35. 2017  Spryker  Systems  GmbH   Modular  database   Challenge:  How  to  handle  cross-­‐bundle  rela>ons?     Example:  “A  product  has  stock”   •  Table  spy_product  belongs  to  Product-­‐bundle   •  Tables  spy_stock_product  and  spy_stock  belong  to  Stock-­‐bundle  
  36. 36. 2017  Spryker  Systems  GmbH   Modular  database   Solu/on:  Each  bundle  ships  with  it’s  own  schema  defini>on!   All  XML-­‐files  are  merged,  diffed  and  migrated  to  the  database.   When  a  bundle  is  removed  ,  all  the  related  tables  and  rela>ons  are  gone  as  well.  
  37. 37. 2017  Spryker  Systems  GmbH   Modular  database   Solu/on:  Each  bundle  ships  with  it’s  own  schema  defini>on!     Discussion:     •  Every  bundle  can  query  it’s  own  data  only.   •  Cross-­‐bundle  joins  are  possible  but  they  create  a  >ght  coupling.   •  Therefore  we  prefer  communica>on  via  messages.     •  The  main  reason  to  do  cross-­‐bundle  joins  is  performance.  It’s  a  tradeoff.  
  38. 38. 2017  Spryker  Systems  GmbH   Designing  Dependencies  
  39. 39. 2017  Spryker  Systems  GmbH   Designing  dependencies   •  Product  bundle  holds  the  product  en>>es   •  Stock  bundle  manages  the  inventory     What  are  their  dependencies?  
  40. 40. 2017  Spryker  Systems  GmbH   Designing  dependencies   Considera>ons:   •  There  can  be  products  without  stock   •  There  is  never  a  stock  without  a  product       è  Stock-­‐bundle  requires  Product-­‐bundle   (~  Product-­‐bundle  is  responsible  for  Stock-­‐bundle)  
  41. 41. 2017  Spryker  Systems  GmbH   Another  simple  example  (availability)   Availability of a product = Stock – Sold Items Bundle   Responsibility   Availability-­‐bundle   Calculates  the  availability  of  a  product   Oms-­‐bundle   Order  Management  System  knows  the  state  of  an  order-­‐item   Product-­‐bundle     Holds  the  product  en>>es     Stock-­‐bundle     Holds  the  stock  of  a  product     We  need  several  bundles:  
  42. 42. 2017  Spryker  Systems  GmbH   Another  simple  example  (availability)   Availability of a product = Stock – Sold Items What  are  their  dependencies?  
  43. 43. 2017  Spryker  Systems  GmbH   Another  simple  example  (availability)   Availability of a product = Stock – Sold Items
  44. 44. 2017  Spryker  Systems  GmbH   Another  simple  example  (availability)   Availability of a product = Stock – Sold Items
  45. 45. 2017  Spryker  Systems  GmbH   Another  simple  example  (availability)   Availability of a product = Stock – Sold Items
  46. 46. 2017  Spryker  Systems  GmbH   Some>mes  it‘s  not  that  simple   Use  case:  Discounts  are  valid  for  specific  customer-­‐groups         What  are  the  dependencies?  
  47. 47. 2017  Spryker  Systems  GmbH   Some>mes  it‘s  not  that  simple   What  are  the  dependencies?   •  Both  bundles  can  be  used  separately!     •  Discounts  are  possible  without  any  customer-­‐groups  and  vice-­‐versa.     Run-­‐/me  call:  Discount-­‐bundle  needs  to  check  if  current  customer  belongs  to   the  related  Customer-­‐Group   Does  that  mean  that  Discount-­‐bundle  requires  Customer-­‐Group?  No!  
  48. 48. 2017  Spryker  Systems  GmbH   Some>mes  it‘s  not  that  simple   We  put  a  Connector  in  the  middle!   Now  both  bundles  are  independent  from  each  other  BUT  can  s>ll  work   together.     How  does  that  technically  work?  
  49. 49. 2017  Spryker  Systems  GmbH   Some>mes  it‘s  not  that  simple   ✔ The  connector  makes  a  direct  call  to  the  CustomerGroupFacade     ?  But  how  can  we  make  a  call  from  discount-­‐bundle  to  the  connector  with  a   direct  dependy?  
  50. 50. 2017  Spryker  Systems  GmbH   Inversion  of  control   Solu=on:  Inversion  of  control  with  Plugins  
  51. 51. 2017  Spryker  Systems  GmbH   Plugin  configura>on   Solu=on:  Inversion  of  control  with  Plugins   The  plugin  is  configured  into  the  Discount-­‐bundle!  
  52. 52. 2017  Spryker  Systems  GmbH   Learning:  Dependencies  !=  Coupling   Conceptual  dependencies  are  given  and  need  to  be  discovered  by  the  team   •  Stock  requires  Product   •  Product-­‐OpTon  requires  Product   •  Customer-­‐Group  requires  Customer   •  Etc.     Technical  coupling  between  bundles  doesn’t  just  happen!   •  Dependencies  can  be  inverted  if  needed.   •  Dependencies  can  be  mandatory  or  op>onal  
  53. 53. 2017  Spryker  Systems  GmbH   Experience   Although  we  managed  our  dependencies  very  carefully,  we  ended  up  with   this:  
  54. 54. 2017  Spryker  Systems  GmbH   Learning   We  need  to  obey  the  Principles  of  Package  Design!     Principles  of  cohesion   •  The  Release/reuse  equivalence  principle   •  The  Common  reuse  principle   •  The  Common  closure  principle   Principles  of  coupling   •  The  Acyclic  dependencies  principle   •  The  Stable  dependencies  principle   •  The  Stable  abstrac>ons  principle  
  55. 55. 2017  Spryker  Systems  GmbH   Recommended  books   The  classic  book  about  agile   sooware  design  from  Robert  Mar>n.   A  modernized  explana>on  of  SOLID   and  Package  Principles  for  modern  PHP   development  by  MaGhias  Noback.  
  56. 56. 2017  Spryker  Systems  GmbH   The  Acyclic  dependencies  principle   The  dependency  graph  of  packages  must  have  no  cycles.  
  57. 57. 2017  Spryker  Systems  GmbH   The  Acyclic  dependencies  principle  
  58. 58. 2017  Spryker  Systems  GmbH   The  Stable  dependencies  principle   Depend  in  the  direc>on  of  stability.  
  59. 59. 2017  Spryker  Systems  GmbH   What  is  Stability?   A  bundle  is  more  stable  the  more  bundles  requires  it.   Stability = Outgoing Dependencies / (Outgoing + Incoming Dependencies)
  60. 60. 2017  Spryker  Systems  GmbH   Stability  measurement  in  Spryker  
  61. 61. 2017  Spryker  Systems  GmbH   Aoer  cleanup   Dependency  Cleanup  was  possible  by  splipng  bundles  and  inver>ng   dependencies.
  62. 62. 2017  Spryker  Systems  GmbH   Real  world  dependencies  (Spryker)  
  63. 63. 2017  Spryker  Systems  GmbH   Benefits   A  clean  dependency  tree  has  a  lot  of  advantages:     •  Reduced  system  complexity.  Avoidance  of  side-­‐effects.   •  Team  can  work  independently  even  with  large  systems.   •  It  becomes  possible  to  split  the  applica>on  into  services   •  Single  bundles  can  be  released,  replaced  and  extended  
  64. 64. 2017  Spryker  Systems  GmbH   Managing  Dependencies  
  65. 65. 2017  Spryker  Systems  GmbH   Bundle  organiza>on   Every  bundle  has  it’s  own:   •  Repository     •  Seman>c  version  number   •  Composer.json  
  66. 66. 2017  Spryker  Systems  GmbH   Example:  Spryker  Cart  
  67. 67. 2017  Spryker  Systems  GmbH   Composer.json   We  use  Composer.json  in  the  following  way:     Require    For  direct  dependencies  (~  Calls  to  a  facade)     Suggest    For  plugins  that  offer  addi>onal  func>onality     Require-­‐Dev  For  dependencies  that  only  exist  in  tests.  
  68. 68. 2017  Spryker  Systems  GmbH   Challenge:  Slow  composer  update   Spryker  releases  in  >120   bundles  now!     And  many  more  in  the   future!     Problem:  Composer   update  takes  ages….  
  69. 69. 2017  Spryker  Systems  GmbH   Solu>on:  Toran  Proxy   We  run  a  public  Toran  Proxy  that  pre-­‐fetches  the  composer.json   files  from  Github  and  allows  to  execute  composer  update  in  <  1   minute.   In  case  you  prefer  SaaS,  then  public  or  private  Packagist  will  do  the  same  job.  
  70. 70. 2017  Spryker  Systems  GmbH   Challenge:  Core  development   How  does  core  development  work  with  >  120  bundles?     Should  we  commit  to  all  bundles?       What  about  branches  and  tags?       What  about  conflicts?     How  do  release  features?  
  71. 71. 2017  Spryker  Systems  GmbH   Solu>on:  Git  Subtree  Split   Git  has  a  handy  tool  for  this  problem.  Git  Subtree  Split:     •  The  core  team  works  with  a  single  (private  repository).     •  There  is  a  liGle  app  installed  on  a  server  that  reacts  on  every   commit  to  master  and  executes  the  subtree  split  opera>on.     •  The  opera>on  maps  single  directories  to  other  repositories  and   forwards  the  commits.  
  72. 72. 2017  Spryker  Systems  GmbH   Solu>on:  Git  Subtree  Split   You  can  start  with  the  open  source  bash  script  from  Fabien  Potencier:     hGps://  
  73. 73. 2017  Spryker  Systems  GmbH   How  to  release  >  120  bundles?   There  are  two  approaches:     Tags  in  the  source-­‐repository  are  forwarded  to  all  splits   •  Every  bundle  has  all  versions   •  Some  versions  contain  no  changes  for  a  single  bundle  (Phantom  release)   •  Advantage:  There  is  a  global  version  number  (e.g.  Symfony  3.0.0)   •  This  is  how  Symfony  does  it   Tags  in  the  source-­‐repository  are  forwarded  only  to  these  splits  that  contain  a  change   •  Bundles  have  different  versions   •  Advantage:  Bundles  can  be  released  independently  (Atomic  Release)   •  This  is  how  Spryker  does  it  
  74. 74. STRICTLY CONFIDENTIAL 74   You  cannot  solve  21st  century  e-­‐commerce  problems  with  20th  century  technology   @sprysys