Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Develop modern apps using Spring ecosystem at time of BigData

1,104 views

Published on

Slides from JavaDay Kiev 2015 talk

Published in: Software
  • Be the first to comment

Develop modern apps using Spring ecosystem at time of BigData

  1. 1. Develop  modern  apps  using   Spring  ecosystem  at  BigData  era  
  2. 2. LEAD  SOFTWARE  ENGINEER  AT  EPAM  SYSTEMS.   PATIONATE  DEVELOPER,  SPEAKER,  ACTIVE  MEMBER  OF  KIEV   JUG.   PARTICIPATE  IN  DIFFERENT  EDUCATIONAL  INITIATIVES,   ENGINEERING  EVENTS  AND  JCP/ADOPTJSR  PROGRAMS.     OLEG TSAL-TSALKO ABOUT ME
  3. 3. Spring     for  a  long  Mme     no  more  than     just  a  Framework  
  4. 4. Spring  Boot   •  General  purpose  plaRorm  for  Web  app  and   client  app  development  which  makes  complex   apps  development  look  easy     •  Rapid  development  of  diff  type  applicaMons   based  on  solid  Spring  foundaMon   •  No  boilerplate  code.  You  write  only  needfull/ funcMonal  code.  Spring  knows  beVer  how   make  it  all  work.   •  Nevertheless  easy  to  customize  and  extend   •  Simple  and  robust  dependency  management  
  5. 5. [DEMO:]  Tweets  Boot  Data  REST  app   1.  Basic  5  mins  Boot  Data  Rest  app  which  simply  works!   2.  Lets  add  logging  and  external  properMes  support   3.  Lets  add  integraMon  tests   4.  Lets  add  Tweet  lifecycle  event  listeners   5.  Lets  add  Actuator  monitoring  support   6.  Lets  add  much  more  fields  to  Tweet  enMty   7.  Lets  add  Tweet  compact  projecMon   8.  Lets  add  validaMon   9.  Lets  add  addiMonal  stuff  to  Tweet  HATEOUS  json  view   10. Lets  add  more  custom  queries   11. But  beVer  do  not  try  to  make  Java  8  data  and  Mme   types  work))    
  6. 6. Spring  XD   •  General  purpose  plaRorm  for  developing  BigData  apps   for  data  ingesMon,  data  stream  and  batch  processing,   data  export  and  data  analyMcs   •  Scalable  and  fault  tolerant  plaRorm  in  your  service   •  DeclaraMve  processing  workflow  development,   deployment  and  management   •  Huge  number  of  predefined  available  modules  that  can   be  used  straight  away  to  bring  your  processing   pipelines  in  life  in  seconds   •  Easy  to  extend  plaRorm  and  pluggable  custom   modules   •  Build  on  top  of  rich  and  robust  Spring  foundaMon  
  7. 7. Real  world  example  for  XD   Raw  data  is  captured  from  mobile  phones  in  JSON  format  by  a  Spring  XD  cluster     where  several  processes  are  performed.     The  data  is  then  stored  in  an  HDFS  cluster  where  Spring  XD  batch  jobs  use  SQL  via  the  HAWQ     interface  to  HDFS  and  store  the  calculated  reports  in  Redis.     Spring  Boot  is  then  used  with  Angular.js  and  D3.js  to  show  analyMcs  to  end  users.  
  8. 8. [DEMO:]  Spring  XD  toy.  Play  with  me!   1.  Show  XD  bootstrap  in  singlenode  mode  and  deploy  simple   streams.  Show  XD  shell  and  XD  UI  usages.   2.  Show  XD  distributed  mode  configuraMon  with  external   Zookeeper,  DB  and  Redis.  Show  XD  meta  data  in   Zookeeper.  Deploy  same  simple  streams  on  cluster  and   show  XD  cluster  fault  tolerance  and  recovery.   3.  Show  example  of  predefined  XD  module  configuraMon   (e.g.  hdfs)   4.  Create  custom  module  and  upload  it  into  XD   5.  Show  DEMO  for  stream  reading  tweets  from  TwiVer  and   posMng  them  into  REST  client  app  developed  before  via   HTTP  and  wriMng  them  into  HDFS  in  parallel  
  9. 9. Spring  XD  (out  of  scope…)   •  Show  XD  job  running  over  Hadoop   •  Show  RabbitMQ  integraMon   •  Show  DEMO  on  running  either  MapReduce   job  using  Cascading  or  Pig  over  HDFS  tweets   data  OR  running  Spark  app  over  HDFS  tweets   data  
  10. 10. Spring  Boot/XD  enable  twelve  factor  apps   I.  Codebase   One  codebase  tracked  in  revision  control,  many  deploys   II.  Dependencies   Explicitly  declare  and  isolate  dependencies   III.  Config   Store  config  in  the  environment   IV.  Backing  Services   Treat  backing  services  as  aVached  resources   V.  Build,  release,  run   Strictly  separate  build  and  run  stages   VI.  Processes   Execute  the  app  as  one  or  more  stateless  processes   VII.  Port  binding   Export  services  via  port  binding   VIII.  Concurrency   Scale  out  via  the  process  model   IX.  Disposability   Maximize  robustness  with  fast  startup  and  graceful  shutdown   X.  Dev/prod  parity   Keep  development,  staging,  and  producMon  as  similar  as  possible   XI.  Logs   Treat  logs  as  event  streams   XII.  Admin  processes   Run  admin/management  tasks  as  one-­‐off  processes  
  11. 11. Everything  wasn’t  so  smooth  though…   •  SPEL  doesn’t  work  on  filter  with  JSON  elements  access  implicitly  as  it  works  with  transformer   module.  To  make  filter  work  you  need  either  to  convert  JSON  to  Tuple  or  to  use  JSON  Xpath  SPEL   syntax   •  When  transform  JSON  into  Tuple  and  back  WARNings  are  thrown  on  Jackson  transformaMon???   •  In  order  implicit  converMon  to  Tuple  works,  you  need  to  specify  contentType  MIME  header  in  HTTP   request  otherwise  it  can’t  automaMcally  cast  text/plain  MIME  message  to  Tuple   •   If  you  deploy  hVp  module  on  port  which  is  occupied  it  won’t  fail  on  deploy  but  your  stream  will   simply  silently  not  work   •  SomeMmes  container  gets  broken  and  smth  stops  to  work  like  Taps  in  my  case  for  example  and  no   errors  in  logs.  Only  container  restart  helps.   •  Spring  data  rest  doesn’t  expose  enMty  IDs  in  JSON  body  by  default   •  When  Mongo  enMty  save  operaMon  failing    as  a  result  of  HTTP  call  no  excepMons  visible  in  logs,  just   500  ERROR  response  to  client  without  proper  stack  trace,  however  if  you  call  MongoTemplate   directly  you’ll  get  proper  stack  trace.   •  If  HTTP  GET  request  contentType=applicaMon/json  then  response  will  be  rendered  as  applicaMon/x-­‐ spring-­‐data-­‐compact+json,  when  by  default  it  will  be  HAL  document   •  By  default  HVp  POST  request  set  contentType=text/plain  which  failed  to  be  transformed  into  enMty   •  hVp://wiki.fasterxml.com/JacksonFAQDateHandling   •  hVps://github.com/FasterXML/jackson-­‐datatype-­‐jsr310   •  Mongo  doesn’t  support  Java  8  Date  and  Time  types  naMvely   •  hVps://jira.spring.io/browse/DATAREST-­‐524   •  hVp://stackoverflow.com/quesMons/25072967/spring-­‐xd-­‐redis-­‐message-­‐bus-­‐removing-­‐headers-­‐ from-­‐the-­‐message  issue  with  headers  passing  in  Redis   •  SpringXD  doesn’t  support  packages  starMng  with  ‘java’))!!!  
  12. 12. Links   •  Spring  -­‐  hVp://spring.io   •  Zookeeper  -­‐   hVp://zookeeper.apache.org/doc/trunk/zookeeperStarted.html   •  Hadoop  -­‐   hVp://hadoop.apache.org/docs/r2.7.1/hadoop-­‐project-­‐dist/hadoop-­‐ common/SingleCluster.html   •  XD  -­‐   hVp://docs.spring.io/spring-­‐xd/docs/current-­‐SNAPSHOT/reference/html   •  Boot  -­‐   hVp://docs.spring.io/autorepo/docs/spring-­‐boot/1.2.0.M2/reference/ htmlsingle/   •  DataRest  -­‐   hVp://docs.spring.io/spring-­‐data/rest/docs/2.3.2.RELEASE/reference/ html/   •  Redis  -­‐  hVp://redis.io/documentaMon   •  RestTemplate  -­‐   hVps://spring.io/blog/2009/03/27/rest-­‐in-­‐spring-­‐3-­‐resVemplate    
  13. 13. Thank  you!   TwiVer:  @tsaltsol   Skype:  oleg.tsalko   GitHub:  olegts     Code  shown:   hVps://github.com/olegts/ SpringXDCustomTweetsTransformer   hVps://github.com/olegts/TweetsBootDataRestApp    

×