Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[db tech showcase Tokyo 2016] E22: Getting real time Oracle data into Kafka and unlocking the data in your database by Dbvisit Software Limited Chris Lawless

627 views

Published on

Kafka is quickly gaining momentum as a very popular and very fast messaging platform that is very good at integrating different types of data quickly. Kafka makes this data available as a real-time data stream for consumption by enterprise users.There is so much hidden data available in our Oracle databases. How can we turn the database inside out to make this data available real-time to Kafka along with the other data sources in our enterprise. This paper will present the use cases of Oracle real-time data streaming as well as an introduction into Kafka and how to use Oracle logical replication to get Oracle real time into Kafka. This paper will include a real life real time demo from Oracle into Kafka.

Published in: Technology
  • Be the first to comment

[db tech showcase Tokyo 2016] E22: Getting real time Oracle data into Kafka and unlocking the data in your database by Dbvisit Software Limited Chris Lawless

  1. 1. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com Dbvisit  So*ware   Real-­‐2me  Oracle  Database  Streaming  into  Ka9a     Chris  Lawless    
  2. 2. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com Agenda   •  Oracle  OLTP   •  Evolu2on  of  data  warehouses   •  Data  Lake   •  Intro  to  Ka9a  -­‐  what  need  does  it  fill?       •  Marriage  of  the  two  
  3. 3. © 2016 Dbvisit Software | dbvisit.com About  Dbvisit  So*ware     •  Real-time Oracle Database Streaming software solutions •  In the Cloud | Hybrid | On-Premise •  New Zealand-based, US office, Asia sales office, EU office (Prague) •  Unique offering: disaster recovery solutions for Oracle Standard Edition •  Low cost Oracle GoldenGate alternative •  Flexible licensing, pricing models available •  Peerless customer support
  4. 4. © 2016 Dbvisit Software | dbvisit.com Result:  1,100+  customers  in  6  con2nents  
  5. 5. © 2016 Dbvisit Software | dbvisit.com About  Chris   巨人  
  6. 6. © 2016 Dbvisit Software | dbvisit.com About  Chris   •  4 Years at Oracle University teaching DBA courses •  5 Years at GoldenGate Support and Product Management •  4 Years at Oracle GoldenGate Product Management •  Past 3 years at Dbvisit
  7. 7. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com The  World  we  live  in   The  Situa2on:     ü  The  enterprise  is  increasingly  powered  by  data   ü  OLTP  transac2onal  data  essen2al   ü  The  use  of  real-­‐2me  data  for  compe22ve  advantage  is  disrup2ng  most   industries   ü  Tradi2onal  databases  are  not  going  away,  new  database  technologies  are   being  added   ü  Con2nuous  replica2on  data  streams  becoming  a  “first  class  ci2zen”    
  8. 8. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com Reality  of  RDBMS   RDBMS     ü  Millions  of  Oracle  databases  out  there   ü  OLTP  databases  are  ingrained  in  the  business   ü  Pervasive   ü  ERPs   ü  CRMs    
  9. 9. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com OLTP   RDBMS   ü  MySQL    #1  leader  in  databases   ü  MSSQL    #1  leader  is  sold   ü  IBM  DB2  #1  in  most  installs   ü  Oracle  #1  in  most  sales   ü  Oracle  is  reported  to  have  over  50%  of  all  RDBMS  sales   ü  Oracle  is  here  to  stay  
  10. 10. © 2016 Dbvisit Software | dbvisit.com OLTP  Structured  Data   •  Nice  and  structured   •  Columns   •  Rows   •  Rela2onships  
  11. 11. © 2016 Dbvisit Software | dbvisit.com OLTP  systems   •  Banking   •  Online  shopping   •  Stock  Markets   •  Healthcare   •  ERP  Systems   •  Customer  Rela2ons  Management  
  12. 12. © 2016 Dbvisit Software | dbvisit.com OLTP  systems   OLTP  Database  
  13. 13. © 2016 Dbvisit Software | dbvisit.com OLTP  systems  with  Data  Warehouses    Old   school   •  OLTP  systems  typically  will  feed  Data  Warehouses  via  Batch  jobs   •  Banking  statements  that  get  mailed  monthly   •  Sales  analysis  on  what  was  sold  last  month   •  Repor2ng  on  ERP  systems   •  Quarterly  Financial  reports  
  14. 14. © 2016 Dbvisit Software | dbvisit.com OLTP  with  Data  Warehouse   Batch  ETL   Process   Data  Warehouse   database   OLTP  Database  
  15. 15. © 2016 Dbvisit Software | dbvisit.com OLTP  systems  with  Data  Warehouses       REAL-­‐TIME   •  Online  Shopping  with  INSTANT  emails  regarding  your  shopping  habits   •  ERP  systems  with  INSTANT  informa2on  regarding  current  sales   •  Online  Banking  with  access  to  years  of  historical  data  
  16. 16. © 2016 Dbvisit Software | dbvisit.com OLTP  with  Data  Warehouse   Real-­‐2me   Streaming   Data  Warehouse   database   OLTP  Database  
  17. 17. © 2016 Dbvisit Software | dbvisit.com The  concept  of  Data  Lake  or  Data   Reservoir   Not  all  data  is  structured     •  What  about  IOT  data?   •  What  about  machine  data?   •  What  about  log  data?   •  Semi  Structured  data?  
  18. 18. © 2016 Dbvisit Software | dbvisit.com The  new  concept  of  Data  Lake  or  Data   Reservoir   •  A  Data  Lake  is  storage  to  hold  vast  amounts  of  RAW  data  that  is  typically   kept  in  the  na2ve  format   •  O*en  using  huge  unstructured  nodes   •  Hadoop  is  the  frequent  repository  of  choice  
  19. 19. © 2016 Dbvisit Software | dbvisit.com The  new  concept  of  Data  Lake  or  Data  Reservoir   Machine   Data   loT   Web   logs   Applica2on   logs   Streaming   Web  Data   Other   OLTP  Database   OLTP  Database   ETL   Real-­‐2me   Streaming   Data  Lake  
  20. 20. © 2016 Dbvisit Software | dbvisit.com Ka9a  a  brief  History   •  Open  Sourced  in  2011   •  Developed  at  Linkedin  and  then  ‘released  to  the  world’  as  part  of  Apache   Founda2on.   •  These  guys  spun  off  to  form  Confluent   -­‐  Ka9a  Connect.    A  framework  which  makes  it  simple  to  define  connectors  to   move  data  in  and  out  of  Ka9a   •  Key  features:   -­‐  Simple  API  for  producers  and  consumers   -­‐  High  Throughput   -­‐  Scaled  out  Architecture   -­‐  Non  formaeed  messages  
  21. 21. © 2016 Dbvisit Software | dbvisit.com Intro  to  Ka9a   What  is  Ka9a?   A  distributed  system  where  messages  are  kept  in  topics  that  are  par22oned   and  replicated  across  mul2ple  nodes.     Message   Simply  put…  the  data     Messages  can  be  in  any  format:   Common  ones  are  String,  JSON,  Avro  
  22. 22. © 2016 Dbvisit Software | dbvisit.com Intro  to  Ka9a   Topics   One  or  more  Par22ons  that  are  ordered  sequences  of  messages.     Producers    (Publishers)   Produce  data  to  one  or  more  topics     Consumers  (Subscribers)   Subscribe  to  topics  and  process  the  messages      
  23. 23. © 2016 Dbvisit Software | dbvisit.com Old  method   Source   Target   Target   Target   Ka9a  Source   Source   Target   Target   Target   Source   Source   Source  
  24. 24. © 2016 Dbvisit Software | dbvisit.com Intro  to  Ka9a   Producer   Producer   Producer   Consumer   Consumer   Consumer   Ka9a  
  25. 25. © 2016 Dbvisit Software | dbvisit.com Ka9a   Par22on  0   Par22on  1   Par22on  2   Old   New  
  26. 26. © 2016 Dbvisit Software | dbvisit.com Ka9a   •  Ka9a  treats  each  topic  par22on  as  a  log  (a  sequen2al  ordered  set  of   messages)   •  You  can  call  Ka9a  a  log  reader  and  a  log  writer  
  27. 27. © 2016 Dbvisit Software | dbvisit.com Ka9a   •  Log  compac2on/log  reten2on   •  Ka9a  Streams  –  the  new  stuff  from  Confluent   -­‐  No  need  for  Spark  or  other  tools   -­‐  Pure  streaming  of  the  data  -­‐  process  data  “on  the  fly”   -­‐  Ka9a  0.10.0  
  28. 28. © 2016 Dbvisit Software | dbvisit.com Marriage  of  two  worlds   •  If  we  mix  the  ‘old  world’  log  readers  with  the  new  world  log  readers  and   writers.   •  Blended  technology   -­‐  Using  the  Oracle  logical  replica2on  tools  with  Ka9a  as  the  message   broker   -­‐  Oracle  becomes  ‘just  another  feed  for  Ka9a’  
  29. 29. © 2016 Dbvisit Software | dbvisit.com Oracle  Redo  logs   •  Reading  the  Oracle  redo  logs  is  not  easy.    Oracle  doesn’t  really  publish  the   API.   •  Because  of  this  replica2on  companies  have  ‘sprung  up’  around  the  moving  of   Oracle  data.  
  30. 30. © 2016 Dbvisit Software | dbvisit.com Who  can  do  this?  
  31. 31. © 2016 Dbvisit Software | dbvisit.com Logical  Replica2on  to  Ka9a  high  level   overview   JSON   THL  
  32. 32. © 2016 Dbvisit Software | dbvisit.com Way  of  the  (New)  World  
  33. 33. © 2016 Dbvisit Software | dbvisit.com Key  Concepts   Real-­‐Time  Data/Event  Streaming   •  A  con2nuous  flow  of  instantaneous  data  with  as  close  to  zero  latency  as  possible.     Real-­‐Time  Stream  Processing   •  Systems  that  con2nuously  process  incoming  data,  and  will  con2nue  to  process  that   incoming  data  un2l  the  applica2on  is  stopped,  rather  than  opera2ng  on  a  fixed  set  of  data.     •  Indica2ve  use  cases:   -­‐  Financial  Trading   -­‐  Real-­‐2me  System  Monitoring   -­‐  Business  Intelligence   -­‐  Real-­‐2me  Analy2cs  
  34. 34. © 2016 Dbvisit Software | dbvisit.com Automo2ve:  Ka9a  Streaming     OLTP  and  Ka9a   Streaming  data  that  can  be  USED  as  it  moves       •  Weather   •  Tolls   •  Sensor   •  Mileage  data   •  Tire  pressure   •  GPS  
  35. 35. © 2016 Dbvisit Software | dbvisit.com Healthcare:  Ka9a  Streaming       OLTP  and  Ka9a   •  Prescrip2ons   •  Insurance   •  Medical  devices   •  Medical  history   •  etc  
  36. 36. © 2016 Dbvisit Software | dbvisit.com ⾁肉  
  37. 37. © 2016 Dbvisit Software | dbvisit.com       © 2016 Dbvisit Software | dbvisit.com Thank  you     Q  &  A    

×