SOA & Big Data

1. SOA & Big data Arnon Rotem-‐Gal-‐Oz

2. Sept 2012 – iOS6 launched with new maps applica>on

3. hEp://theamazingios6maps.tumblr.com/ But something went terribly wrong….

4. •  It isn’t just about geKng all the data there •  Algorithms are cool but we need humans in the loop •  Hire the right people •  Test ! Test ! Test! hEp://theamazingios6maps.tumblr.com/

5. hEp://theamazingios6maps.tumblr.com/ It isn’t just one pile of data

6. Integra>ng Big data & SOA Yoel Ben Avraham -‐ hEp://www.ﬂickr.com/photos/epublicist/3546059144/

8. Data Reﬁnery Ofer Berger hEp://www.haifacity.com/allsites/allpic/a/A1738/A1738Pic3326.jpg

10. ETL integra>on Department DB integra>on Server File-‐based integra>on Online integra>on DB

11. Object soup ASB BLT AFT TGI FRY ECP HDL SWG DRW MFP WCP QYD DLY SKD DLY XPS WIU KYF XOI ZIS CUI WKD WHR ASB GEX RMO JIA HCO BST VUH KFC AJT FQA DKE

12. Services ASB BLT AFT TGI FRY Customer HDL SWG Orders ECP DRW MFP WCP QYD DLY SKD DLY XPS WIU KYF XOI ZIS CUI WKD Promotions WHR ASB Invoices RMO HCO JIA GEX BST VUH KFC AJT FQA DKE

13. Adheres to Policy Governed by Binds to Endpoint Exposes Serves Service consumer Understands Contracts Implements Service Describes Key Component Sends/receives Messages Sends/receives Rela>on

14. Customer Interac>ons Agents Categories

15. Integra>ng Big data & SOA Yoel Ben Avraham -‐ hEp://www.ﬂickr.com/photos/epublicist/3546059144/

16. Saga Key SOA component PaEern component Rela>on Concern/aEribute Prepare/commit/undo Protocol Perform acDvity Register RegistraDon Compensate Prepare / Coordinator* commit / Perform undo acDvity AcDviDes and replies Par>cipator Compensate Create AcDviDes and replies context Ini>ator Service consumer Service

17. HCatalog HBase Data Management HBase Interactions HBase Customer HBase Interaction ETL Recordings HBase NIM Raw Categories (HDFS) Resolved Interactions(H DFS) Resolved Interactions(H DFS) Hadoop Cluster

18. So, what’s the problem ?

19. & Big data can’t move

20. Performance of joins in distributed system sucks! Interactions Interactions Interactions 0-99 200-299 100-199 customers A-H customers I-M customers N-Z Node 1 Node 2 Node 3 {”Interac>on": { "id": ”5", ”par>cipants": { ”customer": [ {”surname": ”McDonalds", ”name": ”Old"},] } }}

21. Cookie cuEer scalability

22. Cell architecture Node 2 Node 3 Node 1 Node N

23. Cell Architecture HBase HBase HBase Categories Customers ORCA BUS Interactions Reference … Data HDFS HBase HBase

24. Orchestra>on Initiate business process Endpoint Manage process Service Schedule Route request Invoke servic Workflow instance es Workflow Host Manage Monitor Endpoint engine workflows workflows workflows Service

25. Map Reduce processing pipeline Customers Local cache Retrive segment Categorize Update Segment InteractionID, Segment Row Segment Row data - create Resolve segment Customer IDs Segment document document (Custoemr) (Categorization) (Interaction) (Interaction) Map pipeline Write Write Categories Interaction Results (interaction) (Categorization) Map Interaction & Update Categorize Prepare data Segments Interaction Interaction mart Export document (Categorization) (Datamart) (Interaction) Reduce pipeline Write Write Write Categories Interaction Interaction Results (interaction) (interaction) (Categorization) Reduce Hadoop Map/Reduce

26. Map Reduce processing pipeline Customers Local cache Retrive segment Categorize Update Segment InteractionID, Segment Row Segment Row data - create Resolve segment Customer IDs Segment document document (Custoemr) (Categorization) (Interaction) (Interaction) Map pipeline Write Write Categories Interaction Results (interaction) (Categorization) Map

27. Data Facets

28. Memcached GigaSpaces Redis GridGain Caching Data grid Oracle Coherence Columnar WebSphere eXtreme Scale Hama HBase Cassandra In-memory Pregel Accumulo Hypertable Key-value store Neo4j Graph Hadoop GlusterFS Distributed file systems RavenDB ScaleBase MongoDB Relational Document CouchDB NewSQL IndexTank Amazon RDS Analytics/MPP VoltDB Apache Solr AKvio Aster Data Microsoo PDW Indexing Columnar ParAccel SAP HANA Oracle Exadata HP Ver>ca IBM Netezza EMC Greenplum

29. Data is mul>-‐>ered Datamart(s) Cube Real-‐>me Datawarehouse (RDBMS) (MOLAP) (in memory) (Hadoop/Hbase) 1-‐7 days detailed 6-‐12 months 6-‐? Months Detailed aggregated 20 years detailed 1-‐3 years aggregated aggregated

30. Data is mul>-‐>ered Datamart(s) Real-‐>me Data warehouse (Columnar) (Hadoop/Hbase) 1-‐7 days 6-‐12 months detailed Detailed 20 years detailed aggregated

31. SOA leaves us with a lot of isolated data

32. Aggregated Repor>ng SQL endpoint Endpoint Produce Request reports ODS/DM Report Report Raw data Out Transpose Endpoint Pull data Ingest Join Subscribed/ Load Landing area Clean pulled data Transform Service Data backend SQL endpoint

33. 5 Report service Views 2 Raw data Load DW/ODS 1 service 4 2 3 Transforma>on service Landing 1

34. 3 Report tool Drill through 7 REST API 5 9 6 8 10 HBase 2 Aggregation 4 map/reduce 2 1 ETL Details Aggregates Raw data (map/reduce (HDFS) +ETL) Data mart

35. Take aways SOA & Big data are beEer together

36. Arnon Rotem-‐Gal-‐Oz arnonr@nice.com hEp://www.nice.com hEp://arnon.me/soa-‐paEerns arnon@rgoarchitects.com @arnonrgo hEp://arnon.me

SOA & Big Data

SOA & Big Data

Recommended

More Related Content

Viewers also liked

Viewers also liked(7)

More from Arnon Rotem-Gal-Oz

More from Arnon Rotem-Gal-Oz(20)

Recently uploaded

Recently uploaded(20)

SOA & Big Data