Taming Big Data With Modern Software Architecture


Published on

Big Data kennen sehr viele IT-Experten, wenigstens haben Sie eine Vorstellung davon. In der Praxis arbeiten damit in Deutschland derzeit nur wenige. Dabei bringt Big Data ein ganz neues Momentum in moderne Softwarelösungen und ist im Kontext der Mobil-, Cloud- und Social-Veränderungen nicht wegzudenken. Big Data macht Software intelligent und damit auf eine ganz neue Art für die Benutzer erlebbar. Mit Big Data entstehen neue Softwarearchitekturen, weil Informationen völlig anders verarbeitet werden - nämlich schneller, differenzierter und oft mit dem Ziel, Schlüsse zu ziehen und Vorhersagen zu treffen.

In diesem Vortrag wird erläutert, wie moderne Softwarearchitekturen gestaltet werden, sodass Sie Big Data Paradigmen erfolgreich umsetzen und welche Vorteile sich für die zunehmend mobilen Softwarelösungen ergeben. Wir werfen zudem einen Blick auf die Potentiale und Optionen in Branchen wie Banken, Versicherung oder Handel.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Taming Big Data With Modern Software Architecture

  1. 1. Software Architecture Taming Big Data With Modern Software Architecture Big Data User Group Stuttgart, 26.09.2013 Thomas Widmann Managing Director, WidasConcepts Vinoth Kannan Senior Consultant, WidasConcepts
  2. 2. About the Speakers 2   Thomas Widmann Vinoth Kannan !   Senior Consultant in WidasConcepts !   Working primarily around Big Data ecosystems of Hadoop and Storm !   Experienced in IT industries of Europe, North America and Asia !   Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France ! Contact: vinoth.kannan@widas.de !   Managing Director of WidasConcepts !   Expert in modern Sales- and Service-Solutions !   More than two decades of consulting experience in Web X.0 concepts, continuous delivery and IT-strategies !   Big Data Evangelist ! Contact: thomas.widmann@widas.de
  3. 3. Who we are !   Innovative IT-Consulting Company founded in 1997 ! Headquaters in Wimsheim with offices in Munich and Essen ! Specialized in Software for Modern Business in the areas of !   Strategic IT Consulting !   Next Business Intelligence !   Software Development !   35 expert consultants specializing in !   Data Science !   Software Architecture !   Mobile & Web Development !   Business Consulting WidasConcepts GmbH 3  
  4. 4. Agenda 4   How Big Data can help business Big Data applied in CarbookPlus Platform New Software Architectures for Big Data Q & A
  5. 5. How big data can help business 5 Big challenges of todays business eCommerce Financial Services Telecommunication HealthCare Marketing National Security o  Increase in Sales o  Determine pricing o  Avoid Shop cart abandoners o  Manage logistics and inventory o  Fraud detection o  Real Time analytics on financial data o  Managing mobile payments o  Manage network traffic overload o  Hardware cost optimisation based on traffic data o  Real time monitoring of patients vitals for predictive intervention and care. o  Cost savings on patient care o  Mood prediction for market research o  Real time bidding o  Location based marketing o  Monitor real time feed from CCTV, traffic data, mobile data o  Find patterns/cluster for people, places, products, things Building sector, all manufacturers, energy, media and many more
  6. 6. Getting the Right Data to the Right People at the Right Time 6   Mobile Payment Money Cards / POS Banking Portal Adviser / Agency Banking Operations Event Database Business Activity Monitoring CEP Making Use of Events Real Time Analysis Cross-Up-Selling Fraud Detection Product Innovation
  7. 7. Impact of Big Data on Business Results on Europe-wide survey by TU-Berlin, Sept 2013 Source: European Smart Data Survey, Sept 2013 7  
  8. 8. Need for Big Data 4Vs concept of Big Data Volume To handle the huge amount of data genereted by businesses Velocity To store, analyse and retrive huge dataset in good speed Variety To process data from multiple sources, majorly unstructured data Source: Oracle Value To ask the right questions to generate maximum value 8  
  9. 9. What makes data so big? Combination of structured, unstructured data created by sensors, actors, humans 9  
  10. 10. New Software Architecture Need of the hour for modern architecture analyse   huge   volume  of   data   fast   consolidate   data  from   mul;ple   sources   Intelligent   predic;ve   10  
  11. 11. Agenda 11   How Big Data can help business Big Data applied in CarbookPlus Platform New Software architectures for Big Data Q & A
  12. 12. Datawarehouse – Classic BI Online System Database n Classic Software Architectures Main goal: keep business records <Client> Browser Presentation Business Logic Database 2 StagingETL ETL BI Reporting Database 1 Batch Processing Main goal: Reporting 12  
  13. 13. Need for real time system for enterprise 13   The Approach Impact of analytics, event processing and decision management in building a real time big data system for enterprise Acquire Data: •  Business Trx./Events •  Observational Events from anywhere Observe   Orient   Decide   Act   Systems: •  Various systems •  Preferrable: MQ •  put in context •  Evaluate data, predict •  determine actions •  Initiate actions •  manage data Systems: •  complex event processing, rule management •  realtime analysis with new big data technologies •  select action, if necessary •  Perform action (manually or automated) Systems: •  Business software •  Decision support software •  Reporting (John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich
  14. 14. Phases of realising real time system Step by step approach for the architecture • Create structured data from raw data • Extract, merge, filter etcData Extraction • Aggregate and select relevant data • Speed and flexibilityModel development • Testing the model • Comparing the result Evaluation & deployment • Triggered by action in the decision layer • Score to validate outputReal-time Scoring • Need to refresh the ever changing data • Real time data viewModel Refresh 14  
  15. 15. Big Data Architecture Service Layer CQRS Distributed DB Distributed Messaging System Recommendations Realtime-Processing Batch-Processing Lambda architecture Business Transactional Data Business Analytics Data Recommendation Data Real Time Event Alert Data Big Data Eco System Distributed Search Engine Business Data Formats Contextual Data Business Transactions Reporting Notifications Notification SystemData Retrieval In-Memory Technology 15   Distributed DBNoSQL DB
  16. 16. Basic idea behind Lambda Architecture query = function(all data) Nathan Marz Big Data - principles and best practices of scalable realtime data systems 16  
  17. 17. Basic Idea behind Lambda 𝑓(​ 𝑎↓0 …​ 𝑎↓𝑚 ) Perform some function from real-time data “0“ to the history data “n“ Real Time Big Data 𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 …​ 𝑎↓𝑚 ) = 𝑓(​ 𝑎↓0 ..​ 𝑎↓𝑛 ) + 𝑓(​ 𝑎↓𝑛+1 …​ 𝑎↓𝑚 ) Lambda Architecture Hadoop ProcessStorm Process Real Time Big Data } } } Letting the History data processed by Hadoop makes process faster 17  
  18. 18. Agenda 18   How Big Data can help business Big Data applied in CarbookPlus Platform New Software architectures for Big Data Q & A
  19. 19. What  is  Carbookplus?     Change  the  way  you  drive.   19  
  20. 20. Why  Carbookplus?     Benefits  of  being  a  Carbookplus  member   Track trips Maintain Logbook Get Notified about best gas stations & snipers Manage and compare vehicle cost Fleet management Social platform connecting drivers
  21. 21. Big  Data  in  Carbookplus   Where  do  we  use  it   Trip  Transac;on  Data   Recommend  possible  trips     Create  trips     Update  logbook  entry     No;fy  user  for  discrepancy  in  the   logbook  entry     No;fy  user  for  best  gas  sta;on     Update  vehicle  mileage       Generate  analy;c  reports   Real Time !!!
  22. 22. Big  Data  in  Carbookplus   Event  Driven  Model   Use Case : Track trips on Real Time Start  address   Intermediate  address  1   End  address   Event 1 Event 2 Event n Intermediate  address  n   …..   Event n+1 Object Model Event …
  23. 23. Big  Data  in  Carbookplus   How  do  we  use  it   Trip  Transac;on   Data   Real Time Storm Cluster In  Memory  Data  Processing   MongoDB   Trip  Data   Hadoop  Cluster   Map  &  Reduce   Analy;cal  Data   Decision  Management           Recommender   Engine   Rule   Engine   KaYa   ETL   Output   Mobile   Web   BI   User  Data   ETL   Vehicle  Data   ETL   Historical Data Real Time Data
  24. 24. Conclusion   Ø  Big Data clearly improves business efficiency and answers lot of the questions which were previously unanswerable in this data driven world. Ø  Need to adapt our IT architecture to remain competitive and see the big picture Ø  Flipside : No standardised implementation architecture. Need to be tailor-made for individual business needs. Ø  Road Ahead : Need for phased integration into Big Data ecosystem Ø  We are hiring !! §  Data Scientists §  Big Data Engineers §  Interns Visit us : www.widas.de Email : contact@widas.de
  25. 25. Agenda 25   How BigData can help business BigData applied in CarbookPlus Platform New Software architectures for Big Data Q & A