Big Data – What’s the Big Deal?

732 views
626 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
732
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • REST = representationalstatetransferJSON = JavaScript object notation (Data interchange format)
  • REST = representationalstatetransferJSON = JavaScript object notation (Data interchange format)
  • REST = representationalstatetransferJSON = JavaScript object notation (Data interchange format)
  • REST = representationalstatetransferJSON = JavaScript object notation (Data interchange format)
  • REST = representationalstatetransferJSON = JavaScript object notation (Data interchange format)
  • Big Data – What’s the Big Deal?

    1. 1. 0 Copyright 2013 FUJITSUBig Data –What’s the Big Deal?Gernot Fels
    2. 2. 1 Copyright 2013 FUJITSUBig Data – What’s the Big Deal?Gernot Fels, Fujitsu Technology Solutions, Solution Marketing
    3. 3. 2 Copyright 2013 FUJITSUSome dreams organizations dreamWhat if you could … Predict what will happen –trends, customer behavior,business opportunities, problems? Take the right decisions? Accelerate your decisions? Have respective actions initiated automatically? Fully understand the root cause of costs? Skip useless activities? Quantify and minimize risks? Know, what you don’t know?Can these dreams come true?
    4. 4. 3 Copyright 2013 FUJITSUIs Business Intelligence an option?Concept Turn data into information Collect and consolidate data Transform data into usable form Analyze data (discover relations, patterns)Decision support based on data instead of intuition.Data warehouseAnalyzeETLActDecideObjectives Improve business processes Reduce costs Minimize risk Increase business value
    5. 5. 4 Copyright 2013 FUJITSUIs Business Intelligence an option?BI as it used to be Internal Structured Few sources GB and TB At rest Reports on history Avoid risk Periodic Batch Static model Few direct users On-premise
    6. 6. 5 Copyright 2013 FUJITSUThe world has changed since thenBI as it used to be Internal Structured Few sources GB and TB At rest Reports on history Avoid risk Periodic Batch Static model Few direct users On-premiseToday’s demands Internal and external Un- / semi- / poly- / structured Versatile sources TB and PB At rest / in motion Predict Recognize opportunities Ad-hoc Real-time Try and innovate Many direct users Anywhere, from any device
    7. 7. 6 Copyright 2013 FUJITSUBig DataBe welcome in the world of Big DataBI as it used to be Internal Structured Few sources GB and TB At rest Reports on history Avoid risk Periodic Batch Static model Few direct users On-premiseToday’s demands Internal and external Un- / semi- / poly- / structured Versatile sources TB and PB At rest / in motion Predict Recognize opportunities Ad-hoc Real-time Try and innovate Many direct users Anywhere, from any deviceAffordable technologies to quickly capture, store and analyze data.VolumeVarietyVelocityVersatilityValue
    8. 8. 7 Copyright 2013 FUJITSUManufacturingEnergyMaintenanceAgricultureBig Data matters to every industryBig DataMarketingHealthcare Traffic, TransportPublic SectorNew opportunities, new values for enterprises and society.Retail Finance
    9. 9. 8 Copyright 2013 FUJITSUWhy traditional solutions do not workObjective: Keep processing time constant while volumes increase.Exponentialgrowth ofdata
    10. 10. 9 Copyright 2013 FUJITSUWhy traditional solutions do not workRow-oriented storeKey Region Product Sales Vendor0 Europe Mice 750 Tigerfoot1 America Mice 1100 Lionhead2 America Rabbits 250 Tigerfoot3 Asia Rats 2000 Lionhead4 Europe Rats 2000 TigerfootSELECT SUM (Sales) GROUP BY RegionData requiredData read, but not requiredIndex AreaRDBMS Data in rows Queries imply many read ops for irrelevant data Slow data access, long-lasting queries Unstructured data requires pre-processingObjective: Keep processing time constant while volumes increase.
    11. 11. 10 Copyright 2013 FUJITSUWhy traditional solutions do not workObjective: Keep processing time constant while volumes increase.RDBMS Data in rows Queries imply many read ops for irrelevant data Slow data access, long-lasting queries Unstructured data requires pre-processingRDBMS and server scale-up Server performance is limited Hard limit for each resource type
    12. 12. 11 Copyright 2013 FUJITSUWhy traditional solutions do not workRDBMS Data in rows Queries imply many read ops for irrelevant data Slow data access, long-lasting queries Unstructured data requires pre-processingRDBMS and server scale-up Server performance is limited Hard limit for each resource type+RDBMS and server scale-out Effort to coordinate access to shared data Decreasing server efficiency with increasing number Storage connection as bottleneckTime-consuming analysis – Far away from real-time
    13. 13. 12 Copyright 2013 FUJITSUWhat will work for Big Data?New demands require new solution approaches.
    14. 14. 13 Copyright 2013 FUJITSUDistributed parallel processingData NodeTask TrackerData NodeTask TrackerData NodeTask TrackerName NodeJob TrackerClientMasterSlaves DFSConcept Distribute data and I/O to many nodes Local server storage Move computing to where data resides Shared nothing architecture Non-blocking networkBenefits High performance, fast results Unlimited scalability Fault-tolerance (data replication to several nodes) Cost-efficiency (standard servers with OSS)Examples
    15. 15. 14 Copyright 2013 FUJITSUIn-Memory DB (IMDB)ConceptBenefits Load entire DB into RAM of server(s) for analysis Operation in RAM Scalability by adding servers Persistence layer for recovery Eliminate I/O (Avoid read ops) 1K-10K times faster analysis than on disk Fast data storing, retrieval, sort Real-time analysisRestriction Data size is limited by RAM size and budgetEliminate I/O.IMDBRAM …Persistence Layer:Snapshot, Replication, Backup, LogRAM …RAM …Table TableTableTableTable TableTableAppApp …
    16. 16. 15 Copyright 2013 FUJITSUSSDConceptBenefitsIn-Memory Data Grid (IMDG) Servers with distributed in-memory cachebetween app servers and data storage Caching of complex queries and reusable results Data operation in RAM Access to storage only when needed Transparent access by apps possible,optimized cache utilization by adapting apps Data replication to multiple servers Data consistency and quick failover after server failure SSD to avoid data loss after power outage I/O bottleneck resolved (by reduced I/O) Fast response, scalability, business continuityReduce I/O.IMDGAppApp …RAM …RAM …RAM …Query ReportWeb pageQuery ReportWeb pageQuery ReportWeb page
    17. 17. 16 Copyright 2013 FUJITSUAll Flash Array (AFA)ConceptBenefits Designed and optimized for flash Scalability by adding flash memory arrays No single point of failure High-speed interconnect to servers Highest I/O performance (6- to 7-digit number of IOPS) Low latency Consistent user experience Data integrity, no data loss Less space and power consumptioncompared to hard disk arraysMore IOPS, shorter latency – Accelerated I/O.
    18. 18. 17 Copyright 2013 FUJITSUColumn-oriented databasesRow-oriented data storeColumn-oriented data store Billions of rows (records) Many queries relate to small number of columns Inefficient to read all rows Reduced I/O, fast analysis Access only relevant columns Split column into multiple sectionsfor simple parallelized processing High compression Column contains only 1 data type Often only few values per column (to store) Typical compression factor 10-50NoSQL databases help overcome the limitations of RDBMS.Row-oriented storeKey Region Product Sales Vendor0 Europe Mice 750 Tigerfoot1 America Mice 1100 Lionhead2 America Rabbits 250 Tigerfoot3 Asia Rats 2000 Lionhead4 Europe Rats 2000 TigerfootColumn-oriented storeKey Region Product Sales Vendor0 Europe Mice 750 Tigerfoot1 America Mice 1100 Lionhead2 America Rabbits 250 Tigerfoot3 Asia Rats 2000 Lionhead4 Europe Rats 2000 TigerfootSELECT SUM (Sales) GROUP BY RegionData requiredData read, but not required
    19. 19. 18 Copyright 2013 FUJITSUComplex Event Processing (CEP)ConceptCritical parameters Collect and analyze continuously generated data(event streams) in real-time Compare events to set of rules Aggregate, filter (acc to relevance), sort Detect patterns (Logical / temporal correlation) Time-based rules (e.g. alert if >10 matches /sec) Raise alert when match is found / absence of events Trigger actions in real-time In-memory cache for acceleration (IMDS or IMDG) Throughput: 10K’s to M’s of events per sec Latency: µs to ms (time from event input to output)CEP requires parallel processing and in-memory technologies.StreamingSensorEvent inputRulesState transitionApply Event outputControlCEP Engine
    20. 20. 19 Copyright 2013 FUJITSUUn- /Semi-/Poly-structureddataBig Data solution architectureIMDBDB / DWData Sources Analytics Platform AccessExtract, Collect Clean, Transform Decide, ActAnalyze, VisualizeConsolidated data Distilled essence Applied knowledgeVarious data
    21. 21. 20 Copyright 2013 FUJITSUUn- /Semi-/Poly-structureddataBig Data solution architectureIMDBDB / DWData Sources Analytics Platform AccessExtract, Collect Clean, Transform Decide, ActAnalyze, VisualizeConsolidated data Distilled essence Applied knowledgeVarious dataDistributedData StoreMap ReduceData StoreCEP
    22. 22. 21 Copyright 2013 FUJITSUUn- /Semi-/Poly-structureddataBig Data solution architectureIMDBDB / DWData Sources Analytics Platform AccessExtract, Collect Clean, Transform Decide, ActAnalyze, VisualizeConsolidated data Distilled essence Applied knowledgeVarious dataDistributedData StoreMap ReduceData StoreCEPVisualizationReportingNotificationQueries…IMDBDB / DW
    23. 23. 22 Copyright 2013 FUJITSUUn- /Semi-/Poly-structureddataBig Data solution architectureIMDBDB / DWData Sources Analytics Platform AccessExtract, Collect Clean, Transform Decide, ActAnalyze, VisualizeConsolidated data Distilled essence Applied knowledgeVarious dataDistributedData StoreMap Reduce IMDBDB / DWVisualizationReportingNotificationQueries…Data StoreCEP
    24. 24. 23 Copyright 2013 FUJITSUUn- /Semi-/Poly-structureddataBig Data solution architectureIMDBDB / DWData Sources Analytics Platform AccessExtract, Collect Clean, Transform Decide, ActAnalyze, VisualizeConsolidated data Distilled essence Applied knowledgeVarious dataIMDBDB / DWVisualizationReportingNotificationQueries…CEPIMDS IMDGDistributedData StoreMap ReduceIMDG IMDG
    25. 25. 24 Copyright 2013 FUJITSUBig Data is not just about infrastructure Hoarding data is not enough Transforming data into high quality is key Interactive and exploratory analysis Understand data and discover meaning Lots of questions Which data? What to look for? Which questions to ask? Which tools? How to use them? What about data retention? Analytic skills (data scientist) Change to analytical cultureDeep knowledge of data, tools and intended results.
    26. 26. 25 Copyright 2013 FUJITSUHow can helpOptimum concept for each situationEnd-to-end services Use right combination of technologies Infrastructure products, software and middleware Appliances for IM computing Assessment, consulting Optimum Solution Design Deployment, integration, supportSourcing options Self-managed (on-premise) Managed services (on- / off-premise) Fujitsu CloudOne-stop shop for Big Data: Reduce complexity, time and risk.
    27. 27. 26 Copyright 2013 FUJITSUVisit our Big Data Internet SiteFor more information: http://www.fujitsu.com/fts/solutions/high-tech/bigdata/ http://www.fujitsu.com/de/solutions/high-tech/bigdata http://www.fujitsu.com/global/services/software/interstage/solutions/big-data/
    28. 28. 27 Copyright 2013 FUJITSUSummary Big Data offers enormous potentialfor business opportunities and value It change the way companiesmake decisions, do business, succeed or fail New infrastructure concepts New types of analytics New tools, SW and MW New ideas and use cases New skills New value Fujitsu – A one-stop shop for Big DataFujitsu turns Big Data into a Big Deal.
    29. 29. 28 Copyright 2013 FUJITSU

    ×