Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BA4All Sweden: State of the Art in In-Memory Analytics

5,045 views

Published on

A review of some of the history and advantages of in-memory analytics

Published in: Data & Analytics, Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The Scrambler Unlock Her Legs | 95% Off by Bobby Rio-Rob Judge? ♣♣♣ https://tinyurl.com/unlockherlegss
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi there! Essay Help For Students | Discount 10% for your first order! - Check our website! https://vk.cc/80SakO
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @ https://www.ThesisScientist.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

BA4All Sweden: State of the Art in In-Memory Analytics

  1. 1. What is the Present State of the Art Of In-Memory Analytics? Timo Elliott, Innovation Evangelist timoelliott.com
  2. 2. Disclaimer “i think you’ll find it’s a bit more complicated than that.”
  3. 3. A Bit of History
  4. 4. LEO: Lyon’s Electronic Office, 1951 Sixty-four 5ft-long mercury tubes, each weighing half a ton, were used to provide a massive 8.75 Kb of memory (i.e. one hundred-thousandth of a today’s entry-level iPhone).
  5. 5. 1980s – first in-memory BI tools Usefulness limited by high cost of memory and limitations of 16bit memory addressing 640KB max memory
  6. 6. 1995: Windows 95 & 32-bit Architectures Qlikview, TimesTen, and others take advantage of new 32bit memory addressing to provide in-memory analytics
  7. 7. Complex Event Processing Sensor readings – 10’s of thousands per second Virtually no useful information in a single isolated event history e.g. Compare variance of trends across multiple sensors against historical norms Event window – e.g. 30 min Alert Extracting insight from events
  8. 8. Complex Event Processing Tradtional BI: “How many Fraudulent credit card transactions occurred last week in Madrid?” 1 2 3 4 5 6 7 8 9 time Complex Event Processing: “when three credit card authorizations for the same card occur in any five seconds window, deny the requests and check for fraud.” Continuous Queries
  9. 9. In-Memory and The Internet of Things CEP Engine Studio Input Streams Sensors Messages Transactions Market data Clicks … Other data storage Alerts Dashboards Applications adapters
  10. 10. Reporting “Traditional” Business Intelligence Slow Painful Expensive Operational Data Store Data Warehouse Indexes Aggregates DataBusiness Applications Copy ETLCalculation Engine Business Intelligence Query Results Query Slow Painful Expensive Operational Data Store Data Warehouse Indexes Aggregates DataBusiness Applications Copy ETL Calculation Engine Business Intelligence Query Results Query Data Marts
  11. 11. It’s Like An Onion… The more layers there are, the more it makes you cry…
  12. 12. What Was The Problem? Slow Disks & CPUs I/O Bottleneck Expensive Memory Optimized for Transactions BI is an Afterthought 30 Year-Old Database Design Principles
  13. 13. Why Talk About In-Memory?
  14. 14. Analysts Recommend In-Memory . “An in-memory data platform offers more than performance benefits” “Recommendations: Invest in an in-memory data platform to gain competitive edge” “In-Memory Database Is Gaining Momentum Across All Use Cases” “In-Memory Delivers Extreme Performance And Scalability” “In-Memory Data Platform Is No Longer An Option — It’s A Necessity!”
  15. 15. Companies Like Yours Are Implementing In-Memory 32% run in-memory databases at their location today 75% expect to expand their in-memory use in the next 3 years 21% 25% 88% More rapid deployments Greater flexibility Faster response times / less latency 25% 42% 58% IT operations Core business functions Analytics Source: 2014 DBTA survey of IT and data managers Top Uses Top Benefits
  16. 16. Database vendors are investing in in-memory The Forrester Wave: In- Memory Database Platforms, Q3 ‘15
  17. 17. All Analytics Vendors Now Support In-Memory To Some Extent Oracle Database In-Memory Option “The Oracle Database In-Memory option dramatically accelerates the performance of analytic queries by storing data in a highly optimized columnar in-memory format.” Microsoft SQL Server In-Memory OLTP ‘When data lives totally in memory, we can use much, much simpler data structures. When a table is declared memory- optimized, all of its records live in memory.” DB2 with BLU Acceleration “IBM DB2 with BLU Acceleration speeds analytics and reporting using dynamic in-memory columnar technologies. In- memory columnar technologies provide an extremely efficient way to scan and find relevant data.“ Qlik “In-memory indexing automatically builds and maintains all data relationships from multiple sources for unrestricted exploration” SAP HANA “A good example of a modern in-memory database technology is SAP's HANA platform. “ Teradata “Teradata uses a hybrid approach to in-memory that intelligently puts the right data in memory to deliver high-speed in-memory performance at a fraction of the cost of putting all data in memory.“ Tableau “The Data Engine is a high-performing analytics database on your PC. It has the speed benefits of traditional in-memory solutions without the limitations that your data must fit in memory.“ Spark “Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.“
  18. 18. What Is In-Memory? And why now?
  19. 19. What Is In-Memory? Data access times of various storage types relative to RAM (logarithmic scale) RAM is 300,000 times faster than hard disks CPU register is 61 million times faster than hard disks
  20. 20. In-Memory Databases vs. Caching Traditional Computing Application Code App. Data Main memory (DRAM) "Database of Record" Application Code App. Data In-memory Computing Main memory (DRAM) "Database of Record" • Persistency • Recovery • Post-processing • Backup Application Code App. Data Application Code App. Data “Much of the work that is done by a conventional, disk-optimized RDBMS is done under the assumption that data primarily resides on disk. Even when a disk-based RDBMS has been configured to hold all of its data in main memory, its performance is hobbled by assumptions of disk-based data residency. When the assumption of disk-residency is removed, complexity is dramatically reduced.” - Oracle TimesTen Overview
  21. 21. In-Memory Computing Costs have Plummeted Turning Torso: 190m Cost of 1 Mb of memory in 2000: ≈$1
  22. 22. In-Memory Computing Costs have Plummeted Cost of 1 Mb of memory today: ≈ ½ cent 75cm And shrinking, and shrinking, and shrinking…. IKEA MICKE Skrivbord 399 kr
  23. 23. Prices Continue to Slide DRAM production costs drop by 30% every 12 months
  24. 24. In-Memory Computing Operational Data Store Data Warehouse Indexes Aggregates DataBusiness Applications Copy ETL Calculation Engine Business Intelligence Query Results Query Up to 1,000x faster No optimizations required Data Marts
  25. 25. Row vs. Column Databases My Filing System My Wife’s Filing System Row-based Column-based
  26. 26. Data WarehouseData Warehouse Column Databases Operational Data Store Data Warehouse DataBusiness Applications Copy ETL Calculation Engine Business Intelligence Query Results Query Up to 1,000x faster More data in less space
  27. 27. Massively Parallel Systems E.g. Netezza technology now part of IBM PureSystems E.g. Greenplum, now part of EMC
  28. 28. Column Stores, Compression, and Parallel Processing E.g. DB2 with BLU acceleration
  29. 29. “In-Chip” Processing E.g. SiSense Vector-based instructions Cache-optimized Decompression Close collaboration between in-memory software vendors and chip developers (e.g. SAP & Intel Haswell)
  30. 30. Data Warehouse Massively Parallel Hardware Operational Data Store DataBusiness Applications Copy ETL Business Intelligence Query Results Query Up to 1,000x faster Optimized for hardware Calculation Engine
  31. 31. In-Database Processing E.g. SAS & Teradata
  32. 32. Move Processing to the Data Operational (OLTP) Analytics (OLAP) Planning Predictive Text Search Spatial Processing Engines Relational Stores Row based Columnar ETL Data Quality Document Store Object Graph Store
  33. 33. Data Warehouse In-Database Analytics Operational Data Store DataBusiness Applications Copy ETL Business Intelligence Query Results Query Up to 1,000x faster Push processing down to dedicated hardware, less traffic Analytic Appliance Calculation Engine
  34. 34. Real-Time Data Operational Data Store Copy ETL Real-time replication — why have a separate operational data store? DataBusiness Applications Analytic ApplianceBusiness Intelligence
  35. 35. Transactions ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee that database transactions are processed reliably.
  36. 36. In-Memory Enterprise Applications E.g. Microsoft SQL Server In-Memory OLTP
  37. 37. In-Memory Enterprise Applications E.g. SAP S/4 HANA
  38. 38. Hybrid Transactional Analytical Processing Copy Business Applications Analytic ApplianceBusiness Intelligence Use a single platform for both analytics and applications Data
  39. 39. Virtuous Circle of Technology In-Memory Columnar Databases Hardware Acceleration Calculation Engine Columnar storage increases the amount of data that can be stored in limited memory (compared to disk) Column databases enable easier parallelization of queries In-memory processing gives more time for relatively slow updates to column data In-memory allows sophisticated calculations in real-time Hardware acceleration makes sophisticated calculations possible Each technology works well on its own, but combining them all is the real opportunity — provides all of the upside benefits while mitigating the downsides
  40. 40. Apache Spark MAP Reduce HDFS MAP Reduce Data Source 2 map() join() cache() transform Hadoop V1 Spark
  41. 41. Lots of Support for Spark
  42. 42. YARN HDFS Other Apps Files Files Files HANA-Spark Adapter for improved performance between distributed systems Compiled queries enable applications & data analysis to work more efficiently across nodes Familiar OLAP experience on Hadoop to derive business insights from big data such as drill-down into HDFS data Compiled Queries Spark Adapter Drill Downs SAP HANA in-memory platform Vora Spark Vora Spark In-Memory Store Application Services Database Services Integration Services Processing Services Vora Spark HANA-Spark Adaptor HANA Smart Data Access, UDFs, Others Extensive programming support for Scala, python, C, C++, R, and Java allow data scientists to use their tool of choice, Enable data scientists and developers who prefer Spark R, Spark ML to mash up corporate data with Hadoop/Spark data easily Optionally, leverage HANA’s multiple data processing engines for developing new insights from business and contextual data. Spark Extensions SAP HANA Vora
  43. 43. Persistence & Failover
  44. 44. Next-Generation Chips Are On Their Way NVM non-volatile memory
  45. 45. Scale Up 4,294,967,296x256x 16 bit 32 bit 64 bit 64 kilobytes 4 gigabytes 16 exabytes Directly addressable memory
  46. 46. What About Scale? There are now systems with more than half a petabyte of in-memory, and growing…
  47. 47. Balancing Data Temperature and Costs Hot Warm Cold Data is accessed frequently Data is not accessed frequently Data is only accessed sporadically Volume of data Performance (and direct cost) Many different solutions possible
  48. 48. What Type of In-Memory Is The Right One? Complex ROI calculations Data volumes Relative costs (?) Cost of storage Value of speed Value of agility
  49. 49. Fast-Moving Market
  50. 50. Hybrid vs. Pure In-Memory Tradeoffs dataduplication vssingle source Legacy + In Memory Approach ad hoc: made or done without planning because of an immediate need. (Merriam-Webster dictionary) DISPATCH/ MERGE Results Query CurrentData Stale Duplicated Data CurrentData Query Selectalldata fromone memorystore Results Pure In-Memory Approach Unpredictable Response Times Responses based on Obsolete Data Real-time Responses on Current Data replicatedvsreal-time unpredictable responsetimesvsconsistentresponsetimes
  51. 51. Top Benefits
  52. 52. Speed “If things seem under control, you’re just not going fast enough.” -Mario Andretti
  53. 53. Real-Time Operations Instead of analyzing the shards of glass after the accident, what if you could catch the vase BEFORE it hit the ground?
  54. 54. Agility (Speed of Change)
  55. 55. Simplification = Lower Costs “In-memory changes the cost equation through simplification. It can help save costs on hardware and software, as well as reduce labor required for administration and development needs. Based on a composite cost model, an in-memory platform can save an organization 37% across hardware, software, and labor costs, depending on various factors.” $0 $5,000,000 $10,000,000 $15,000,000 $20,000,000 $25,000,000 $30,000,000 $35,000,000 Baseline In-Memory Platform Custom App ERP DW
  56. 56. Lower Costs “Don’t let somebody say to you we can’t go in-memory because it’s so much more money. Acquisition costs may be higher. If you calculate out a TCO, it’s going to be less.” Donald Feinberg, Gartner
  57. 57. The price of light… …is less than the cost of darkness ROI = Return On Ignorance?
  58. 58. New, Simpler Infrastructures and Business Models Weissbeerger Beverage Analytics
  59. 59. Conclusion
  60. 60. Myths & Facts It’s a niche technology to run analytics faster It has been around since late 1990s The main users of in-memory analytics are SMBs Entire industries (SaaS, social networks, financial trading, online gaming) would not exist as we know them today without in- memory computing More than 50 software vendors deliver in- memory technology Small number of in-memory vendors Only for deep-pocketed organizations New and unproven Myths Facts
  61. 61. Business Impact of In-Memory Computing • Reducing applications running cost via data base/legacy applications offloading • Improving transactional applications performance • Enabling horizontal, elastic scalability (scale up/down) • Boosting response time in analytical applications • Low latency (<1 microsecond) application messaging • Dramatically shortening batch processes execution time • Enabling real-time, "self-service" business intelligence and unconstrained data exploration • Detecting correlations/patterns across million of events in "a blink of an eye" • Supporting "big data" (big data needs big memory) • Running transactional and analytical applications on the same physical dataset Run the business Grow the business Transform the business Opportunities: BusinessImpact
  62. 62. In-Memory Changes Everything “In-memory computing will have a long-term, disruptive impact by radically changing users’ expectations, application design principles, products’ architecture and vendors’ strategy.” — Gartner
  63. 63. Thank you! telliott@timoelliott.com twitter web site email

×