Assessing New Databases– Translytical Use Cases
•Download as PPTX, PDF•
0 likes•101 views
Organizations run their day-in-and-day-out businesses with transactional applications and databases. On the other hand, organizations glean insights and make critical decisions using analytical databases and business intelligence tools. The transactional workloads are relegated to database engines designed and tuned for transactional high throughput. Meanwhile, the big data generated by all the transactions require analytics platforms to load, store, and analyze volumes of data at high speed, providing timely insights to businesses. Thus, in conventional information architectures, this requires two different database architectures and platforms: online transactional processing (OLTP) platforms to handle transactional workloads and online analytical processing (OLAP) engines to perform analytics and reporting. Today, a particular focus and interest of operational analytics includes streaming data ingest and analysis in real time. Some refer to operational analytics as hybrid transaction/analytical processing (HTAP), translytical, or hybrid operational analytic processing (HOAP). We’ll address if this model is a way to create efficiencies in our environments.
Recommended
Recommended
More Related Content
Similar to Assessing New Databases– Translytical Use Cases
Similar to Assessing New Databases– Translytical Use Cases (20)
More from DATAVERSITY
More from DATAVERSITY (20)
Recently uploaded
Recently uploaded (20)
Assessing New Databases– Translytical Use Cases
- 1. Assessing New Databases: Translytical Use Cases Presented by: William McKnight “#1 Global Influencer in Big Data” Thinkers360 President, McKnight Consulting Group A 2 time Inc. 5000 Company @williammcknight www.mcknightcg.com (214) 514-1444 Second Thursday of Every Month, at 2:00 ET With William McKnight
- 2. Enterprises Analysts Vendors • Keynote/Webinar Presentations – Online & In-Person – Great turnouts. • White Paper Development – Use our unique voice to talk about a theme important to you and tie your product to it. • Benchmark Services – Performance, Ease-of-Use, Functionality, TCO. We’ve done 40+ benchmarks; TPCs, others; databases (analytical, operational) and related (lake, integration, APIs, etc.). Impactful. • Day in the Life of Report – We go from zero to production and document the steps, creating comfort for the buyer to make the next step. • Teardown – Comparing and grading vs. competition across 50 +/- factors. Ideal for building product roadmaps. • Competitive Education – We teach vendor competitive teams about the competition with ½ day – 1 day hands-on workshops per competitor. • Technical Specification Development – i.e., Deployment Guide, Best Practices Guide, Reference Architecture. • Test Drives for demonstrations/booth – We build real-world relatable test drives/demos you can use to show off features or performance. • *NEW* McKnight Enterprise Contribution Ranking Report – We’re taking an industry and assessing market leaders against critical capabilities of the market. Industries available for prioritizing research. • Total Addressable Market Report. McKnight Consulting Group Vendor Offerings
- 3. OLTP vs OLAP OLTP • Process business interactions as they occur • Support limited query • Focus on IUD/individual transactions • Low latency and high throughput needed • ACID compliance • Normalized data model OLAP • Analytics/complex analysis • Offload of processing from OLTP • Dimensional data model • Lite data modification from source • Complex queries, frequently long-running • Large data accumulation 3
- 4. Capability Requirements • Analytics on live data, recent data and historical data • Real-time analytics calculated from across data domains • Pre-calculated data • Live analytics usable operationally • A seamless platform • Operational SLAs 4
- 5. Analytics Defined • Analytics is the process of utilizing data to enhance business processes. • Analytics is deeper than simple knowledge; they have depth. • There’s Analytic Projects and… • There’s Analytics Added to projects
- 6. Analytics Origin Batch • Broad Context • Drives Reactions • Action Options • “Most People” • Static Rules Real-Time • Immediate Context • Activity not in Batch • Dynamic Rules 6
- 7. Benefits of Real-Time Analytics • Speed to Insight • Customer Experience • Operational Excellence • Deeper Understanding 7
- 8. Translytical Use Cases • Portfolio Management • Wealth Management • Fraud Analytics • Risk Management • Algorithmic Trading • Crypto Exchange • SC/IoT Analytics • Real-Time Customer Experience • Network Telemetry • Geolocation Analysis • Field Support Optimization • Ad Optimization & Ad Serving • Streaming Media Quality Analytics • Real-Time Recommendations • Video Games • Telemetry Processing • IoT & Smart Meter Analytics • Predictive Maintenance • Geospatial Tracking
- 9. Next Best Offer/Touch • Need to incorporate not only analytics through last night, but also today, all morning, last hour and last second into screen render • Need to incorporate not just the user’s data but all users data – Need to correlate user to other users instantly • Only AI can operate at the needed scale 9
- 10. Financial Market • Billions of API Requests Daily • Need 5-10ms Average Query Response • Data to include: – Real-time and historical stock price – Cryptocurrencies, Forex, Commodities, Currencies, Premium Data • Front-Office Traders Need Real-Time Analysis
- 11. Healthcare • Genomic medicine • Virtual visits • Tele-health and AI Triage • AI Diagnostics • Robotics Automating Lab Work 11
- 12. Retailer • Better & personalized product recommendations for the consumers based on session data, historical order data, and trending products. • Continuous and automatic retraining the recommendation (ML) engine. • Near real-time data integration from their retail application to the analytical platform. • Identify potential compliance issues with customer data, classify and tag sensitive data with labels, and track how sensitive data is being used from the data source to the reports. • Integrate other systems such as their SAP ERP, email and instant messaging platforms with the analytical solution to get a full 360 view over their business operations and to improve customer satisfaction. • Save in operational costs while offering the best customer experience even during peak seasons such as Black Friday, Thanksgiving, Christmas, and Mother’s Day. 12
- 13. Metaverse • VR chairs, vests, scent generators, and better directional sound systems • Avatars fully virtual agents • Surgical implants to the metaverse 13
- 14. Transportation • Driverless and autonomous • Floating or vertical warehouses delivering packages • Urban transportation • Airbus drone-like popup concept 14
- 15. Cameras and Audio Recording • Cameras Will Be Abundant • Person’s Profile Will Be Evident • Third-Party Analytics • AI Will Decide … 15
- 16. Manufacturing • Real-Time Dashboards • Variety of data sources • When they ingest data they must recalculate the entire dataset because business rules change over time • Cross-matching survey results at the team and individual level • Need to know what impact various dimensions, such as product quality, support, cost, and more have on their NPS score • Processes that formerly required 10 steps are streamlined down to just one. 16
- 17. Asset Management • End-to-end asset visibility • Needed one place to discover all assets in environment – With instant context around risk, vulnerability, threat assessment and threat detection • 100 billion events per day – devices, firewalls, IoT, multi-tenant, ServiceNow and network traffic 17
- 18. Security Surveillance • Goal to view all sites in a single, cloud- based package – And offer analytics from video data • Real-Time Insights • Biggest challenge was scalability 18
- 19. Finance: Embedded • Started with easy to prototype, ingest data, do basic reports – Required replica sets • Performance constraints on writes to PostgreSQL • Had to do a bulk load of the data and it was so time-consuming that certain data was skipped 19
- 20. eSports • Need to offer real-time and historical live streaming data to analyze trends and performance across all genres, games, events, and channels • Need to work with thousands of time series data points in complex multi-gigabyte aggregated queries. • Analytics speed is the top priority • Understand spikes in viewership 21
- 21. Data Architecture Needs for Translytical Workloads • Fast Streaming Ingest (millions of events/second) • Low Latency • High Concurrency (thousands of concurrent users) • Unlimited Storage • Pipelines • Transactional Consistency
- 22. Data Architecture Ill-fit for Translytical Logs (Apps, Web, Devices) User tracking Operational Metrics Offload data Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or Transactiona l/ Context Data OLTP/ODS ETL Or EL with T in Spark Batch Low Latency Applications Files In- database analytics Reach through or ETL/ELT or Stream Processing or Stream Processing Q Q Data Warehouse Data Lake
- 25. NoSQL for Operational Big Data More data model flexibility – Web Services as a data model – No “schema first” requirement; load first Faster time to insight from data acquisition Relaxed ACID – Eventual consistency – Willing to trade consistency for availability – ACID would crush things like storing clicks on Google Low upfront software and development costs Fault-tolerant redundancy Linear Scaling to “webscale” 26
- 26. Event-Driven Architectures • Kafka Connector • Realtime Pub/Sub Messaging Platform • Edge Computing • Selective Feed to a Data Lake 27
- 27. Single Product Architectures • Single Table Storage for Transactions and Analytics – Fast IUD and Query – Simplified Data Architecture – Reduced Data Movement • Rowstore + Columnstore 28
- 28. Columnstore • SingleStore uses two storage types internally: an in-memory rowstore and a disk-based columnstore • Columnstore: 29
- 29. Azure Synapse Analytics Data Processing Transactional Database ML Model Training & Deployment Azure Machine Learning Azure Cosmos DB Core API E-Commerce Website Azure Kubernetes Service (AKS) Front-end Back-end Cart Profile Products Stock Azure ML managed online endpoint Deployed Recommender Azure Cosmos DB Analytical Store (Parquet) Analytical Store (HTAP) ADLS Gen2 (data lake) Data Lake + Historical Data Automatic Model deployment Synapse Link Enables automatic sync to analytical store (no ETL) Data Management & Governance Microsoft Purview Classify & protect sensitive data (customer profiles, etc.) Power BI Report & Visualize Power Apps M365 Dataverse Synapse Link Enterprise Data Sources Synapse Pipelines Azure Real-Time Environment
- 30. Amazon Redshift Data Processing Transactional Database ML Model Training & Deployment Amazon SageMaker Amazon DynamoDB E-Commerce Website Amazon Elastic Kubernetes Service (Amazon EKS) Front-end Back-end Cart Profile Products Stock SageMaker model endpoint Deployed Recommender Data Loading S3 (data lake) Data Lake + Historical Data Automatic Model deployment Amazon Glue Data Governance: • AWS Partner solutions • AWS Marketplace solutions AWS Real-Time Environment
- 31. Single Product Logs (Apps, Web, Devices) User tracking Operational Metrics Offload data Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or ETL Or EL with T in Spark Batch Low Latency Applications Files Transactional/ Context Data OLTP Reach through or Stream Processing Data Warehouse Data Lake
- 32. Single Vendor Solutions • SingleStore • Oracle • Snowflake Unistore • Cassandra • Azure • AWS • Google
- 33. Tweak on Traditional Architectures Logs (Apps, Web, Devices) User tracking Operational Metrics Offload data Raw Data Topics JSON, AVRO Processed Data Topics Sensors and / or Transactiona l/ Context Data OLTP/ODS ETL Or EL with T in Spark Batch Low Latency Applications Files In- database analytics Reach through or ETL/ELT or Stream Processing or Stream Processing Q Q Data Warehouse Data Lake Analytics
- 34. Multi-Vendor Architectures • Spark, Cassandra, Elastic, Druid, MongoDB • MySQL, Redis, DynamoDB • Redis, Elastic, Spark, Flink • Storm, Druid • PostgreSQL, Elastic
- 35. Benchmark • We found a single database competitiveness in operating effectively, actually putting it in a winning position for both transactional and operational workloads. – The use of a single database facilitates operational analytics and offers an efficient approach for any organization. • For the TPC H-like workload, it obtained a geometric mean better than both of the pure-play data warehouses. • In the TPC DS-like workload, an analytic db was superior, both with maintenance and without maintenance. Its 4.1 geometric mean outperformed the 1 db without maintenance, while its 3.9 with maintenance likewise bested the 1 db. • Given the vast superiority in transactional processing and the high competitiveness in analytic processing, the efficiencies of one database—the 1 db —across the spectrum of enterprise needs should be considered. • Platform costs favor the 1 db by 1.9x over 1 analytic db and 2.5x over the other in Year 1. • Development costs are 2.5x – 3x and Production Costs are 2.1x – 2.5x for the analytic db. • We calculated the annual costs of the platform stacks and the Time-Effort Costs (People Costs, Development Costs and Production Costs) and concluded that the 1 db is 2 times cheaper than 1 analytic stack and 2.5 times cheaper than the other over 3 years running enterprise-equivalent workloads.
- 36. Summary • Applications are moving translytical as the lines between operational and analytical blur • Analytics are deeper than simple knowledge; they have depth • The need for real-time analytics drives the need for a translytical architecture • There are examples in every industry • Traditional architectures do not meet the requirements • There are multiple vendor, multiple product/same vendor and single product options • Single product solutions combine Rowstore + Columnstore • Given the vast superiority in transactional processing and the high competitiveness in analytic processing, the efficiencies of one database—the 1 db —across the spectrum of enterprise needs should be considered
- 37. Upcoming Topics • Assessing New Database Capabilities: Multi-Model • MLOps: Applying DevOps to Competitive Advantage • 2023 Trends in Enterprise Analytics • Showing ROI for your Analytic Project • Architecture, Products and Total Cost of Ownership of the Leading Machine Learning Stacks 39 Second Thursday of Every Month, at 2:00 ET
- 38. Assessing New Databases: Translytical Use Cases Presented by: William McKnight “#1 Global Influencer in Big Data” Thinkers360 President, McKnight Consulting Group A 2 time Inc. 5000 Company @williammcknight www.mcknightcg.com (214) 514-1444 Second Thursday of Every Month, at 2:00 ET #AdvAnalytics
Editor's Notes
- HTAP, HOAP, Operlytical, Event-driven*
- The dw is not dead, but it is dying. Data lake.
- Analytics needed even if they are not traditionally stored together (e.g. real-time customer event data alongside CRM data; network sensor data alongside marketing campaign management data) Between the pre-calc and the live is design…. On-demand vs. Continuous Real-time Analytics
- Projects that are classified as “analytics”. And there’s analytics added to projects. At some point, all projects are becoming analytic projects which makes it fair to just measure the project roi.
- Red light…. Stop. Car running low on gas, gas station 20 minutes away, home 30 minutes away, tomorrow not as busy can gas in morning.
- Speed to Insight: The primary benefit of real-time analytics is of course speed. It speeds up time to insight and lets businesses work faster to make necessary changes to systems or act on any critical information discovered. This can help organizations not only flag potential problems and mitigate risk, but also seize opportunities when they matter. Customer Experience: Real-time analytics can help businesses anticipate problems and streamline operations to improve the overall customer experience. These on-the-fly adjustments greatly influence customer interactions and can help improve the end-to-end experience. Operational Excellence: Real-time analytics allows organizations to gain a clear view of the business and understand what needs to be done to address potential operational issues. It also allows users to understand what resources are available to make those changes. Deeper Understanding: When there is a need for deeper analytics to make a business decision, real-time analytics can help compare real-time and historical data to inform the decision. Most r/t arch about r/t ingest
- Keywords: real-time, real-time analytics, operational excellence, operational analytics, reatl-time DW: Real-time analytics essentially means that data is provided for analysis almost immediately once it is collected. Way of the future; No 2nd store of data: DW
- Maybe somebody just became a correlated user theme
- Foreign exchange Premium data comes from a growing community of curated partners, such as: Wall Street Horizon Fraud Factors Audit Analytics ValuEngine Stocktwits And much more
- Recalls Outbreaks Latest findings pandemic footprint Human beings have roughly 20,500 genes, in DNA, housed in each and every one of the trillions of cells that make you who you are. What cases what action… it’s complicated. Batch anal needed.
- 360 incl the now
- MV is about simulation. Avatars able to act, within tightly defined parameters, as our agents, our companions, and some may even be considered co-workers. unable to tell the difference between a virtualized real person and an AI-driven avatar. we will vrutally be able to travel the world and experience life on other planets, all from home. Metaverse will give a feeling of actually being there with your family/friends. parallel life in the Metaverse. It has become absolutely necessary for your existence. It is very difficult to be operational outside of the Metaverse. You are connected via multiple devices, wearables and even brain chips. You live in a mixed reality where physical and digital converge. Many people opt to spend most of their day in virtual worlds where they can become whoever they want and live the way they always dreamed. unlimited freedoms in their personal virtual worlds - no liniits. NFTs and crypto …. Take off later. bitcoin will displace the US dollar as the primary form of global finance by 2050
- Traffic and weather – current and patterns. Constantly changing.
- Imagine this: You walk into a furnitue showroom virtually and before you say anything, the store knows your name, employment status, car-buying history, and credit rating. ADD: Where you’ve been today, clothes you’re wearing, etc. Already, data brokers such as Acxiom and LexisNexis compile reams of information on all of us. Clients can purchase a dossier on your criminal, consumer, and marital past. it’s only a matter of time before data brokers begin drawing from online-dating profiles and social-media posts as well. Right now, clients have to log in and search for people by name or buy lists of people with certain traits. But as facial-recognition technology becomes more widespread, , any device with a camera and the right software could automatically pull up your information. Eventually, someone might be able to point a phone at you (or look at you through smart contact lenses) and see a bubble over your head marking you as unemployed or recently divorced. We’ll no longer be able to separate our work selves from our weekend selves. Instead our histories will come bundled as a pop-up on strangers’ screens. With the advent of the Internet of Things, appliances and gadgets will monitor many aspects of our lives, from what we eat to what we flush. Devices we talk to will record and upload our conversations, as Amazon’s Echo already does. Even toys will make us vulnerable. Kids say the darndest things, and the talking Hello Barbie doll sends those things wirelessly to a third-party server, where they are analyzed by speech-recognition software and shared with vendors. Even our thoughts could become hackable. The technology company Retinad can use the sensors on virtual-reality headsets to track users’ engagement. Future devices might integrate electrodes to measure brain waves. In August, Berkeley engineers announced that they had produced “neural dust,” implantable electrodes just a millimeter wide that can record brain activity for scientific or medical purposes. Chicago police use an algorithm that analyzes arrest records, social networks, and other data to identify future criminals.
- xxx previously had to run advanced analytics offline xxx. “if you looked at the dashboard and wanted to drill through, the waiting times were longer than 2 seconds. If it's not instant or very close to instant, it becomes painful. At that point, people just don’t do the analytics and valuable information is lost. If you don't use it, and if you don't analyze, you can’t find these things, you're not going to improve your business.” The data sources could be almost anything, from databases to IoT devices. They can now drill down into things like NPS to get at the root cause of a score, drive those insights back into the business, improve their scores, and most importantly, retain their end customers . In the past, xxx had to provide these analytic insights by moving data into SPSS, which was painfully slow. With a translytical appraoch, they can now slice and dice data in real time, and in the NPS example, instantly understand the validity of a data correlation.
- Armis originally launched its platform using a PostgreSQL database. Over time, the time-based data set got too large for Postgres to handle. At this point the team migrated this data set from 400+ PostgresSQL databases into a huge Elasticsearch cluster (160 nodes). the entire data pipeline including ElasticSearch cost more than $1 million annually.
- Embedded finance is when non-financial companies offer their customers access to credit through their technology platform. Customers can be individuals or businesses, and the credit can be offered by the company or by a third party. The replicated data needed to be re-ingested, and the dashboard only refreshed once every 24 hours, leading to a serious and unacceptable lag in data freshness. didn’t store a lot of information due to performance constraints on writes to PostgreSQL, and data from other partners was nearly impossible to obtain so Ant Money could enrich its first-party data. Ant Money had to do a bulk load of the data and it was so time-consuming that certain data was skipped.
- s customers are the biggest companies in the eSports industry: game publishers, eSports organizers, and other brands. & Help companies analyze eSports data so they can understand how they’re doing and ways they can optimize time and resources Both real-time and historical data were needed to provide the full context of the live streams and eSports events. The data ingestion pipeline included manual metadata input, third-party fact tables, and automated systems.
- Linkedin 80million events/sec + bulk loading Parallel, high-scale streaming data ingest; immediate avail
- Ppl on 1 side. Most of the time post-op is “learning” 1 db solutions: Doing analytics with operational DBs or tying together multiple databases to power their applications with analytics Is it more anal needing op or op needing anal, which way coming? It’s anal trying to do op & fail . Also mysql, postgresql to anal fail.
- r/t for access Single source
- databricks Lake+dw. All points of integration are points of failure. Data lakes (cloud stg) emerged to handle raw data in a variety of formats on cheap storage for data science and machine learning, though lacked critical features from the world of data warehouses: they do not support transactions, they do not enforce data quality well, and their lack of consistency/isolation makes it almost impossible to mix appends and reads, and batch and streaming jobs. There are a few key technology advancements that have enabled the data lakehouse: metadata layers for data lakes to set up drill through paths new query engine designs providing high-performance SQL like execution on data lakes access for data science and machine learning tools. Lake concerns itself w/ DQ not offloads all of the major data platform vendors have converged their messaging around the concept of a lakehouse architecture that takes the best attributes of traditional data warehouses and enables them to run on platforms with data- lake storage architectures.
- Column stores, key-value stores, document stores\ Data fit for nosql
- Linkedin 80million messages/sec
- Readers don’t need to wait on writers. Each version of the row is stored as a fixed-sized struct (variable- length fields are stored as pointers) according to the table schema, along with bookkeeping information such as the timestamp and the commit status of the version. Oracle (Oracle has a dual store approach rather than a single store like SingleStoreDB's Universal Storage) Snowflake has announced combining transactional and #analytical data with Unistore. SingleStore, ….what we encounter is customers trying to do analytics with operational DBs or tying together multiple databases to power their applications with analytics. We also replace a lot of the 1st gen operational DBs (such as the MySQL, Postgres, RDS) and also augment data warehouses or Hadoop to power real-time analytics.
- Microsoft solution Microsoft Azure and Microsoft Intelligent Data Platform Azure Kubernetes Service Azure Cosmos DB Synapse Link for Cosmos DB Synapse Analytics, Synapse Pipelines, ADLS Gen2 Azure ML Power BI Microsoft Purview
- AWS solution Amazon Elastic Kubernetes Service (Amazon EKS) Amazon DynamoDB Amazon Glue Amazon Redshift, S3 Amazon SageMaker For Data Governance: 3rd party Marketplace/Partner solutions
- GCP solution Google Kubernetes Engine (GKE) Cloud Firestore Cloud Data Fusion BigQuery, Cloud Storage, Cloud Dataprep, Cloud Dataflow Vertex AI Prediction, Vertex AI For Data Governance: Dataplex, requires a separate Dataplex lake
- Diff: 1 vendor, 1 product
- Oracle has a dual store approach rather than a single store like SingleStoreDB's Universal Storage). You need (i) basic Oracle DB license (ii) diag + tuning pack (iii) Oracle RAC option (iv) Exadata (for the columnar compression/performance) (v) partitioning pack (vi) Active Data guard option Snowflake has announced combining transactional and #analytical data with Unistore
- Some Ppl on 2 sides. 1. Data lakes can be difficult to manage and govern due to their size and complexity. 2. Data lakes can be difficult to extract from regularly due to the variety and volume of data they contain. Need to add a cache like Redis Op db often nosql
- Organizations are often reluctant to attempt analyzing real-time data, fearing the analytical workload will hamper the performance of the operational work that has to be the priority.
- 2 anal db, 1 1 db solution
- 2 anal db, 1 1 db solution
- Some of these BP you’ll see next month in mature env