Building and Maintaining Bulletproof Systems with DataStax
•Download as PPTX, PDF•
0 likes•448 views
Tim Vincent, Solution Engineering Team Lead discusses about building and maintaining bulletproof systems for your cloud application with DataStax
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building and Maintaining Bulletproof Systems with DataStax
Similar to Building and Maintaining Bulletproof Systems with DataStax (20)
More from DataStax
More from DataStax (20)
Recently uploaded
Recently uploaded (20)
Building and Maintaining Bulletproof Systems with DataStax
- 1. Tim Vincent Solution Engineering Team Lead Welcome and Introduction to DSE Building and Maintaining Bullet Proof Systems
- 3. Feedback • Please use the Event App to give us your feedback • Everyone who submits feedback will be entered in the prize draw for a UE BOOM 2 by Ultimate Ears Bluetooth Waterproof Portable Speaker • Winner will be notified via email © DataStax, All Rights Reserved. 3
- 4. MDM: Customer 360, Product Catalog Personalization and Recommendation Internet of Things and Time Series Fraud Detection List Management Messaging Inventory Management Authentication © 2015 DataStax, All Rights Reserved. 4 DataStax Use Cases
- 5. © DataStax, All Rights Reserved. 5 Epic Scale Real Time Value
- 6. © DataStax, All Rights Reserved. 6 Numerous Endpoints Geographically Distributed Continuous Availability Instantaneously Responsive Immediately Decisive Predictably Scalable Cloud Applications
- 7. Architecture of a Cloud Application © 2015 DataStax, All Rights Reserved. 7 Web App Queue Ingest Operational Database
- 8. Architecture of a Cloud Application © 2015 DataStax, All Rights Reserved. 8 Web App Queue Ingest Operational Database Solr
- 9. Architecture of a Cloud Application © 2015 DataStax, All Rights Reserved. 9 Web App Queue Ingest Operational Database Hadoop Solr Queue Hive Search Batch Processing
- 10. Architecture of a Cloud Application © 2015 DataStax, All Rights Reserved. 10 Web App Queue Ingest Spark Streaming Operational Database Hadoop Solr Queue Hive Search Batch Processing
- 11. IBM Real-time Analytics Reference Architecture Source: Explore the advanced analytics platform (http://ibm.co/1PLAneB)© 2015 DataStax, All Rights Reserved. 11
- 12. Oracle Real-time Analytics Reference Architecture Source: Big Data in Financial Services and Banking (http://bit.ly/1GP8mmE)
- 13. Hadoop Real-time Analytics Reference Architecture Source: Designing Fraud-Detection Architecture That Works Like Your Brain Does (http://bit.ly/1OeAnDb)© 2015 DataStax, All Rights Reserved. 13
- 14. “70% of Hadoop deployments will not meet cost savings and revenue generation objectives due to skills and integration challenges.” © 2015 DataStax, All Rights Reserved. 14 Source: Hadoop promises not yet paying off (http://tek.io/1MrHtSx)
- 15. © 2015 DataStax, All Rights Reserved. 15
- 16. DSE Real-time Analytics Reference Architecture HTTP Application Message Queue Streaming Analytics Batch Analytics Real-time © 2015 DataStax, All Rights Reserved. 16
- 17. • Distributed NoSQL Database • Google Big Table • Amazon Dynamo • Continuous Availability • Disaster Avoidance • Linear Scale Performance • Add nodes to scale • Runs on Commodity Hardware • Cloud or on Premise San Francisco New York London Under Pinned by Apache Cassandra
- 18. Batch Analytics Indexing & Search Streaming Analytics Graph DSE Integrated Multi-Model Platform © 2015 DataStax, All Rights Reserved. 18
- 19. © DataStax, All Rights Reserved. 19 DataStax Enterprise Added Value
- 20. OpsCenter © 2015 DataStax, All Rights Reserved. 20 • Monitoring • Provisioning • Management Services • Backup/Restore • Capacity Planning • Performance • Best Practices
- 21. Netflix Ensures Constant Uptime with DataStax Enterprise World’s leading streaming media provider with digital revenue $1.5BN+ 95% of all Netflix data stored in DSE Introduction of ‘Profiles’ drove throughput to over 10M transactions per second Does 1 trillion transactions/day with DSE Replaced Oracle in six data centers, worldwide, 100% in the cloud 21
- 23. © DataStax, All Rights Reserved. 23 Epic Scale Real Time Value
- 24. © 2015 DataStax, All Rights Reserved. 24 Build Real Time Value Live Search and Indexing Streaming and Batch Analytics Advanced Replication
- 25. © 2015 DataStax, All Rights Reserved. 25 Operate at Scale OpsCenter Security and Auditing Services and Best Practices
- 26. © DataStax, All Rights Reserved. 26 DataStax Enterprise is the database for cloud applications.
Editor's Notes
- These capabilities have allowed us to penetrate deeply into financial services for mission critical applications extraordinarily quickly. I just want to give you a brief overview of some logos and use cases before we dive in. Real Time Payments Engine: Allied Payment Post Trade Risk Reporting/Capital Adeqquacy: Credit Suisse, UBS, BNP Paribas Global Reference Data Replication: UBS Market Data Capture/Replay: Blue Mountain, Sirca Post Trade Surveillance: UBS, Credit Suisse Real Time Fraud Analytics: Can’t share names – BofA – Fraud Analytics as part of the transactional flow. Not going to wait a minute for a period of time to determine fraud at the atm or time of a credit card transaction Forward Curve Analytics: UBS Challenger Commercial Teaching Step = #4 Emotional Impact – is telling stories about other companies and/or introducing the high-level challenges we solve Multi-Channel, Fraud, etc.
- (5 MIN)
- (5 MIN)
- To help place the database in context let’s talk about what a basic cloud application looks like. Cloud apps aren’t just about simple reads and writes to the database. Cloud apps gather event data. Then use that information to gain insights and personalize the response in real time. So, you've got a web application that generates events. Examples might be user searches, clickstreams, purchases, viewing history. Maybe it’s not a user-facing app. We could be talking about data points from sensors, banking transactions, or network activity. Whatever it is generating the events, they typically flow into a queue. Events that come out of the queue are handled by workers that run some logic, ingest the data. They may also trigger some other action like an alert or a recommendation.
- That’s a starting point. But cloud applications don’t deliver value with a database alone. Integrating search functionality supports more complex queries and features like faceted navigation and full text search. Even some basic analytics. People typically address this by adding another system like Solr or ElasticSearch and do synchronization between the main database and the search database. This adds complexity and operational challenges. Now you have two systems to operate. Now you have to keep them in sync. This requires you to build and maintain a custom integration. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- But search only gets you so far. To gain deeper insights from your data and offer more value to the user, you need more sophisticated analytics. In order to address that need people typically add a Hadoop cluster, and that adds a whole lot more complexity and cost. And Hadoop is slow. It’s batch oriented. Hadoop was designed to handle large, long-running analyses that trawl through data from the past day, week, month, or even longer. Hadoop is not immediately responsive. So hadoop doesn’t allow applications to react to what’s happening in real-time. If you’re looking for fraud, or generating context-relevant recommendations, Hadoop is not the answer. Cloud applications need to react to what is happening now. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- To overcome the limitations of hadoop and batch analytics you can add a streaming component, such as Spark Streaming or Storm. This allows you to augment your batch analytics and search, with contextual, real-time data. So this is the conceptual block diagram that shows the components you need to build a cloud application. You’ve got at least 8 separate systems supporting one "basic" application. Let’s take a look at some real reference architectures. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight The architecture slides help provide context as to why a platform or holistic response is necessary to address the complexity problem
- Here is a diagram from IBMs blog that is the functional equivalent of the system I just showed you. It's even more complex with even more moving parts. Each of these boxes is another system, with a different operational run book and another team supporting it. Each box is a staffing and training challenge, a drag on agility and a liability. Each black line is an integration or an ETL job to manage. This is why these solutions are so expensive. Too many moving pieces, too many teams to staff, too many things to learn. This is just too complex, which means missed deadlines, spiraling costs, and projects that never actually make it to production. I don't want to just pick on IBM, Oracle is just as bad. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight IBM and Hadoop architecture slides help reframe that it is about the architecture.
- Sure, there may be fewer boxes, but each one is hiding multiple instances of Hadoop, Exadata or Endeca and other technologies. The interconnectivity is equally complex. Some people think dropping in a NoSQL solution is the answer. But it’s not.
- This is Cloudera’s reference architecture for fraud detection. It's just as complex. Two Hadoop clusters, separate systems for streaming, queueing, ingestion, persistence and search…. And that’s only what we can see here. There are at least seven instances of zookeeper behind this, and we’re only looking at a single datacenter. Hadoop may help with scale, but it certainly does not address the underlying complexity issues. Challenger Commercial Teaching Step = #2 The Reframe – give perspective on challenges connecting to a bigger problem and provide explanations and implications of the insight IBM and Hadoop architecture slides help reframe that it is about the architecture.
- This is the reason why so many Hadoop deployments don't meet revenue or project goals. The big elephant in the room isn't just the Hadoop logo, it's that most big data solutions aren't delivering on their promise. Challenger Commercial Teaching Step = #3 Rational Drowning – this is the business case for why the reframes are worth customers time & insert the fear/uncertainty/doubt
- And this is just one project making technology decisions. Every team is doing this independently with no standardization, often with no support contract. It's resulted in a sprawl of vendors with no ability to gain efficiencies or develop best practices across the organization. Every team is making the same mistakes over and over again. Challenger Commercial Teaching Step = #3 Rational Drowning – this is the business case for why the reframes are worth customers time & insert the fear/uncertainty/doubt
- So how do we solve this? Take a look at this reference diagram and you'll immediately see that we're simpler. We'll dig into this in detail, but first I want to provide you with some context. Let’s look at how we make this work.
- DSE is more than just a database. It’s more than just Cassandra. In order to do something useful you need other functionality. You need to do something with the data. We've integrated an indexing engine with full text search support, both streaming and batch analytics using Spark. And coming soon is Graph with our acquisition of the team behind Titian DB. All of this is tightly integrated within the same platform. You persist the data, and we automatically maintain the indexes and materialized views in order to support these different query paths. This is what it means to be a multi-model database.
- DataStax Enterprise allows your application to provide real-time value while running at epic scale by dramatically simplifying the underlying architecture needed. Well, OpsCenter is the tool that allows you to simplify the operations of that architecture. From OpsCenter you can get a high level view of your DSE Cluster, see the overall health of your system or drill into individual nodes. Check to see whether all nodes are up and running green, under some heavy load, or if you have any failures. You can also configure proactive alerts to bring issues to your attention. For example, node failures, spikes in throughput, or latencies that don’t meet your SLAs. Not only monitoring though, you can also administer changes to your cluster through OpsCenter, run best practice checks to make sure your configuration is optimal for performance, backup and restore your clusters, add more nodes to fuel your applications growth, and more. OpsCenter allows you to manage your cloud application with ease, even while running at epic scale; allowing you to focus on your business value, not the administration of your architecture.
- (5 MIN)
- All of these features are designed to build real time value.
- DataStax also delivers features that let you operate at scale. We provide a visual management and monitoring tool called OpsCenter, which we’ll look at in a moment. Our database includes the security and auditing features you need to meet compliance regulations like HIPPA, PCI and SOX. And we have a skilled team of consultants who can provide the professional services and best practices you need to get into production.