Rob Harrop- Key Note The God, the Bad and the Ugly - NoSQL matters Paris 2015

•

1 like•1,012 views

The impact that NoSQL has had on the technology community cannot be overstated. The proliferation of new and exciting data systems has led to a slew of interesting solutions to problems that were once solved the relational way. In this session we explore all that is great and good about NoSQL: the innovative software, the clever storage paradigms and the reigniting of developer interest in data access. It is unfortunate that NoSQL is not only a force for good in our community. We'll explore some of the darker corners of NoSQL: the disregard for years of proven technology, the overbearing hype, the overblown marketing and the ever present arguments over which technology is best. We close the session by exploring what can be done to extract even more value from the NoSQL movement, where we can improve how the community interacts with the larger technology community and what the future holds for data access technologies.

NoSQL: The Good, the
Bad and the Ugly
Rob Harrop

• Co-founder of Spring Source
• Writer of books
• Now co-founder and CTO at Bamboo
• UK-based consumer ﬁnance startup

• I like data
• My business likes data
• NoSQL matters to me!

–Martin Fowler
“When we say "x is a NoSQL database" I think
it's silly to interpret NoSQL as "Not Only"
because that would render the term
meaningless. (You could then reasonably argue
that SQL Server (say) is a NoSQL database.)”

• Not using the relational model (nor the SQL
language)
• Open source
• Designed to run on large clusters
• Based on the needs of 21st century web properties
• No schema, allowing ﬁelds to be added to any
record without controls

Data Model
• Relational
• Key-value
• Document
• Column
• Graph
• And many more…

Query Model
• SQL
• Key/value
• Map/reduce
• JSON path
• Search

Storage
• Distributed
• Replicated
• In-memory

Durability
• Ephemeral stores
• System-of-record stores
• Archive store

Consistency
• Linearised
• Eventual
• Causal

Operations
• Backup
• Failover
• HA
• SaaS

• Right tool for the job
• Choose the simplest tool
• Have more fun

• Making the choice is hard
• Danger of analysis paralysis
• Might head down the wrong road

• Consistency is front and centre
• Data models aren’t always straightforward
• Might head down the wrong road

• Illusions of reliability and consistency are removed
• Deeper understanding cannot hurt

• Many elaborate claims
• Plenty of dis-information
• Difﬁcult for beginners to ﬁnd the kernel of truth

• Good quality experience reports
• Solid base of literature
• Deep analysis
• Kyle Kingsbury - Jepsen
• Testing isn’t all that hard

• Sheer number of choices leads to many opinions
• Exaggerated claims compound with opinions
• Claims can be difﬁcult to verify
• Evangelism leads to emotional investment

• No longer one big data model
• Compose models to solve a problem
• Different models with different requirements
• Model per micro-service?

• Split by use-case
• Read model for analytics
• Search model
• Low latency write model
• Layer within use-case
• Caching
• System of record
• Separate write model (CQRS)

• AWS
• Redshift, DynamoDB, Elasticache, S3, RDS
• Google Cloud
• SQL, Storage, BigQuery, Datastore
• Azure
• SQL, DocumentDB, Redis, Storage

Learn
• Understand the concepts
• Don’t get overwhelmed!
• Read the literature
• The Morning Paper from Adrian Colyer

Specify
• What are your performance/reliability/scalability
requirements?
• How will you handle differing consistency models?
• Who owns what data?
• How long does data live for?
• Does data live in only one place?

Get Started
• Implement something small
• Caching, search, analysis
• Use ancillary technologies
• Blob store
• Archive storage

Get Serious
• Get the best out of your SQL database
• Identify core functional areas for NoSQL
• Low-latency
• Document
• Migrate gradually
• Consider parallel runs

Get Sophisticated
• Segment system into multiple models
• Identify consistency/durability/reliability boundaries
• Fit the right store to each model
• Layer stores within a model as needed

What's hot

Writing Space and the Cassandra NoSQL DBMS

DataStax Academy

Иван Глушков (Echo)Ontico

MongoDB

Rony Gregory

BERGER STEVENS BBC Genome

FIAT/IFTA

Elastic @ GetYourGuide

Tobias Rein

Web designing course content

Trainme Softtech

Tips & Tricks SQL in the City Seattle 2014

Ike Ellis

Styling frameworks v2nelsonwong1012

Mongo Seattle - The Business of MongoDBJustin Smestad

Static website generator

Sylvain Witmeyer

Databases and Queries: Matching Performance and Reliability.

Orchestrate

Promotional presentation on new Book "MariaDB - Beginners Guide"

Rodrigo Ribeiro

Postgres resources for beginners

Malcolm McLean

MongoDB Hadoop DC

Mike Dirolf

MongoDB at CodeMash 2.0.1.0

Mike Dirolf

Introduction to Restful Web Services

weili_at_slideshare

Mongo db first steps with csharpSerdar Buyuktemiz

Creating Game Leaderboards with Redis

Social Point

What we learned at PASS Summit in 2019

Red Gate Software

RavenDB in the wild

Mauro Servienti

What's hot (20)

Writing Space and the Cassandra NoSQL DBMS

Иван Глушков (Echo)

MongoDB

BERGER STEVENS BBC Genome

Elastic @ GetYourGuide

Web designing course content

Tips & Tricks SQL in the City Seattle 2014

Styling frameworks v2

Mongo Seattle - The Business of MongoDB

Static website generator

Databases and Queries: Matching Performance and Reliability.

Promotional presentation on new Book "MariaDB - Beginners Guide"

Postgres resources for beginners

MongoDB Hadoop DC

MongoDB at CodeMash 2.0.1.0

Introduction to Restful Web Services

Mongo db first steps with csharp

Creating Game Leaderboards with Redis

What we learned at PASS Summit in 2019

RavenDB in the wild

Similar to Rob Harrop- Key Note The God, the Bad and the Ugly - NoSQL matters Paris 2015

NoSql - mayank singh

Mayank Singh

PostgreSQL, your NoSQL database

Reuven Lerner

NOsql Presentation.pdf

AkshayDwivedi31

Oracle Week 2016 - Modern Data Architecture

Arthur Gimpel

Solr cloud the 'search first' nosql database extended deep dive

lucenerevolution

Presented by Mark Miller, Software Engineer, Cloudera As the NoSQL ecosystem looks to integrate great search, great search is naturally beginning to expose many NoSQL features. Will these Goliath's collide? Or will they remain specialized while intermingling – two sides of the same coin. Come learn about where SolrCloud fits into the NoSQL landscape. What can it do? What will it do? And how will the big data, NoSQL, Search ecosystem evolve. If you are interested in Big Data, NoSQL, distributed systems, CAP theorem and other hype filled terms, than this talk may be for you.

NoSQLRadu Vunvulea

NoSQL in the context of Social Web

Bogdan Gaza

UNIT I Introduction to NoSQL.pptx

Rahul Borate

No SQL- The Future Of Data Storage

Bethmi Gunasekara

Key-Value NoSQL Database

Heman Hosainpana

UNIT I Introduction to NoSQL.pptx

Introducción a NoSQL

Lviv EDGE 2 - NoSQL

NoSQL and MongoDB

Scaling the Web: Databases & NoSQL

Richard Schneeman

Comparative study of modern databases

Anirban Konar

No sql or Not only SQL

Ajay Jha

Introduction to CQRS - command and query responsibility segregation

Andrew Siemer

noSQL choiceslugiamaster4

Data Modeling for NoSQL

Tony Tam

Similar to Rob Harrop- Key Note The God, the Bad and the Ugly - NoSQL matters Paris 2015 (20)

NoSql - mayank singh

PostgreSQL, your NoSQL database

NOsql Presentation.pdf

Oracle Week 2016 - Modern Data Architecture

Solr cloud the 'search first' nosql database extended deep dive

NoSQL

NoSQL in the context of Social Web

UNIT I Introduction to NoSQL.pptx

No SQL- The Future Of Data Storage

Key-Value NoSQL Database

UNIT I Introduction to NoSQL.pptx

Introducción a NoSQL

Lviv EDGE 2 - NoSQL

NoSQL and MongoDB

Scaling the Web: Databases & NoSQL

Comparative study of modern databases

No sql or Not only SQL

Introduction to CQRS - command and query responsibility segregation

noSQL choices

Data Modeling for NoSQL

More from NoSQLmatters

Nathan Ford- Divination of the Defects (Graph-Based Defect Prediction through...

NoSQLmatters

While metrics generated by static code analysis are well established as predictors of possible future defects, there is another untapped source of useful information, namely your source code revision history. This presentation will discuss converting this revision information into a graph representation, various defect prediction models and how to generate their related change metrics through graph traversal, as well as the potential applications and benefits of these graph enabled prediction models.

Stefan Hochdörfer - The NoSQL Store everyone ignores: PostgreSQL - NoSQL matt...

NoSQLmatters

Adrian Colyer - Keynote: NoSQL matters - NoSQL matters Dublin 2015

NoSQLmatters

Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spar...

NoSQLmatters

Dan Sullivan - Data Analytics and Text Mining with MongoDB - NoSQL matters Du...

NoSQLmatters

Data analysis is an exploratory process that requires a variety of tools and a flexible data store. Data analysis projects are easy to start but quickly become difficult to manage and error prone when depending on file-based data storage. Relational databases are poorly equipped to accommodate the dynamic demands complex analysis. This talk describes best practices for using MongoDB for analytics projects. Examples will be drawn from a large scale text mining project (approximately 25 million documents) that applies machine learning (neural networks and support vector machines) and statistical analysis. Tools discussed include R, Spark, Python scientific stack, and custom pre-processing scripts but the focus is on using these with the document database.

Mark Harwood - Building Entity Centric Indexes - NoSQL matters Dublin 2015

NoSQLmatters

Sometimes we need to step back and take a look at the bigger picture - not just counting huge piles of individual log records, but reasoning about the behaviors of the people who are ultimately generating this firehose of data. While your DevOps folks care deeply about log records from a machine utlization perspective, marketing wants to know what these records tell us about the customers' needs. Elasticsearch Aggregations are a great feature but are not a panacea. We can happily use them to summarise complex things like the number of web requests per day broken down by geography and browser type on a busy website, but we would quickly run out of memory if we tried to calculate something as simple as a single number for the average duration of visitor web sessions when using the very same dataset. Why does this occur? A web session duration is an example of a behavioural attribute not held on any one log record; it has to be derived by finding the first and last records for each session in our weblogs, requiring some complex query expressions and a lot of memory to connect all the data points. We can maintain a more useful joined-up-picture if we run an ongoing background process to fuse related events from one index into ?entity-centric? summaries in another index e.g: • Web log events summarised into ?web session? entities • Road-worthiness test results summarised into ?car? entities • Reviews in a marketplace summarised into a ?reviewer? entity Using real data, this session will demonstrate how to incrementally build entity-centric indexes alongside event-centric indexes by using simple scripts to uncover interesting behaviours that accumulate over time. We'll explore: • Which cars are driven long distances after failing roadworthiness tests? • Which website visitors look to be behaving like ?bots?? • Which seller in my marketplace has employed an army of ?shills? to boost his feedback rating? Attendees will leave this session with all the tools required to begin building entity-centric indexes and using that data to derive richer business insights across every department in their organization.

Prassnitha Sampath - Real Time Big Data Analytics with Kafka, Storm & HBase -...

NoSQLmatters

Relevance and Personalization is crucial to building personalized local commerce experience at Groupon. Talk covers overview of the real time analytics infrastructure that handles over 3 million events/ second and stores and scales to billions of data points. Solution covers how our Kafka -> Storm -> Redis/ HBase pipeline is used to generate real time analytics for hundreds of millions of users of Groupon. Solution includes various architecture design choices and tradeoffs including some interesting algorithmic choices such as Bloom Filters & Hyper Log Log. Attendees can take away learnings from our real-life experience that can help them understand various tuning methods, their tradeoffs and apply them in their solutions

Akmal Chaudhri - How to Build Streaming Data Applications: Evaluating the Top...

NoSQLmatters

Building applications on streaming data has its challenges. If you are trying to use programs such as Apache Spark or Storm to build applications, this presentation will explain the advantages and disadvantages of each solution and how to choose the right tool for your next streaming data project. Building streaming data applications that can manage the massive quantities of data generated from mobile devices, M2M, sensors and other IoT devices, is a big challenge that many organizations face today. Traditional tools, such as conventional database systems, do not have the capacity to ingest data, analyze it in real-time, and make decisions. New technologies such as Apache Spark and Storm are now coming to the forefront as possible solutions to handing fast data streams. Typical technology choices fall into one of three categories: OLAP, OLTP, and stream-processing systems. Each of these solutions has its benefits, but some choices support streaming data and application development much better than others. Employing a solution that handles streaming data, provides state, ensures durability, and supports transactions and real-time decisions is key to benefitting from fast data. During this presentation you will learn: - The difference between fast OLAP, stream-processing, and OLTP database solutions. - The importance of state, real-time analytics and real-time decisions when building applications on streaming data. - How streaming applications deliver more value when built on a super-fast in-memory, SQL database.

Michael Hackstein - NoSQL meets Microservices - NoSQL matters Dublin 2015

NoSQLmatters

Just a few years ago all software systems were designed to be monoliths running on a single big and powerful machine. But nowadays most companies desire to scale out instead of scaling up, because it is much easier to buy or rent a large cluster of commodity hardware then to get a single machine that is powerful enough. In the database area scaling out is realized by utilizing a combination of polyglot persistence and sharding of data. On the application level scaling out is realized by microservices. In this talk I will briefly introduce the concepts and ideas of microservices and discuss their benefits and drawbacks. Afterwards I will focus on the point of intersection of a microservice based application talking to one or many NoSQL databases. We will try and find answers to these questions: Are the differences to a monolithic application? How to scale the whole system properly? What about polyglot persistence? Is there a data-centric way to split microservices?

Chris Ward - Understanding databases for distributed docker applications - No...

NoSQLmatters

In this talk we'll focus on the use of Crate alongside Weave in Docker containers, the technical challenges, best practices learned, and getting a big data application running alongside it. You'll learn about the reasons why Crate.IO is building "yet another NoSQL database" and why it's unique and important when running web scale containerized applications. We'll show why the shared-nothing architecture is so important when deploying large clusters in containers and how it addresses the issues and fears of a Docker-based persistence layer. You will learn how to deploy a Crate cluster in the cloud within minutes using Docker, some of the challenges you'll encounter, and how to overcome them in order to scale your backends efficiently. We focused on super simple integration with any cloud provider, striving it to be as turnkey as possible with minimal up-front configuration required to establish a cluster. Once established, we'll show how to scale the cluster horizontally by simply adding more nodes. The session will also give you examples when you should use Crate compared to other similar technologies such as MongoDB, Hadoop, Cassandra or FoundationDB. We'll talk about this approach's strengths and what types of applications are well-suited for this type of data store, as well what is not. Finally we'll outline how to architect an application that is easy to scale using Crate and Docker.

Philipp Krenn - Host your database in the cloud, they said... - NoSQL matters...

NoSQLmatters

More than two years ago we faced the decision whether to run our MongoDB database on Amazon's EC2 ourselves or to rely on a Database as a Service provider. Common wisdom told us that a well known provider, focusing all its knowledge and energy on running MongoDB, would be a better choice than us trying it on the side. Well, this talk describes what can go wrong, since we have seen a lot of interesting minor and major hiccups — including stopped instances, broken backups, a major security incident, and more broken backups. Additionally, we discuss some reasons why a hosted solution is not always the better choice and which new challenges arise from it.

Lucian Precup - Back to the Future: SQL 92 for Elasticsearch? - NoSQL matters...

NoSQLmatters

What if we would try to make Elasticsearch SQL 92 compliant (http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt)? This wouldn't serve that much nowadays, you would say. Well, we actually tried to do the exercise and we have some interesting conclusions. While we take Elasticsearch as an example for this "side by side", the issues we are addressing also apply to nosql in general. With this unusual exercise, we take the occasion to compare relational databases / sql with Elasticsearch / nosql on all the levels : functionality, semantics, performance and user experience.

Bruno Guedes - Hadoop real time for dummies - NoSQL matters Paris 2015

NoSQLmatters

DuyHai DOAN - Real time analytics with Cassandra and Spark - NoSQL matters Pa...

NoSQLmatters

Apache Spark is a general data processing framework which allows you perform map-reduce tasks (but not only) in memory. Apache Cassandra is a highly available and massively scalable NoSQL data-store. By combining Spark flexible API and Cassandra performance, we get an interesting alternative to the Hadoop eco-system for both real-time and batch processing. During this talk we will highlight the tight integration between Spark & Cassandra and demonstrate some usages with live code demo.

Benjamin Guinebertière - Microsoft Azure: Document DB and other noSQL databas...

NoSQLmatters

When deploying your service to Microsoft Azure, you have a number of options in terms of noSQL: you can install databases on Linux or Windows virtual machines by yourself, or via the marketplace, or you can use open source databases available as a service like HBase or proprietary and managed databases like Document DB. After showing these options, we'll show Document DB in more details. This is a noSQL database as a service that stores JSON.

David Pilato - Advance search for your legacy application - NoSQL matters Par...

NoSQLmatters

How do you mix SQL and NoSQL worlds without starting a messy revolution?This live coding talk will show you how to add Elasticsearch to your legacy application without changing all your current development habits. Your application will have suddenly have advanced search features, all without the need to write complex SQL code!David will start from a Spring, Hibernate and Postgresql based application and will add a complete integration of Elasticsearch, all live from the stage during his presentation.

Tugdual Grall - From SQL to NoSQL in less than 40 min - NoSQL matters Paris 2015

NoSQLmatters

During this live-coding session, Tugdual will move an old fashion full SQL application (JavaEE) to the new NoSQL world.Using MongoDB, and REST, he will show the benefits of this new architecture: * Easyness * Flexibility * High availability * Scalability; During this presentation, you will learn more about: * Document Oriented Model * JSON * REST * Iterative development; This demonstration is also a good opportunity to see how you can migrate data from a relational database, and the various schema options.

Gregorry Letribot - Druid at Criteo - NoSQL matters 2015

NoSQLmatters

How do you monitor performance for one of your clients on a specific user segmentation when dealing with billions of events a day ? With over 2 billion ads served and 230Tb of data processed a day, we at Criteo have a comprehensive need for an interactive analytics stack. And by interactive, we mean a querying system with dynamic filtering to drill down over multiple dimensions, answering within sub-second latency. This session will take you on our journey with Druid, ""an open-source data store designed for real-time exploratory analytics on large data sets"". We will explore Druid's architecture and noticeable concepts, how relevant they are for some use cases and how it really performs.

Michael Hackstein - Polyglot Persistence & Multi-Model NoSQL Databases - NoSQ...

NoSQLmatters

In many modern applications the database side is realized using polyglot persistence – store each data format (graphs, documents, etc.) in an appropriate separate database. This approach yields several benefits, databases are optimized for their specific duty, however there are also drawbacks: * keep all databases in sync * queries might require data from several databases * experts needed for all used systems A multi-model database is not restricted to one data format, but can cope with several of them. In this talk i will present how a multi-model database can be used in a polyglot persistence setup and how it will reduce the effort drastically.

Alexandre Vasseur - Evolution of Data Architectures: From Hadoop to Data Lake...

NoSQLmatters

Come to this deep dive on how Pivotal's Data Lake Vision is evolving by embracing next generation in-memory data exchange and compute technologies around Spark and Tachyon. Did we say Hadoop, SQL, and what's the shortest path to get from past to future state? The next generation of data lake technology will leverage the availability of in-memory processing, with an architecture that supports multiple data analytics workloads within a single environment: SQL, R, Spark, batch and transactional.

More from NoSQLmatters (20)