Arrays can be simple; arrays can be complex. JSON arrays give you a method to collapse the data model while retaining structure flexibility. Arrays of scalars, objects, and arrays are common structures in a JSON data model. Once you have this, you need to write queries to update and retrieve the data you need efficiently. This talk will discuss modeling and querying arrays. Then, it will discuss using array indexes to help run those queries on arrays faster.
From SQL to NoSQL: Structured Querying for JSONKeshav Murthy
Can SQL be used to query JSON? SQL is the universally known structured query language, used for well defined, uniformly structured data; while JSON is the lingua franca of flexible data management, used to define complex, variably structured data objects.
Yes! SQL can most-definitely be used to query JSON with Couchbase's SQL query language for JSON called N1QL (verbalized as Nickel.)
In this session, we will explore how N1QL extends SQL to provide the flexibility and agility inherent in JSON while leveraging the universality of SQL as a query language.
We will discuss utilizing SQL to query complex JSON objects that include arrays, sets and nested objects.
You will learn about the powerful query expressiveness of N1QL, including the latest features that have been added to the language. We will cover how using N1QL can solve your real-world application challenges, based on the actual queries of Couchbase end-users.
Tuning for Performance: indexes & QueriesKeshav Murthy
There are three things important in databases: performance, performance, performance. From a simple query to fetch a document to a query joining millions of documents, designing the right data models and indexes is important. There are many indices you can create, and many options you can choose for each index. This talk will help you understand tuning N1QL query, exploiting various types of indices, analyzing the system behavior, and sizing them correctly.
Understanding N1QL Optimizer to Tune QueriesKeshav Murthy
Every flight has a flight plan. Every query has a query plan. You must have seen its text form, called EXPLAIN PLAN. Query optimizer is responsible for creating this query plan for every query, and it tries to create an optimal plan for every query. In Couchbase, the query optimizer has to choose the most optimal index for the query, decide on the predicates to push down to index scans, create appropriate spans (scan ranges) for each index, understand the sort (ORDER BY) and pagination (OFFSET, LIMIT) requirements, and create the plan accordingly. When you think there is a better plan, you can hint the optimizer with USE INDEX. This talk will teach you how the optimizer selects the indices, index scan methods, and joins. It will teach you the analysis of the optimizer behavior using EXPLAIN plan and how to change the choices optimizer makes.
Querying NoSQL with SQL - KCDC - August 2017Matthew Groves
Until recently, agile business had to choose between the benefits of JSON-based NoSQL databases and the benefits of SQL-based querying. NoSQL provides schema flexibility, high performance, and elastic scaling, while SQL provides expressive, independent data access. Recent convergence allows developers and organizations to have the best of both worlds.
Developers need to deliver apps that readily evolve, perform, and scale, all to match changing business needs. Organizations need rapid access to their operational data, using standard analytical tools, for insight into their business. In this session, you will learn the ways that SQL can be applied to NoSQL databases (N1QL, SQL++, ODBC, JDBC, and others), and what additional features are needed to deal with JSON documents. SQL for JSON, JSON data modeling, indexing, and tool integration will be covered.
Google's research results have gotten a number of new features over the last few years. Features like the image carousel, people also asked and videos are all known as "Rich Results." While earning these spots can seem impossible, using structured data any site can stake claim to these features.
From SQL to NoSQL: Structured Querying for JSONKeshav Murthy
Can SQL be used to query JSON? SQL is the universally known structured query language, used for well defined, uniformly structured data; while JSON is the lingua franca of flexible data management, used to define complex, variably structured data objects.
Yes! SQL can most-definitely be used to query JSON with Couchbase's SQL query language for JSON called N1QL (verbalized as Nickel.)
In this session, we will explore how N1QL extends SQL to provide the flexibility and agility inherent in JSON while leveraging the universality of SQL as a query language.
We will discuss utilizing SQL to query complex JSON objects that include arrays, sets and nested objects.
You will learn about the powerful query expressiveness of N1QL, including the latest features that have been added to the language. We will cover how using N1QL can solve your real-world application challenges, based on the actual queries of Couchbase end-users.
Tuning for Performance: indexes & QueriesKeshav Murthy
There are three things important in databases: performance, performance, performance. From a simple query to fetch a document to a query joining millions of documents, designing the right data models and indexes is important. There are many indices you can create, and many options you can choose for each index. This talk will help you understand tuning N1QL query, exploiting various types of indices, analyzing the system behavior, and sizing them correctly.
Understanding N1QL Optimizer to Tune QueriesKeshav Murthy
Every flight has a flight plan. Every query has a query plan. You must have seen its text form, called EXPLAIN PLAN. Query optimizer is responsible for creating this query plan for every query, and it tries to create an optimal plan for every query. In Couchbase, the query optimizer has to choose the most optimal index for the query, decide on the predicates to push down to index scans, create appropriate spans (scan ranges) for each index, understand the sort (ORDER BY) and pagination (OFFSET, LIMIT) requirements, and create the plan accordingly. When you think there is a better plan, you can hint the optimizer with USE INDEX. This talk will teach you how the optimizer selects the indices, index scan methods, and joins. It will teach you the analysis of the optimizer behavior using EXPLAIN plan and how to change the choices optimizer makes.
Querying NoSQL with SQL - KCDC - August 2017Matthew Groves
Until recently, agile business had to choose between the benefits of JSON-based NoSQL databases and the benefits of SQL-based querying. NoSQL provides schema flexibility, high performance, and elastic scaling, while SQL provides expressive, independent data access. Recent convergence allows developers and organizations to have the best of both worlds.
Developers need to deliver apps that readily evolve, perform, and scale, all to match changing business needs. Organizations need rapid access to their operational data, using standard analytical tools, for insight into their business. In this session, you will learn the ways that SQL can be applied to NoSQL databases (N1QL, SQL++, ODBC, JDBC, and others), and what additional features are needed to deal with JSON documents. SQL for JSON, JSON data modeling, indexing, and tool integration will be covered.
Google's research results have gotten a number of new features over the last few years. Features like the image carousel, people also asked and videos are all known as "Rich Results." While earning these spots can seem impossible, using structured data any site can stake claim to these features.
MongoDB .local Munich 2019: Still Haven't Found What You Are Looking For? Use...MongoDB
Come and hear more about our new full-text search operator for MongoDB Atlas. This is a significant enhancement to MongoDB search features and is the easiest and most powerful full-text search solution for databases on MongoDB Atlas.
This talk is important for anyone who has implemented search or is considering a search feature in their MongoDB application.
You will see a demo of $searchBeta, learn about how it works, discover specific features to help you deliver relevant search results, and learn how you can start using full-text search in your application today.
Speaker: Daniel Coupal
At this point, you may be familiar with the design of MongoDB databases and collections – but what are the frequent patterns you may have to model?
This presentation will add knowledge of how to represent common relationships (1-1, 1-N, N-N) in MongoDB. Going further than relationships, this presentation identifies a set of common patterns, in a similar way to what the Gang of Four did for Object Oriented Design. Finally, this presentation will guide you through the steps of modeling those patterns in MongoDB collections.
In this session, you will learn about:
How to create the appropriate MongoDB collections for some of the patterns discussed.
Differences in relationships vs. the relational database world, and how those differences translate to MongoDB collections.
Common patterns in developing applications with MongoDB, plus a specific vocabulary with which to refer to them.
The N1QL is a developer favorite because it’s SQL for JSON. Developer’s life is going to get easier with the upcoming N1QL features. We have exciting features in many areas including language to performance, indexing to search, and tuning to transactions. This session will preview new the features for both new and advanced users.
MongoDB .local Chicago 2019: Practical Data Modeling for MongoDB: TutorialMongoDB
For 30 years, developers have been taught that relational data modeling was THE way to model, but as more companies adopt MongoDB as their data platform, the approaches that work well in relational design actually work against you in a document model design. In this talk, we will discuss how to conceptually approach modeling data with MongoDB, focusing on practical foundational techniques, paired with tips and tricks, and wrapping with discussing design patterns to solve common real world problems.
To understand how to make your application fast, it's important to understand what makes the database fast. We will take a detailed look at how to think about performance, and how different choices in schema design affect your cluster performances depending on storage engines used and physical resources available.
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQLKeshav Murthy
Abstract
NoSQL databases bring the benefits of schema flexibility and
elastic scaling to the enterprise. Until recently, these benefits have
come at the expense of giving up rich declarative querying as
represented by SQL.
In today’s world of agile business, developers and organizations need
the benefits of both NoSQL and SQL in a single platform. NoSQL
(document) databases provide schema flexibility; fast lookup; and
elastic scaling. SQL-based querying provides expressive data access
and transformation; separation of querying from modeling and storage;
and a unified interface for applications, tools, and users.
Developers need to deliver applications that can easily evolve,
perform, and scale. Otherwise, the cost, effort, and delay in keeping
up with changing business needs will become significant disadvantages.
Organizations need sophisticated and rapid access to their operational data, in
order to maintain insight into their business. This access should
support both pre-defined and ad-hoc querying, and should integrate
with standard analytical tools.
This talk will cover how to build applications that combine the
benefits of NoSQL and SQL to deliver agility, performance, and
scalability. It includes:
- N1QL, which extends SQL to JSON
- JSON data modeling
- Indexing and performance
- Transparent scaling
- Integration and ecosystem
You will walk away with an understanding of the design patterns and
best practices for effective utilization of NoSQL document
databases - all using open-source technologies.
Variety is the spice of life, but it’s also the reality of big data. For this reason, JSON has now becoming lingua franca of data in the internet – for APIs, data exchange, data storage and data processing. In the business intelligence world, SQL is the language to analyze the data in other forms. Hence, the myriad of “SQL-on-Hadoop” projects. However, traditional SQL isn’t JSON/Parquet/etc. friendly. ETL into flattened tables is costly and not real time.
Apache Drill unifies SQL with variety of data forms on Hadoop. That enables interactive analytics using your favorite BI tool and visualization tool on you data simultaneously. In this talk, we’ll introduce Apache Drill and describe use cases.
- See more at: http://nosql2014.dataversity.net/sessionPop.cfm?confid=81&proposalid=6850#sthash.NhuLz6Dq.dpuf
Introduction to NoSQL (Use case, considerations to NoSQL). Presentation of Couchbase with update on Couchbase 4.5 and Couchbase mobile 1.2. Special thanks to Dipti Borkar who inspired a large part of this presentation.
Rolling presentation during Couchbase Day. Including
Introduction to NoSQL
Why NoSQL?
Introduction to Couchbase
Couchbase Architecture
Single Node Operations
Cluster Operations
HA and DR
Availability and XDCR
Backup/Restore
Security
Developing with Couchbase
Couchbase SDKs
Couchbase Indexing
Couchbase GSI and Views
Indexing and Query
Couchbase Mobile
SDEC2011 Using Couchbase for social game scaling and speedKorea Sdec
A social game, by it's very nature, can spread very quickly to a large user population. Because the game is typically interactive, the speed of retrieving information needed for the user's interactions with the system is critical. When building their new game Animal Party, the developers at Tribal Crossing needed to get away from the complexity of sharding an SQL database. They also were looking for a solution to the administration cost associated with the operation of traditional data stores. When evaluating multiple different NoSQL solutions, they realized that Couchbase's Membase server meets most of their critical requirements in developing their game software. Simple to use, Couchbase's model allows Tribal Crossing to easily model their game interactions with the key/value data store. Fast read and write performance is required with interactive, social games, and they found that support in Membase as well. Elastic scalability is easily achieved by simply adding more nodes to the Couchbase cluster without any modifications required to the application. Relying on Couchbase's technology Tribal Crossing has been able to quickly build and scale Animal Party with a small team and no dedicated system administrators.
http://sdec.kr/
Accelerating analytics on the Sensor and IoT Data. Keshav Murthy
Informix Warehouse Accelerator (IWA) has helped traditional
data warehousing performance to improve dramatically. Now,
IWA accelerates analytics over the sensor data stored in relational and timeseries data.
N1QL = SQL + JSON. N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We begin with a brief overview. Couchbase 5.0 has language and performance improvements for pagination, index exploitation, integration, and more. We’ll walk through scenarios, features, and best practices.
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5Keshav Murthy
N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We’ll begin this session with a brief overview of N1QL and then explore some key enhancements we’ve made in the latest versions of Couchbase Server. Couchbase Server 5.0 has language and performance improvements for pagination, index exploitation, integration, index availability, and more. Couchbase Server 5.5 will offer even more language and performance features for N1QL and global secondary indexes (GSI), including ANSI joins, aggregate performance, index partitioning, auditing, and more. We’ll give you an overview of the new features as well as practical use case examples.
MongoDB .local Munich 2019: Still Haven't Found What You Are Looking For? Use...MongoDB
Come and hear more about our new full-text search operator for MongoDB Atlas. This is a significant enhancement to MongoDB search features and is the easiest and most powerful full-text search solution for databases on MongoDB Atlas.
This talk is important for anyone who has implemented search or is considering a search feature in their MongoDB application.
You will see a demo of $searchBeta, learn about how it works, discover specific features to help you deliver relevant search results, and learn how you can start using full-text search in your application today.
Speaker: Daniel Coupal
At this point, you may be familiar with the design of MongoDB databases and collections – but what are the frequent patterns you may have to model?
This presentation will add knowledge of how to represent common relationships (1-1, 1-N, N-N) in MongoDB. Going further than relationships, this presentation identifies a set of common patterns, in a similar way to what the Gang of Four did for Object Oriented Design. Finally, this presentation will guide you through the steps of modeling those patterns in MongoDB collections.
In this session, you will learn about:
How to create the appropriate MongoDB collections for some of the patterns discussed.
Differences in relationships vs. the relational database world, and how those differences translate to MongoDB collections.
Common patterns in developing applications with MongoDB, plus a specific vocabulary with which to refer to them.
The N1QL is a developer favorite because it’s SQL for JSON. Developer’s life is going to get easier with the upcoming N1QL features. We have exciting features in many areas including language to performance, indexing to search, and tuning to transactions. This session will preview new the features for both new and advanced users.
MongoDB .local Chicago 2019: Practical Data Modeling for MongoDB: TutorialMongoDB
For 30 years, developers have been taught that relational data modeling was THE way to model, but as more companies adopt MongoDB as their data platform, the approaches that work well in relational design actually work against you in a document model design. In this talk, we will discuss how to conceptually approach modeling data with MongoDB, focusing on practical foundational techniques, paired with tips and tricks, and wrapping with discussing design patterns to solve common real world problems.
To understand how to make your application fast, it's important to understand what makes the database fast. We will take a detailed look at how to think about performance, and how different choices in schema design affect your cluster performances depending on storage engines used and physical resources available.
Bringing SQL to NoSQL: Rich, Declarative Query for NoSQLKeshav Murthy
Abstract
NoSQL databases bring the benefits of schema flexibility and
elastic scaling to the enterprise. Until recently, these benefits have
come at the expense of giving up rich declarative querying as
represented by SQL.
In today’s world of agile business, developers and organizations need
the benefits of both NoSQL and SQL in a single platform. NoSQL
(document) databases provide schema flexibility; fast lookup; and
elastic scaling. SQL-based querying provides expressive data access
and transformation; separation of querying from modeling and storage;
and a unified interface for applications, tools, and users.
Developers need to deliver applications that can easily evolve,
perform, and scale. Otherwise, the cost, effort, and delay in keeping
up with changing business needs will become significant disadvantages.
Organizations need sophisticated and rapid access to their operational data, in
order to maintain insight into their business. This access should
support both pre-defined and ad-hoc querying, and should integrate
with standard analytical tools.
This talk will cover how to build applications that combine the
benefits of NoSQL and SQL to deliver agility, performance, and
scalability. It includes:
- N1QL, which extends SQL to JSON
- JSON data modeling
- Indexing and performance
- Transparent scaling
- Integration and ecosystem
You will walk away with an understanding of the design patterns and
best practices for effective utilization of NoSQL document
databases - all using open-source technologies.
Variety is the spice of life, but it’s also the reality of big data. For this reason, JSON has now becoming lingua franca of data in the internet – for APIs, data exchange, data storage and data processing. In the business intelligence world, SQL is the language to analyze the data in other forms. Hence, the myriad of “SQL-on-Hadoop” projects. However, traditional SQL isn’t JSON/Parquet/etc. friendly. ETL into flattened tables is costly and not real time.
Apache Drill unifies SQL with variety of data forms on Hadoop. That enables interactive analytics using your favorite BI tool and visualization tool on you data simultaneously. In this talk, we’ll introduce Apache Drill and describe use cases.
- See more at: http://nosql2014.dataversity.net/sessionPop.cfm?confid=81&proposalid=6850#sthash.NhuLz6Dq.dpuf
Introduction to NoSQL (Use case, considerations to NoSQL). Presentation of Couchbase with update on Couchbase 4.5 and Couchbase mobile 1.2. Special thanks to Dipti Borkar who inspired a large part of this presentation.
Rolling presentation during Couchbase Day. Including
Introduction to NoSQL
Why NoSQL?
Introduction to Couchbase
Couchbase Architecture
Single Node Operations
Cluster Operations
HA and DR
Availability and XDCR
Backup/Restore
Security
Developing with Couchbase
Couchbase SDKs
Couchbase Indexing
Couchbase GSI and Views
Indexing and Query
Couchbase Mobile
SDEC2011 Using Couchbase for social game scaling and speedKorea Sdec
A social game, by it's very nature, can spread very quickly to a large user population. Because the game is typically interactive, the speed of retrieving information needed for the user's interactions with the system is critical. When building their new game Animal Party, the developers at Tribal Crossing needed to get away from the complexity of sharding an SQL database. They also were looking for a solution to the administration cost associated with the operation of traditional data stores. When evaluating multiple different NoSQL solutions, they realized that Couchbase's Membase server meets most of their critical requirements in developing their game software. Simple to use, Couchbase's model allows Tribal Crossing to easily model their game interactions with the key/value data store. Fast read and write performance is required with interactive, social games, and they found that support in Membase as well. Elastic scalability is easily achieved by simply adding more nodes to the Couchbase cluster without any modifications required to the application. Relying on Couchbase's technology Tribal Crossing has been able to quickly build and scale Animal Party with a small team and no dedicated system administrators.
http://sdec.kr/
Accelerating analytics on the Sensor and IoT Data. Keshav Murthy
Informix Warehouse Accelerator (IWA) has helped traditional
data warehousing performance to improve dramatically. Now,
IWA accelerates analytics over the sensor data stored in relational and timeseries data.
N1QL = SQL + JSON. N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We begin with a brief overview. Couchbase 5.0 has language and performance improvements for pagination, index exploitation, integration, and more. We’ll walk through scenarios, features, and best practices.
N1QL+GSI: Language and Performance Improvements in Couchbase 5.0 and 5.5Keshav Murthy
N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We’ll begin this session with a brief overview of N1QL and then explore some key enhancements we’ve made in the latest versions of Couchbase Server. Couchbase Server 5.0 has language and performance improvements for pagination, index exploitation, integration, index availability, and more. Couchbase Server 5.5 will offer even more language and performance features for N1QL and global secondary indexes (GSI), including ANSI joins, aggregate performance, index partitioning, auditing, and more. We’ll give you an overview of the new features as well as practical use case examples.
Introducing N1QL: New SQL Based Query Language for JSONKeshav Murthy
This session introduces N1QL and sets the stage for the rich selection of N1QL-related sessions at Couchbase Connect 2015. N1QL is SQL for JSON, extending the querying power of SQL with the modeling flexibility of JSON. In this session, you will get an introduction to the N1QL language, architecture, and ecosystem, and you will hear the benefits of N1QL for developers and for enterprises.
SQL for JSON: Rich, Declarative Querying for NoSQL Databases and Applications Keshav Murthy
In today’s world of agile business, Java developers and organizations benefit when JSON-based NoSQL databases and SQL-based querying come together. NoSQL provides schema flexibility and elastic scaling. SQL provides expressive, independent data access. Java developers need to deliver apps that readily evolve, perform, and scale with changing business needs. Organizations need rapid access to their operational data, using standard analytical tools, for insight into their business. In this session, you will learn to build apps that combine NoSQL and SQL for agility, performance, and scalability. This includes
• JSON data modeling
• Indexing
• Tool integration
Persisting data in NoSQL document databases, such as Couchbase, offers a lot more options and flexibility than relational databases (RDBMS) like SQL Server. These choices can be daunting at first, and involve trade-offs between concurrency, consistency, and performance.
The goal of this session will be to demystify NoSQL data modeling techniques for Couchbase. We will cover everything from a basic overview of data types and relationships all the way to how the Domain Driven Design approach to modeling can be applied to Couchbase.
Document Data Modelling with Couchbase Server 4.0Cihan Biyikoglu
Couchbase Server 4.0 is both a document and a key-value data store. In this talk, I detail the steps for designing an efficient data model for Couchbase Server with examples that demonstrate when referencing vs embedding is the right choice. I also walk through how N1QL, the new SQL-like query language for JSON data, makes it easy to reshape the data into the format your application excepts without friction.
NoSQL databases threw out SQL for querying, while their authors focused on solving problems on scale, speed and availability. The trouble is, the need for rich query never went away. Neither did SQL; it was only resting. Today, non-relational databases are bringing back SQL-like languages and other query mechanisms to help them integrate with existing data query layers (e.g. Hibernate) and to fit in with the overwhelming weight of database query practice of the past 40 years. In this talk I’ll cover how to query data in Couchbase using a SQL like query language called N1QL (Nickel) from .NET. I will also touch on topics like installing, setup, hosting, cloud, azure and perfomance. Demo's will be in C#, but there are SDK's and API’s for all the major languages, therefor if you don’t have a .NET background you can still use what you learn in your favourite language.
You can also check this version with animations: http://www.slideshare.net/martinesmann/no-sq-lsbiggestliesqlneverwentawaymartinesmann
0 to 60 with AWS AppSync: Rapid Development Techniques for Mobile APIs (MOB32...Amazon Web Services
Building a working prototype can help you quickly gather feedback from users, but mock data can be limiting, and backend services might not exist in the state you need. In this chalk talk, we explore how you can use AWS AppSync to build a mobile API with little to no coding and manage existing APIs to empower small development teams or bring your prototypes to life.
Speaker:Drew DiPalma
Learn more about MongoDB Stitch, our new Backend as a Service (BaaS) that makes it easy for developers to create and launch applications across mobile and web platforms. Stitch provides a REST API on top of MongoDB with read, write, and validation rules built-in and full integration with the services you love. This talk will cover the what, why, and how of MongoDB Stitch. We'll discuss everything from features to the architecture. You'll walk away knowing how Stitch can kickstart your new project or take your existing application to the next level.
Queries need indexes to speed up and optimize resource utilization. What indexes to create and what rules to follow to create right indexes to optimize the workload? This presentation gives the rules for those.
MongoDB offers two native data processing tools: MapReduce and the Aggregation Framework. MongoDB’s built-in aggregation framework is a powerful tool for performing analytics and statistical analysis in real-time and generating pre-aggregated reports for dashboarding. In this session, we will demonstrate how to use the aggregation framework for different types of data processing including ad-hoc queries, pre-aggregated reports, and more. At the end of this talk, you should walk aways with a greater understanding of the built-in data processing options in MongoDB and how to use the aggregation framework in your next project.
XLDB Lightning Talk: Databases for an Engaged World: Requirements and Design...Keshav Murthy
Traditional databases have been designed for system of record and analytics. Modern enterprises have orders of magnitude more interactions than transactions. Couchbase Server is a rethinking of the database for interactions and engagements called, Systems of Engagement. Memory today is much cheaper than disks were when traditional databases were designed back in the 1970's, and networks are much faster and much more reliable than ever before. Application agility is also an extremely important requirement. Today's Couchbase Server is a memory- and network-centric, shared-nothing, auto-partitioned, and distributed NoSQL database system that offers both key-based and secondary index-based data access paths as well as API- and query-based data access capabilities. This lightning talk gives you an overview of requirements posed by next-generation database applications and approach to implementation including “Multi Dimensional Scaling.
Couchbase 5.5: N1QL and Indexing featuresKeshav Murthy
This deck contains the high-level overview of N1QL and Indexing features in Couchbase 5.5. ANSI joins, hash join, index partitioning, grouping, aggregation performance, auditing, query performance features, infrastructure features.
N1QL (SQL for JSON) has built-in rule based optimizer. In Couchbase 5.0, N1QL's optimizer has a number of improvements for resource utilization, performance improvements. This deck deck by Couchbase Principal Engineer, Sitaram describes those improvements.
Mindmap: Oracle to Couchbase for developersKeshav Murthy
This deck provides a high-level comparison between Oracle and Couchbase: Architecture, database objects, types, data model, SQL & N1QL statements, indexing, optimizer, transactions, SDK and deployment options.
N1QL supports select, join, project,nest,unnest operations on flexible schema documents represented in JSON.
Couchbase 4.5 enhances the data modeling and query flexibility.
When you have parent-child relationship, children documents point to parent document, you join from child to parent. Now, how would you join from parent to child when parent does not contain the reference to child? How would you improve performance on this? This presentation explain the syntax, execution of the query.
Enterprise Architect's view of Couchbase 4.0 with N1QLKeshav Murthy
Enterprise architects have to decide on the database platform that will meet various requirements: performance and scalability on one side, ease of data modeling, agile development on the other, elasticity and flexibility to handle change easily, and a database platform that integrates well with tools and within ecosystem. This presentation will highlight the challenges and approaches to solution using Couchbase with N1QL.
You know what iMEAN? Using MEAN stack for application dev on InformixKeshav Murthy
You know what iMEAN? Using MEAN stack for application dev on Informix. MongoDB, ExpressJS, AngularJS, NodeJS combine to form a MEAN stack for quick appdev. iMEAN is using the same stack to develop applications on Informix.
Informix SQL & NoSQL: Putting it all togetherKeshav Murthy
Building IoT applications and handling IoT data with SQL, NoSQL, JSON, Spatial, timeseries, etc. Details of Informix SQL and NoSQL implementation and some things planned for 12.10.FC4.
NoSQL Deepdive - with Informix NoSQL. IOD 2013Keshav Murthy
Deepdive into IBM Informix NoSQL with Mongodb API, hybrid access. Informix now supports MongoDB API and stores JSON natively, thus supporting flexible schema. Informix NoSQL also supports sharding, enabling scale out. This presentation gives overview to a real application to technical details of this
Table for two? Hybrid approach to developing combined SQL, NoSQL applications...Keshav Murthy
Informix embraces the NoSQL by implementing flexible schema via JSON, sharding and MongoAPI. Additionally, data in sql tables and JSON collections can by accessed by any API and use the power of the RDBMS engine.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Arrays can be simple; arrays can be complex. JSON arrays give you a method to collapse the data model while retaining structure flexibility. Arrays of scalars, objects, and arrays are common structures in a JSON data model. Once you have this, you need to write queries to update and retrieve the data you need efficiently. This talk will discuss modeling and querying arrays. Then, it will discuss using array indexes to help run those queries on arrays faster.
Arrays can be simple; arrays can be complex. JSON arrays give you a method to collapse the data model while retaining structure flexibility. Arrays of scalars, objects, and arrays are common structures in a JSON data model. Once you have this, you need to write queries to update and retrieve the data you need efficiently. This talk will discuss modeling and querying arrays. Then, it will discuss using array indexes to help run those queries on arrays faster.
Arrays can be simple; arrays can be complex. JSON arrays give you a method to collapse the data model while retaining structure flexibility. Arrays of scalars, objects, and arrays are common structures in a JSON data model. Once you have this, you need to write queries to update and retrieve the data you need efficiently. This talk will discuss modeling and querying arrays. Then, it will discuss using array indexes to help run those queries on arrays faster.
cbq> select distinct type from `travel-sample`;
{
"requestID": "458b7651-53a3-4a83-9abe-b65959420010",
"signature": {
"type": "json"
},
"results": [
{
"type": "route"
},
{
"type": "airport"
},
{
"type": "hotel"
},
{
"type": "airline"
},
{
"type": "landmark"
}
],
"status": "success",
"metrics": {
"elapsedTime": "840.518052ms",
"executionTime": "840.478414ms",
"resultCount": 5,
"resultSize": 202
}
An array is a way to hold more than one value at a time. It’s like a list of items.
Think of an array as the columns in a spreadsheet. You can have a spreadsheet with only one column or lots of columns.
An array is a way to hold more than one value at a time. It’s like a list of items.
Think of an array as the columns in a spreadsheet. You can have a spreadsheet with only one column or lots of columns.
An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).
An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).
An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).
An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
Let’s look at modeling Customer data.
Rich Structure
In relational database, this customers data would be stored in five normalized tables.
Each time you want to construct a customer object, you JOIN the data in these tables;
Each time you persist, you find the appropriate rows in relevant tables and insert/update.
Relationship
Enforcement is via referential constraints. Objects are constructed by JOINS, EACH time.
Value Evolution
Additional values of the SAME TYPE (e.g. additional phone, additional address) is managed by additional ROWS in one of the tables.
Customer:contacts will have 1:n relationship.
Structure Evolution:
This is the most difficult part.changing the structure is difficult, within a table, across tae table.
While you can do these via ALTER TABLE, requires downtime, migration and application versioning.
This is one of the problem document databases try to handle by representing data in JSON.
Let’s see how to represent customer data in JSON.
So, finally, you have a JSON document that represents a CUSTOMER.
In a single JSON document, relationship between the data is implicit by use of sub-structures and arrays and arrays of sub-structures.
The whole array is one blob of value that was indexed before 4.5. Any query should have to specify the entire array to find a match, which was not practical
Let’s see how array indexing helps. First it enables visibility into the array structure, so index can be created on subset of finer array elements or attributes.
With Array Indexing, subset of the array elements or attributes can be individually indexed & searched
We can index only required subset of the array, and hence be efficient on Index storage & search times.
Clearly, Benefits are lot more effectively visible with nested arrays/objects
For example, index created in earlier versions would look like the blue triangle with whole array indexed.
With array indexes in 4.5 only flight attributes with in the array can be indexed, which is much more efficient on storage and performance.
In summary, array indexing brings Performance, and ease of querying with arrays
For ex: this SELECT statement finds the total number of flights scheduled on 3rd day of the week,
It iterates using the ANY operator to find matching index keys.
Note that, the DML statement uses the exact array variables and predicates which are used in create index
this example creates composite index with attributes in the array such as ‘v.flight’, where v is an array element, and non-array attribute such as ‘stops’.
The SELECT query Finds all scheduled flights with one or more stops, and groups the result by number of stops.
Note how the array elements can be iterated in the projection list of SELECT
Lets look at an example with nested arrays. Consider the schedule array in travel-sample, with the nested array special-flights.
So, the create index statement also uses nested DISTINCT ARRAY construct to create the index on each distinct special flight.
Here is a SELECT statement to find the total number of scheduled special flights, which uses.
Again, note the nested form of ANY construct and the use of matching variables names & index keys of the corresponding CREATE index statement.
This feature has few. First the variable names and index keys, such as v & v.day, that are used in CREATE INDEX & SELECT must exactly match
The query predicate, which must appear in the WHERE clause of a SELECT, UPDATE, or DELETE statement, must have the exact matching format as the variable in the array index key, including the name of the variable like v.
Only the operators… are supported.
3. SELECT * FROM default WHERE ANY c IN cities SATISFIES c = "Bombay" END;
4. SELECT * FROM default WHERE ANY c IN cities SATISFIES c = "Bombay" END AND age < 35 ;
The select in #4 can be done using index in #3. But range low and high are different depending on index created.
If #4 select is used with #3 create index, then range is: High= ["\"Bombay\"”] Low= ["\"Bombay\"”] Inclusion: 3
If #4 select is used with #4 create index, then range is: High= ["\"Bombay\"”,"35”] Low= ["\"Bombay\"”,"null”] Inclusion: 0
#6 :
If two docs are:
D1 =
{
"age": 25,
"cities":[["Bangalore","Mysore"],["Chennai","Ooty"]]
}
D2 =
{
"age": 30,
"cities":[["Siliguri","Kolkata"],["Kohlapur","Mumbai"]]}
}
Then Create index query would be:
CREATE INDEX idcities_nested ON default(ALL ARRAY (ALL ARRAY y for y IN c END) FOR c IN cities END)
The above query throughput (queries per second) measured for ForestDB with 3.6K set ops per second.
The above query throughput (queries per second) measured for MOI with 30K set ops per second.
Q1 took 13 ms but with Couchbase query, it took about 1500 ms.